figure out a bunch of weird stuff and doc it, add struct to hold torrent info

README.md

@@ -1,6 +1,23 @@
 # Torznab Toolkit
 
-A safe, multi-threaded toolkit for adding Torznab APIs to programs.
+A safe, multi-threaded toolkit for adding Torznab APIs to programs. You just focus on the indexer itself, we abstract away the hell that is the Torznab API.
+
+Just fill in your own relevant functions and config, and torznab-toolkit will run the API for you
+
+```rs
+use torznab_toolkit;
+let config: torznab_toolkit::config::Config = /* config goes here */
+
+torznab_toolkit::run(config);
+```
+
+The environment variables `ROCKET_ADDRESS` and `ROCKET_PORT` specify the address and port it will run on; these currently cannot be configured any other way. See the [relevant docs](https://rocket.rs/guide/v0.5/deploying/) for details.
+
+---
+
+This program is brought to you by: metaphorical *and* literal truckloads of structs!
+
+Note: I wrote the line above when I was tired. Don't ask me what *literal* truckloads of structs means, I don't know either.
 
 ## Functionality
 
@@ -16,6 +33,11 @@ A safe, multi-threaded toolkit for adding Torznab APIs to programs.
 <!-- for copy-pasting: ❌ ✅ -->
 (copied from [torznab.github.io](https://torznab.github.io/spec-1.3-draft/torznab/Specification-v1.3.html))
 
+## Limitations
+
+- Currently this does not allow for returning errors from the program using the library, such as API limits for an account.
+- Currently this does not allow for requiring authentication for `caps`; it's against spec (not that that's worth much), but common and perfectly fine to do.
+
 ## Notes
 
-Thanks to [torznab.github.io](https://torznab.github.io/spec-1.3-draft/index.html), as it's my primary reference for this.
+Thanks to [torznab.github.io](https://torznab.github.io/spec-1.3-draft/index.html), as it's my primary reference for this; NZBDrone's [Implementing a Torznab indexer](https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/) was also rather helpful.

dev-notes.md

@@ -4,19 +4,82 @@
 
 - <https://torznab.github.io/spec-1.3-draft/index.html>
 - <https://www.git.je/Mirrors/Sonarr/wiki/Implementing-a-Torznab-indexer>
+- <https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/>
 - for testing: <https://fosstorrents.com/thankyou/?name=debian&cat=Installation%20-%20amd64&id=0&hybrid=0>
 
 ---
 
-```rs
+example usage:
-struct TorznabToolkitConfig {
-    auth_func: auth,
-    search_func: search,
-    whateverotherfunc: otherfunc,
-    port: 5309
-}
 
-fn launch() -> Result {
+```rs
-    rocket::build().mount("/", routes![tt::search, tt:otherfunc])
+let config = /* config goes here */
+
+fn main() -> Result {
+    run(config);
 }
 ```
+
+## Torznab spec
+
+### Query results
+
+Queries are returned as an RSS feed something like this:
+
+```rss
+<rss version="1.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:torznab="http://torznab.com/schemas/2015/feed">
+  <channel>
+    <atom:link rel="self" type="application/rss+xml" />
+    <title>Prowlarr</title>
+    <item>
+      <title>Item title</title>
+      <description />
+      <link>http://localhost:9999/</link>
+      <torznab:attr name="category" value="1010" />
+      <torznab:attr name="category" value="84570" />
+      <torznab:attr name="size" value="1073741824" />
+      <torznab:attr name="">
+      <enclosure url="http://localhost/" length="0" type="application/x-bittorrent" />
+    </item>
+  </channel>
+</rss>
+```
+
+Item attributes:
+
+- RSS:
+  - `title`: title of the thing; can maybe be empty? unsure - torznab-toolkit will treat this as required, and set description to be empty
+  - `description`: description; can just be empty
+  - `link`: a link
+    - Preferably specified, if not then fallback to .torrent link, then magnet url
+  - at least title *or* description is required; link is recommended for compatibility with some non-compliant software, like headphones (see [bitmagnet issue 349](https://github.com/bitmagnet-io/bitmagnet/issues/349))
+- Torznab
+  - Required:
+    - `size`: Size in bytes
+    - `category`: (Sub)category id; can occur multiple times for several categories
+  - Recommended (by me, based off my own thoughts and NZBDrone's [Implementing a Torznab indexer](https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/) page):
+    - `seeders`: Number of seeders
+    - `leechers`: Number of leechers
+    - `peers`: Number of peers
+    - `infohash`: Torrent infohash
+    - `magneturl`: The magnet URI for the torrent
+  - URLs:
+    - Main URI can either be a magnet URI or a link to a .torrent file: `<enclosure url="http://localhost/" length="0" type="application/x-bittorrent" />`
+      - Length is ambiguous, so it will just be 0 (see below)
+    - If .torrent URL is provided, use that, if not use the magnet; also put the magnet in `magneturl`
+  - Rest of available attributes: <https://torznab.github.io/spec-1.3-draft/torznab/Specification-v1.3.html?highlight=server#predefined-attributes>
+
+---
+
+## Non-compliance and errors in the Torznab spec
+
+- Prowlarr's built-in indexer for sites it scrapes (like 1337x) does not respect `limit`
+- Negative `limit`: The spec says that if `limit` is negative, then to return `201 - Incorrect parameter`; but 201 is Created, and there's no Incorrect Parameter HTTP status code (my best guess is it should be 400?). Additionally, other indexers just ignore the limit if it's negative, so in that case, torznab-toolkit will just set the limit to the maximum to match the behavior of other widely-used indexers.
+
+## Ambiguous behavior
+
+- Due to ambiguous behavior, torznab-toolkit will set `length` to 0
+
+> The correct definition of the enclosure length attribute should have been the size of the .torrent file, however newznab defined it as the length of the usenet download rather than the .nzb size.
+> Therefore the length attribute can be either 0, or the .torrent/.nzb file size, or the actual media size. Given this ambiguity, services should instead provide the size extended attribute with the size of the media. (eg. <torznab:attr name=”size” value=”1460985071” />)
+
+(from [Torznab spec - Torznab Torrent Support](https://torznab.github.io/spec-1.3-draft/revisions/1.0-Torznab-Torrent-Support.html]))

src/api.rs

@@ -8,6 +8,7 @@ use rocket::http::Status;
 use rocket::response::status;
 use rocket::FromForm;
 use rocket::{get, response::content::RawXml};
+use std::collections::HashMap;
 use xml::writer::{EmitterConfig, XmlEvent};
 
 #[derive(Debug, Clone, PartialEq, Eq, FromForm)]
@@ -34,6 +35,7 @@ struct SearchForm {
 }
 
 impl SearchForm {
+    /// Converts it to a SearchParameters object
     fn to_parameters(&self, conf: Config) -> SearchParameters {
         // TODO: Clean up this code - split it into a separate function?
         let mut categories: Option<Vec<u32>> = None;
@@ -131,6 +133,7 @@ pub(crate) fn caps() -> status::Custom<RawXml<String>> {
 
 #[get("/api?t=search&<form..>")]
 /// The search function for the API
+// FIXME: VERY incomplete also
 pub(crate) fn search(form: SearchForm) -> status::Custom<RawXml<String>> {
     // The compiler won't let you get a field from a struct in the Option here, since the default is None
     // So this is needed
@@ -151,10 +154,7 @@ pub(crate) fn search(form: SearchForm) -> status::Custom<RawXml<String>> {
             match parameters.apikey {
                 Some(apikey) => {
                     if !auth(apikey).unwrap() {
-                        return status::Custom(
+                        unauthorized = true;
-                            Status::Unauthorized,
-                            RawXml("401 Unauthorized".to_string()),
-                        );
                     }
                 }
                 None => {

src/data.rs

@@ -1,5 +1,7 @@
 //! Contains tons of structs used by the library
 
+use std::collections::HashMap;
+
 use rocket::FromForm;
 pub(crate) type AuthFunc = fn(String) -> Result<bool, String>;
 // TODO: Figure out what the arguments should be for a search function and what it should return
@@ -97,17 +99,23 @@ pub struct Tag {
 #[derive(Debug, Clone, PartialEq, Eq)]
 /// Holds the configuration for the capabilities of the Torznab server (used in `/api?t=caps`)
 ///
-/// <div class="warning">Note that this library might not support all the capabilities listed in yet, so check the README before listing capabilities, or just accept that unsupported capabilities will return error 404.
+/// <div class="warning">Note that this library might not support all the capabilities listed in yet, so check the README before listing capabilities, or just accept that unsupported capabilities will return error 501.
 ///
 /// It's recommended to add any capabilities you want, and set `available` to `false` in the [`Caps`] struct for any currently unsupported search types.</div>
 ///
 /// TODO: Add a way to partially(?) generate automatically from the Config
 pub struct Caps {
-    pub server_info: ServerInfo,
+    /// The server info, like title - optional
+    pub server_info: Option<ServerInfo>,
+    /// The max and default number of items to be returned by queries - see [`Limits`]
     pub limits: Limits,
+    /// Info about each type of search
     pub searching: Vec<SearchInfo>,
+    /// What categories the server has - see [`Category`]
     pub categories: Vec<Category>,
+    /// What genres the server has - see [`Genre`] (optional)
     pub genres: Option<Vec<Genre>>,
+    /// What torrents can be tagged with - see [`Tag`] (optional)
     pub tags: Option<Vec<Tag>>,
 }
 
@@ -116,12 +124,19 @@ pub struct Caps {
 /// The search function (`/api?t=search`) and capabilities (`/api?t=caps` - struct [`Caps`]) are required
 /// Everything else is optional
 pub struct Config {
+    /// The function to use for a free text search
     pub search: SearchFunc,
+    /// The auth function - if not specified, then no authorization is needed.
     pub auth: Option<AuthFunc>,
+    /// The capabilities of the indexer - see [`Caps`]
     pub caps: Caps,
+    /// The function to use for a tv search
     pub tvsearch: Option<SearchFunc>,
+    /// The function to use for a movie search
     pub movie: Option<SearchFunc>,
+    /// The function to use for a music search
     pub music: Option<SearchFunc>,
+    /// The function to use for a book search
     pub book: Option<SearchFunc>,
 }
 
@@ -142,3 +157,20 @@ pub(crate) struct SearchParameters {
     /// The maximum number of items to return - also limited to whatever `limits` is in [`Caps`]
     pub(crate) limit: u32,
 }
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+/// Holds the info for a torrent
+pub struct Torrent {
+    title: String,
+    description: Option<String>,
+    size: u64,
+    categories: Vec<Category>,
+    seeders: Option<u32>,
+    leechers: Option<u32>,
+    peers: Option<u32>,
+    infohash: Option<String>,
+    link: Option<String>,
+    torrent_file_url: Option<String>,
+    magnet_uri: Option<String>,
+    other_attributes: Option<HashMap<String, String>>,
+}

src/dummy.rs

@@ -48,12 +48,12 @@ pub(crate) fn create_empty_config() -> Config {
         search: dummy_search_func,
         auth: Some(dummy_auth_func),
         caps: Caps {
-            server_info: ServerInfo {
+            server_info: Some(ServerInfo {
                 title: Some("Test Torznab server".to_string()),
                 email: Some("test@example.com".to_string()),
                 image: None,
                 version: Some("1.0".to_string()),
-            },
+            }),
             limits: Limits {
                 max: 100,
                 default: 20,

src/lib.rs

@@ -1,23 +1,5 @@
 #![warn(missing_docs)]
-//! A toolkit for adding Torznab APIs to programs.
+#![doc = include_str!("../README.md")]
-//!
-//! Just fill in your own relevant functions and config, and
-//! torznab-toolkit will run the API for you
-//!
-//! ```rs
-//! use torznab_toolkit;
-//! let config: torznab_toolkit::config::Config =
-//!
-//! ```
-//!
-//! The environment variables `ROCKET_ADDRESS` and `ROCKET_PORT` specify the address and port it will run on; these currently cannot be configured any other way. See the [relevant docs](https://rocket.rs/guide/v0.5/deploying/) for details.
-//!
-//! ---
-//!
-//! This program is brought to you by: metaphorical *and* literal truckloads of structs!
-//!
-//! Note: I wrote the line above when I was tired. Don't ask me what *literal* truckloads of structs means, I don't know either.
-
 pub mod api;
 pub mod data;
 mod dummy;
@@ -42,3 +24,6 @@ pub async fn run(conf: data::Config) -> Result<bool, rocket::Error> {
         }
     }
 }
+
+/// Notes regarding the usage of torznab-toolkit and how it implements the Torznab API.
+pub mod notes;

src/notes/implementation.rs

@@ -0,0 +1,6 @@
+//! Notes regarding the implementation of Torznab
+//!
+//! - Because the behavior of `length` is ambiguous, torznab-toolkit just sets it to 0; the size is just specified by the `size` attribute
+//!   - See [here](https://torznab.github.io/spec-1.3-draft/revisions/1.0-Torznab-Torrent-Support.html) for details
+//! - Many indexers do not have the appropriate behavior according to the spec when `limit` is negative, and that behavior doesn't even make sense; instead, it follows the behavior of other indexers, and just ignores `limit` if it's negative.
+//! - If a link isn't specified for a Torrent

src/notes/mod.rs

@@ -0,0 +1 @@
+pub mod usage;

src/notes/usage.rs

@@ -0,0 +1,8 @@
+//! Notes regarding the usage of torznab-tooolkit
+//!
+//! ---
+//!
+//! - Please implement the `season`, `ep`, and `id` attributes for torrents when possible
+//!   - Implementing `id`, at least, is far out of scope of this library, and providing `season` and `ep` more effective than this library parsing for them. However, parsing for those as an optional fallback may be added later.
+//!   - See [here](https://torznab.github.io/spec-1.3-draft/revisions/1.0-Torznab-Torrent-Support.html) for details
+// TODO: Add parsing for `season` and `ep`