askiiart/torznab-toolkit
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

figure out a bunch of weird stuff and doc it, add struct to hold torrent info

Browse source
This commit is contained in:
askiiart 2024-11-29 15:49:33 -06:00
parent 666ea1fd43
commit 8ad7ba87ac
Signed by untrusted user who does not match committer: askiiart
GPG key ID: EA85979611654C30
9 changed files with 155 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
README.md
View file

@ -1,6 +1,23 @@
# Torznab Toolkit
A safe, multi-threaded toolkit for adding Torznab APIs to programs.
A safe, multi-threaded toolkit for adding Torznab APIs to programs. You just focus on the indexer itself, we abstract away the hell that is the Torznab API.
Just fill in your own relevant functions and config, and torznab-toolkit will run the API for you
```rs
use torznab_toolkit;
let config: torznab_toolkit::config::Config = /* config goes here */
torznab_toolkit::run(config);
```
The environment variables `ROCKET_ADDRESS` and `ROCKET_PORT` specify the address and port it will run on; these currently cannot be configured any other way. See the [relevant docs](https://rocket.rs/guide/v0.5/deploying/) for details.
---
This program is brought to you by: metaphorical *and* literal truckloads of structs!
Note: I wrote the line above when I was tired. Don't ask me what *literal* truckloads of structs means, I don't know either.
## Functionality
@ -16,6 +33,11 @@ A safe, multi-threaded toolkit for adding Torznab APIs to programs.
<!-- for copy-pasting: ❌ ✅ -->
(copied from [torznab.github.io](https://torznab.github.io/spec-1.3-draft/torznab/Specification-v1.3.html))
## Limitations
- Currently this does not allow for returning errors from the program using the library, such as API limits for an account.
- Currently this does not allow for requiring authentication for `caps`; it's against spec (not that that's worth much), but common and perfectly fine to do.
## Notes
Thanks to [torznab.github.io](https://torznab.github.io/spec-1.3-draft/index.html), as it's my primary reference for this.
Thanks to [torznab.github.io](https://torznab.github.io/spec-1.3-draft/index.html), as it's my primary reference for this; NZBDrone's [Implementing a Torznab indexer](https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/) was also rather helpful.

81
dev-notes.md
View file

@ -4,19 +4,82 @@
- <https://torznab.github.io/spec-1.3-draft/index.html>
- <https://www.git.je/Mirrors/Sonarr/wiki/Implementing-a-Torznab-indexer>
- <https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/>
- for testing: <https://fosstorrents.com/thankyou/?name=debian&cat=Installation%20-%20amd64&id=0&hybrid=0>
---
```rs
struct TorznabToolkitConfig {
auth_func: auth,
search_func: search,
whateverotherfunc: otherfunc,
port: 5309
}
example usage:
fn launch() -> Result {
rocket::build().mount("/", routes![tt::search, tt:otherfunc])
```rs
let config = /* config goes here */
fn main() -> Result {
run(config);
}
```
## Torznab spec
### Query results
Queries are returned as an RSS feed something like this:
```rss
<rss version="1.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:torznab="http://torznab.com/schemas/2015/feed">
<channel>
<atom:link rel="self" type="application/rss+xml" />
<title>Prowlarr</title>
<item>
<title>Item title</title>
<description />
<link>http://localhost:9999/</link>
<torznab:attr name="category" value="1010" />
<torznab:attr name="category" value="84570" />
<torznab:attr name="size" value="1073741824" />
<torznab:attr name="">
<enclosure url="http://localhost/" length="0" type="application/x-bittorrent" />
</item>
</channel>
</rss>
```
Item attributes:
- RSS:
- `title`: title of the thing; can maybe be empty? unsure - torznab-toolkit will treat this as required, and set description to be empty
- `description`: description; can just be empty
- `link`: a link
- Preferably specified, if not then fallback to .torrent link, then magnet url
- at least title *or* description is required; link is recommended for compatibility with some non-compliant software, like headphones (see [bitmagnet issue 349](https://github.com/bitmagnet-io/bitmagnet/issues/349))
- Torznab
- Required:
- `size`: Size in bytes
- `category`: (Sub)category id; can occur multiple times for several categories
- Recommended (by me, based off my own thoughts and NZBDrone's [Implementing a Torznab indexer](https://nzbdrone.readthedocs.io/Implementing-a-Torznab-indexer/) page):
- `seeders`: Number of seeders
- `leechers`: Number of leechers
- `peers`: Number of peers
- `infohash`: Torrent infohash
- `magneturl`: The magnet URI for the torrent
- URLs:
- Main URI can either be a magnet URI or a link to a .torrent file: `<enclosure url="http://localhost/" length="0" type="application/x-bittorrent" />`
- Length is ambiguous, so it will just be 0 (see below)
- If .torrent URL is provided, use that, if not use the magnet; also put the magnet in `magneturl`
- Rest of available attributes: <https://torznab.github.io/spec-1.3-draft/torznab/Specification-v1.3.html?highlight=server#predefined-attributes>
---
## Non-compliance and errors in the Torznab spec
- Prowlarr's built-in indexer for sites it scrapes (like 1337x) does not respect `limit`
- Negative `limit`: The spec says that if `limit` is negative, then to return `201 - Incorrect parameter`; but 201 is Created, and there's no Incorrect Parameter HTTP status code (my best guess is it should be 400?). Additionally, other indexers just ignore the limit if it's negative, so in that case, torznab-toolkit will just set the limit to the maximum to match the behavior of other widely-used indexers.
## Ambiguous behavior
- Due to ambiguous behavior, torznab-toolkit will set `length` to 0
> The correct definition of the enclosure length attribute should have been the size of the .torrent file, however newznab defined it as the length of the usenet download rather than the .nzb size.
> Therefore the length attribute can be either 0, or the .torrent/.nzb file size, or the actual media size. Given this ambiguity, services should instead provide the size extended attribute with the size of the media. (eg. <torznab:attr name=”size” value=”1460985071” />)
(from [Torznab spec - Torznab Torrent Support](https://torznab.github.io/spec-1.3-draft/revisions/1.0-Torznab-Torrent-Support.html]))

8
src/api.rs
View file

@ -8,6 +8,7 @@ use rocket::http::Status;
use rocket::response::status;
use rocket::FromForm;
use rocket::{get, response::content::RawXml};
use std::collections::HashMap;
use xml::writer::{EmitterConfig, XmlEvent};
#[derive(Debug, Clone, PartialEq, Eq, FromForm)]
@ -34,6 +35,7 @@ struct SearchForm {
}
impl SearchForm {
/// Converts it to a SearchParameters object
fn to_parameters(&self, conf: Config) -> SearchParameters {
// TODO: Clean up this code - split it into a separate function?
let mut categories: Option<Vec<u32>> = None;
@ -131,6 +133,7 @@ pub(crate) fn caps() -> status::Custom<RawXml<String>> {
#[get("/api?t=search&<form..>")]
/// The search function for the API
// FIXME: VERY incomplete also
pub(crate) fn search(form: SearchForm) -> status::Custom<RawXml<String>> {
// The compiler won't let you get a field from a struct in the Option here, since the default is None
// So this is needed
@ -151,10 +154,7 @@ pub(crate) fn search(form: SearchForm) -> status::Custom<RawXml<String>> {
match parameters.apikey {
Some(apikey) => {
if !auth(apikey).unwrap() {
return status::Custom(
Status::Unauthorized,
RawXml("401 Unauthorized".to_string()),
);
unauthorized = true;
}
}
None => {

36
src/data.rs
View file

@ -1,5 +1,7 @@
//! Contains tons of structs used by the library
use std::collections::HashMap;
use rocket::FromForm;
pub(crate) type AuthFunc = fn(String) -> Result<bool, String>;
// TODO: Figure out what the arguments should be for a search function and what it should return
@ -97,17 +99,23 @@ pub struct Tag {
#[derive(Debug, Clone, PartialEq, Eq)]
/// Holds the configuration for the capabilities of the Torznab server (used in `/api?t=caps`)
///
/// <div class="warning">Note that this library might not support all the capabilities listed in yet, so check the README before listing capabilities, or just accept that unsupported capabilities will return error 404.
/// <div class="warning">Note that this library might not support all the capabilities listed in yet, so check the README before listing capabilities, or just accept that unsupported capabilities will return error 501.
///
/// It's recommended to add any capabilities you want, and set `available` to `false` in the [`Caps`] struct for any currently unsupported search types.</div>
///
/// TODO: Add a way to partially(?) generate automatically from the Config
pub struct Caps {
pub server_info: ServerInfo,
/// The server info, like title - optional
pub server_info: Option<ServerInfo>,
/// The max and default number of items to be returned by queries - see [`Limits`]
pub limits: Limits,
/// Info about each type of search
pub searching: Vec<SearchInfo>,
/// What categories the server has - see [`Category`]
pub categories: Vec<Category>,
/// What genres the server has - see [`Genre`] (optional)
pub genres: Option<Vec<Genre>>,
/// What torrents can be tagged with - see [`Tag`] (optional)
pub tags: Option<Vec<Tag>>,
}
@ -116,12 +124,19 @@ pub struct Caps {
/// The search function (`/api?t=search`) and capabilities (`/api?t=caps` - struct [`Caps`]) are required
/// Everything else is optional
pub struct Config {
/// The function to use for a free text search
pub search: SearchFunc,
/// The auth function - if not specified, then no authorization is needed.
pub auth: Option<AuthFunc>,
/// The capabilities of the indexer - see [`Caps`]
pub caps: Caps,
/// The function to use for a tv search
pub tvsearch: Option<SearchFunc>,
/// The function to use for a movie search
pub movie: Option<SearchFunc>,
/// The function to use for a music search
pub music: Option<SearchFunc>,
/// The function to use for a book search
pub book: Option<SearchFunc>,
}
@ -142,3 +157,20 @@ pub(crate) struct SearchParameters {
/// The maximum number of items to return - also limited to whatever `limits` is in [`Caps`]
pub(crate) limit: u32,
}
#[derive(Debug, Clone, PartialEq, Eq)]
/// Holds the info for a torrent
pub struct Torrent {
title: String,
description: Option<String>,
size: u64,
categories: Vec<Category>,
seeders: Option<u32>,
leechers: Option<u32>,
peers: Option<u32>,
infohash: Option<String>,
link: Option<String>,
torrent_file_url: Option<String>,
magnet_uri: Option<String>,
other_attributes: Option<HashMap<String, String>>,
}

4
src/dummy.rs
View file

@ -48,12 +48,12 @@ pub(crate) fn create_empty_config() -> Config {
search: dummy_search_func,
auth: Some(dummy_auth_func),
caps: Caps {
server_info: ServerInfo {
server_info: Some(ServerInfo {
title: Some("Test Torznab server".to_string()),
email: Some("test@example.com".to_string()),
image: None,
version: Some("1.0".to_string()),
},
}),
limits: Limits {
max: 100,
default: 20,

23
src/lib.rs
View file

@ -1,23 +1,5 @@
#![warn(missing_docs)]
//! A toolkit for adding Torznab APIs to programs.
//!
//! Just fill in your own relevant functions and config, and
//! torznab-toolkit will run the API for you
//!
//! ```rs
//! use torznab_toolkit;
//! let config: torznab_toolkit::config::Config =
//!
//! ```
//!
//! The environment variables `ROCKET_ADDRESS` and `ROCKET_PORT` specify the address and port it will run on; these currently cannot be configured any other way. See the [relevant docs](https://rocket.rs/guide/v0.5/deploying/) for details.
//!
//! ---
//!
//! This program is brought to you by: metaphorical *and* literal truckloads of structs!
//!
//! Note: I wrote the line above when I was tired. Don't ask me what *literal* truckloads of structs means, I don't know either.
#![doc = include_str!("../README.md")]
pub mod api;
pub mod data;
mod dummy;
@ -42,3 +24,6 @@ pub async fn run(conf: data::Config) -> Result<bool, rocket::Error> {
}
}
}
/// Notes regarding the usage of torznab-toolkit and how it implements the Torznab API.
pub mod notes;

6
src/notes/implementation.rs Normal file
View file

@ -0,0 +1,6 @@
//! Notes regarding the implementation of Torznab
//!
//! - Because the behavior of `length` is ambiguous, torznab-toolkit just sets it to 0; the size is just specified by the `size` attribute
//! - See [here](https://torznab.github.io/spec-1.3-draft/revisions/1.0-Torznab-Torrent-Support.html) for details
//! - Many indexers do not have the appropriate behavior according to the spec when `limit` is negative, and that behavior doesn't even make sense; instead, it follows the behavior of other indexers, and just ignores `limit` if it's negative.
//! - If a link isn't specified for a Torrent

1
src/notes/mod.rs Normal file
View file

@ -0,0 +1 @@
pub mod usage;

8
src/notes/usage.rs Normal file
View file

@ -0,0 +1,8 @@
//! Notes regarding the usage of torznab-tooolkit
//!
//! ---
//!
//! - Please implement the `season`, `ep`, and `id` attributes for torrents when possible
//! - Implementing `id`, at least, is far out of scope of this library, and providing `season` and `ep` more effective than this library parsing for them. However, parsing for those as an optional fallback may be added later.
//! - See [here](https://torznab.github.io/spec-1.3-draft/revisions/1.0-Torznab-Torrent-Support.html) for details
// TODO: Add parsing for `season` and `ep`