[OSM-talk] Testing torrents for the planet dump

[Christian Quest]

Le 09/02/2020 à 19:21, Yuri Astrakhan a écrit :
> Christian, I would like to add torrent support to the download-osm 
> tool [1].  While I could try to scrape
> https://osm.cquest.org/torrents/ , I would obviously rather use the 
> structured xml file (or if you could provide a JSON file, even better).
>
If you have a model for such a json, I can add it.


> Proposed logic:
> * get the catalog file (xml/json)
> * download the latest torrent
> * (TBD) possibly use some magical aria2c options to stop the download 
> if it is not progressing and fallback to regular http, or possibly 
> pick a slightly older torrent (?). Suggestions are welcome -- see full 
> list of aria2c options [2]
>
> [1] 
> https://github.com/openmaptiles/openmaptiles-tools#multi-streamed-osm-data-downloader
> [2] https://aria2.github.io/manual/en/html/aria2c.html
>
I've not tested aria2c, but I tested lftp which supports bittorrent, 
example:

lftp -c torrent 
https://osm.cquest.org/torrents/planet-latest.osm.pbf.torrent

After the download is done, it switches to a background process to share 
the file up to a ratio of 2:1.


Web seeds can be seen as fallback.

I don't know if lftp (or aria2c) supports web seeds, the lftp 
documentation I found about the torrent support is minimal.

aria2c support metalink files, which could be something to also look at: 
https://en.wikipedia.org/wiki/Metalink

-- 
Christian Quest - OpenStreetMap France