<div dir="ltr">A similar discussion has happened recently when BitTorrent downloads were announced as an experimental feature (<a href="https://wiki.openstreetmap.org/wiki/Talk:Planet.osm#Torrent">wiki discussion</a>, <a href="https://twitter.com/OSM_Tech/status/1222507257478422535">twitter</a>, <a href="https://www.reddit.com/r/openstreetmap/comments/evmqbb/osm_operations_team_we_will_soon_be_adding/">reddit</a>, <a href="https://news.ycombinator.com/item?id=7171847">yc</a>) and the reception seems to be pretty good. My understanding is that there's a use case for projects that don't need to have a permanently updated planet but want to do a full update in a low pace frequency (say twice per year or so). Helping those projects to achieve that part using a parallel download through several mirrors (or BitTorrent) makes sense to me. The increase of bandwidth, and eventually the need for more mirrors/peers could be considered a correlation with the increase of adoption of OSM data, right?<div><br></div><div>Also, this tool is also meant to help to automate the download of data from other extract sources (geofabrik, openstreetmap fr, bbbike) for smaller areas, which is also useful in itself for projects aiming at a smaller geographical context.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 4 Feb 2020 at 01:42, Yuri Astrakhan <<a href="mailto:yuriastrakhan@gmail.com">yuriastrakhan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Andy, I agree that being frugal with bandwidth is important. Yet, there is a significant operations cost involved here, that I suspect very few will actually be willing to spend, unless it is made trivial -- the cost of setting up an independent planet file update pipeline - i.e. a docker image that could be pointed to a planet file, and has an easy way to suspend and resume update when the file needs to be static during the loading process.</div><div dir="ltr"></div><div><br></div><div>I think having an optimized download tool that can both download+validate planet/areas, and could provide other services like diff updating would solve both goals. If the current process is manually pick a mirror, manually validate, and re-download if failed, vs use a dedicated tool that will optimize the download and crash recovery, all the better. Especially if it offers us a way to add more functionality later, like patch updating.<br></div><div><br></div><div>Also, lets not fall into the premature optimization trap, as that only achieves local rather than global maximums. As a theoretical example - what is better - wider OSM adaption with higher number of planet downloads (i.e. some wasted bandwidth), or lower adaption and more optimized download process? I would think OSM project would be better off from wider adaption at a cost of a 10-50% extra bandwidth, right? If the startup cost (human hours) is lower, more people would participate. My numbers could be totally off, but keeping the eyes on bigger picture is very important when deciding such matters. Convenience/low manual labor cost is a big contributor to wider adaption.</div><div dir="ltr"><br></div><div>P.S. Am I correct to assume that every data consumer would need to download each diff file twice -- once for planet file update, and once to update the PostgreSQL database?</div><div dir="ltr"><br></div><div dir="ltr"><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 2, 2020 at 4:17 PM Andy Townsend <<a href="mailto:ajt1047@gmail.com" target="_blank">ajt1047@gmail.com</a>> wrote:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>... that's why I said "... and there are ways of keeping a <font size="+1"><b>.pbf</b></font> up to date" in my message. - the
idea is to avoid large downloads wherever possible (emphasis new
in this message; won't make it to plain text archive).</p></div></blockquote><div>Andy, </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>For more info see:</p>
<p><a href="https://docs.osmcode.org/pyosmium/latest/tools_uptodate.html" target="_blank">https://docs.osmcode.org/pyosmium/latest/tools_uptodate.html</a></p>
<p>or if that's somehow not an option:</p>
<p><a href="https://wiki.openstreetmap.org/wiki/User:EdLoach#Osmosis_to_keep_a_local_copy_of_a_.osm_file_updated" target="_blank">https://wiki.openstreetmap.org/wiki/User:EdLoach#Osmosis_to_keep_a_local_copy_of_a_.osm_file_updated</a></p>
<p>(from a few years back)<br>
</p>
<blockquote type="cite">
<div id="gmail-m_7905311216129275479gmail-m_5465949589758966533gmail-m_-3357346037135988961response_container_BBPPID" style="outline:none" dir="auto">
<div>* Automation / easy adaptation. Providing an out-of-the
box way to set up your own server is much easier if you have a
tool that automatically downloads and validates the planet
file or a portion of it, </div>
</div>
</blockquote>
<p>Sure - an automated process is much easier to follow than a
manual do-it-yourself one, but the fact that a process is
automated doesn't mean that it has to be wasteful. Ultimately,
someone's paying for the bandwidth that downloads from each of the
mirrors at
<a href="https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors" target="_blank">https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors</a>
use, so it seems only fair to not be profligate with it.</p>
<p>Best Regards,</p>
<p>Andy</p>
<p><br>
</p>
<p><br>
</p>
<div id="gmail-m_7905311216129275479gmail-m_5465949589758966533gmail-m_-3357346037135988961blackberry_signature_BBPPID" name="BB10" dir="auto"> </div>
<blockquote type="cite"></blockquote>
</div>
</blockquote></div></div>
_______________________________________________<br>
talk mailing list<br>
<a href="mailto:talk@openstreetmap.org" target="_blank">talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/talk</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Jorge Sanz<br><a href="http://twitter.com/xurxosanz" target="_blank">http://twitter.com/xurxosanz</a><br><a href="http://jorgesanz.net" target="_blank">http://jorgesanz.net</a></div>