[OSM-talk] Somebody should offer planet file postgis dump files

[Stephen Knox]

On Thu 17th Sep 2015 at 10:51 +0100
Badita Florin <baditaflorin at gmail.com> wrote:

>Now, to import the planet it will take around 6-20 days with osmosis

>Other tools behave in the same way ...

>What i think it is missing, is a monthly postgis dump of the planet. Or
weekly, whatever

>To have loaded in postgis the whole planet, and once a
week/day/hour/minute ( what works best ) to apply just the changefiles

>The dump to be as concise as possible, having just the bare necessities,
without the index, etc

>In this way, people that are interested in doing some statistics out of
OSM can >do it in a simple way, without spending 1-2 weeks only downloading
the 29 >osm.pbf file, waiting 6 days for osmosis to load the file, if it
will not crush, , >having 1500 GB to do that, just to revert back to
200-700 GB

>I cannot say how much it will take the database, because i was never able
to >load the entire planet, all the time i had to revert to filtering,
clipping, calculate >for each separate continent, amenity or type of POI,
and then combine it all >back.

>A dump of a planet file would make the osm data more accessible to
scientist, .>researchers and the general osm community

>I would be able to try and find a server where the planet file would lay ,
with the >osmosis script that would update the script and the psql script
that will generate >once per month a dump file for the whole planet, if
somebody can help me .
>create such a setup, if possible

>Have a good day,
>Florin Badita

Does it really take this long to import a planet file these days? I would
be interested to know if anyone else has imported one recently. I may
import one myself at some point, but if it takes 6 days I probably won't
bother.

>
> On a side note, this is for a piece of work looking at using big data
tools to look at using OSM data. More info is available here
https://lists.openstreetmap.org/pipermail/dev/2015-January/028227.html.

Just as a comparative benchmark I was able to import Europe (roughly half
the size of the whole DB) into a suitable spatial form in around 4 hours
using 8 nodes with a combined 104GB of RAM (OK, I know that's a lot but
this is Big Data!). Indexing takes a little longer and I'm not sure the
querying is as quick as PostGIS but it certainly won't take you days to get
up and running.

I'll write up some more about this in due course.

Stephen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20150927/458d05f8/attachment.html>