[OSM-dev] Memory error while converting osm to gml

Stefan de Konink stefan at konink.de
Thu Apr 2 22:31:16 BST 2009


Matt Amos wrote:
> 2009/4/2 Stefan de Konink <stefan at konink.de>:
>> Iván Sánchez Ortega wrote:
>>> El Jueves, 2 de Abril de 2009, Will Nordin escribió:
>>>> I am trying to convert the planet-090319.osm (approx 107GB) to gml. I've
>>>> tried the Java and Python solutions, but both end up thorwing OutOfMemory
>>>> exceptions.
>>> Man, it's one-frakking-hundred gigabytes, how would it *not* cause OOM
>>> errors?!
>> We call it memory mapping or a streaming operation :) Especially with
>> sax like events perfectly doable :)
> 
> except that GML describes linestrings. so you'd need to either store
> all the node locations in ram or some external indexed format... which
> is exactly what osm2pgsql does :-)

mmap the node table in array index form to disk, that will work because 
of hole allocation, that will result in a temporary file of around 5GB 
allocated space, now for any way you encounter you can do { node[2*nd] 
node[2*nd + 1] }. And get the line string upon any nd event.

> i second frederik's recommendation - import into postgres using
> osm2pgsql then export the GML from that. its a little convoluted, but
> the toolchain is well tested.

Does anyone have any timings on importing the current planet.osm in 
PostgreSQL with any tool available? Using lets say 8GB of ram and 
'typical' disk?


Stefan




More information about the dev mailing list