[OSM-dev] Reducing osm2pgsql memory usage using a database method

[Joerg Ostertag (OSM Munich/Germany)]

On Saturday 10 March 2007 22:52, Frederik Ramm wrote:
> Hi,
>
> > 1. We can re-write osm2pgsql in c++ and take advantage of dynamic
> > structures e.g std::map, safe formatting and casting
> > boost::lexiacal_cast,  boost::format and more.

I think it would be great if we just could precalculate the planet.osm to a 
geo-sql format and put it on the server for download. If we do this 
automagically after creation of planet.osm it would ease things a lot for 
many people.

> Hate to spoil your fun & don't want to start an advocacy debate here &
> am a C++ user myself BUT (1) hash tables etc. are readily available in
> plain C (glib), and (2) if you need flexibility why don't you do the
> importer in Perl?

There are several perl interface to osm, but working with planet.osm in perl 
 - takes 40 minutes only for reading 
 - takes 100Bytes of memory for each node stored in memory. 
   Which is more than 10 times more than really needed.

And if you try to use something like a tied has ...
 - It only will take 10 hours to complete reading only

Well this doesn't seem to be the right way into the future.
Everything i tried up to now either would need >4GB RAM in the future, or is 
painfully slow.
Oh the values above are with a Hyperthreading 3GHz 2BG RAM Machine.
This is why for GPSDrive i currently tend to precalculate the SQL stuff and 
make it available on the Server and tend to use a C Program in the long term.

-- 
Jörg (Germany, Munich)

http://www.ostertag.name/
TeamSpeak2: ts2.ostertag.name, user: tweety, Channel: "GPS Drive"
irc://irc.oftc.net/#osm
Tel.: +49 89 420950304