[OSM-dev] planet.openstreetmap.org

[Gabriel Ebner]

On Fri, Aug 18, 2006 at 01:40:23PM +0200, Nicola Ranaldo wrote:
> Umh... this is annoying, i spent about 3 hours with valgrind, my problem the 
> expat library i use to parse data, and when it encounters that it buffers all 
> the subsequent line in ram, ignoring the other elements, on a big file like 
> the last dump my system swaps and goes out of memory :)

Nope, expat should be able to parse with contant memory usage.  Are you using
any bindings, building any XML trees or storing the data somewhere (just
storing the nodes of the last dump in a simple array takes 200M, and that's
nowhere near a hash table).  If you want to check how long expat (just the
parser) takes to parse the planet on your machine, run:

  $ time xmlwf -r -t planet-060818.osm
  xmlwf -r -t planet-060818.osm  102,63s user 11,84s system 78% cpu 2:25,23
  total

(You need the -r because mmaping files > 2G doesn't seem to work -- on
linux-2.6.17, at least.)

> I do not know if i have to double check expat docs... 
> Has anyone the same problem?

Over here expat works just fine, but trying to store all the nodes and
segments in memory for further processing takes more than 6G of heap memory..

  Gabriel.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20060818/6e7c2310/attachment.pgp>