[OSM-dev] Another benchmark

[Pierre-Alain Dorange]

Oliver Tonnhofer <olt at omniscale.de> wrote:

> > 4th file XML : 1680 MB (1.64 GB)
> >        ET (DOM) : n/a
> >        ET (iterparse) :  n/a
> >        xml.sax : 365 seconds (require about 50 MB), 20 Knodes/sec
> > 
> > Note : 
> > - ET (DOM) broke, not enough memory
> > - ET (iterparse) also use all memory (don't understand why, must be a
> > bug in my code)

> 
> iterparse still builds the (Element)Tree. This makes working with it more
> convenient but as you noticed the memory grows. You need to clear() each
> element and for very flat XML structures (like OSM) also the root element.
> See: http://effbot.org/zone/element-iterparse.htm#incremental-parsing

I used elem.clear (that's why the 3rd example do not use memory).
For the 4th one (very large) i think i miss to clear the root.

> Can you make another test with that or can you post your xml.sax example?
> I really like to see how they compare. You can see here
> http://pastie.org/1843106 how to get the root of iterparse and how to use
> clear.

I'm making so tests and debug, and i'll post the final code tomorrow.

The actual code :
https://github.com/padorange/osm-xml-tests

-- 
Pierre-Alain Dorange
OSM experiences : <http://www.leretourdelautruche.com/map/>