[OSM-dev] Another benchmark (was: XAPI and other solutions)

[Oliver Tonnhofer]

On 29.04.2011, at 14:46, Pierre-Alain Dorange wrote:
> 4th file XML : 1680 MB (1.64 GB)
>        ET (DOM) : n/a
>        ET (iterparse) :  n/a
>        xml.sax : 365 seconds (require about 50 MB), 20 Knodes/sec
> 
> Note : 
> - ET (DOM) broke, not enough memory
> - ET (iterparse) also use all memory (don't understand why, must be a
> bug in my code)


iterparse still builds the (Element)Tree. This makes working with it more convenient but as you noticed the memory grows. You need to clear() each element and for very flat XML structures (like OSM) also the root element. See: http://effbot.org/zone/element-iterparse.htm#incremental-parsing

Can you make another test with that or can you post your xml.sax example? I really like to see how they compare. You can see here http://pastie.org/1843106 how to get the root of iterparse and how to use clear.


Regards,
Oliver

-- 
Oliver Tonnhofer    | Omniscale GmbH & Co KG    | http://omniscale.de
http://mapproxy.org | https://bitbucket.org/olt | @oltonn