[OSM-dev] C++ implementation of the API

Tom Hughes tom at compton.nu
Tue May 27 00:03:39 BST 2008


In message <20080526195726.GB3407 at lochewe.mathy.remote.org>
          Frederik Ramm <frederik at remote.org> wrote:

> > If _that_ amount of memory is needed this probably means the XML is
> > build in memory. This could be done the SAX way instead.
> 
> Looking at the code, which is publicly available, would quickly have
> told you that neither a DOM tree nor a SAX approach is used on output.
> It's plain old DIY XML.

Um... no. Looking at the code would reveal that we do indeed create
a DOM tree and then flatten it out.

That said using SAX is no help because SAX is an input method not an
output method. Streaming output is certainly possible in theory though
but the problem is that (a) the libxml streaming output interface is
not currently supported by the ruby bindings and (b) rails does not
support streaming output so we need to build the whole XML document
text in memory before we send it.

> The memory requirement stems from the fact (I believe TomH has said it
> already) that we create one million bloated Rails objects for all the
> nodes in the query and keep them all in memory while we assemble ways
> and relations.

The memory requirement is a combination of the fact that we build
activerecord objects for everything, that we hold all the records
in memory, and that we build a DOM tree.

Basically, everything that we could do wrong we do wrong. Mostly
because the rails model forces it on us.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/




More information about the dev mailing list