[OSM-dev] [OSM-talk] donating read-only api-mirrors

[Matthias Julius]

Frederik Ramm <frederik at remote.org> writes:

> Hi,
>
> Marcus Wolschon wrote:
>> So...why is it that you hold the result-set of the nodes-query in memory
>> again?
>
> While not mandated by the XML structure, the XML document we return is 
> usually sorted - nodes, ways, relations, each in ascending ID order. 
> Because a way or relation may require additional nodes to be included 
> that lie outside of the bounding box, we can only return the nodes after 
> we have processed ways and relations.

The API does not return relation members that are completely outside
the bbox.  Once the ways and nodes are processed they could be
streamed out.

Is anyone relying on the ascending ID order?  If not there could be a
mixed approach: 
 - fetch and stream all nodes in the bbox keeping the IDs
 - fetch all ways and store them in memory
 - fetch and stream all nodes referenced by the ways and not yet
   fetched, keeping the IDs
 - stream the ways, keeping the IDs
 - fetch and stream all relations

To be able to send proper error codes to the client when something
goes wrong halfway down the path the data could be cached on disk and
sent out once it is complete.

Of course, if you have to coerce rails into doing the above that
remains as an issue.

Matthias