[OSM-dev] pbf2osm development has started [code to test!]

Scott Crosby scrosby at cs.rice.edu
Thu Sep 30 11:48:54 BST 2010

On Thu, Sep 30, 2010 at 4:44 AM, Stefan de Konink <stefan at konink.de> wrote:

> >
> >     - allow the use of the bbox
> >
> >
> > What exactly do you mean by this: ?
> Filter output by applying a bbox (the expensive way). Extract nodes
> within a bbox, sort them on ID, extract all ways, validate per item if
> any of the nd's are selected. Append remaining nodes to a miss list.
> Same operation for relations. Run lookup for missing ways, fetch missing
> node list.
I proposed exactly this for the mkgmap splitter. If you're going to do this,
can I propose a tweak where you can output thousands of files
simultaneously? The differences are minor: Instead of tracking if a
node/way/relation was output or was missed with two bitsets, track which
areas it has been dumped to with two multimaps from the ID to the list of
areas it was output to or the list of areas in which it was missed.

As the typical keycount in the multimap is low, you can use the trick I used
a few weeks ago in the splitter: build a multimap by layering a set of
individual hash tables, one for the first value of each key, a second for
the second value of each key (if any), etc. Use sparse hash tables (
http://code.google.com/p/google-sparsehash/) for the first few layers, and
std::map for the remaining layers.

> >      + based on the index
> The 'open index' that is not implemented should be implemented.

Ok. What kinds of things might we want to index?


Count of nodes/ways/relations in that block?

What else?

> > And this: ?
> >
> >
> >      + based on geos like solution
> Allow the user to make non-rectangle selections.
> >     My personal interest is going for output that support output that
> >     can be used to 'copy into' data into a database. And extend the
> >     current protocol for diff support. (Hence: replacing osmsucker-ykw)
> >
> >
> > What kinds of extensions are you thinking of?
> What we discussed {create,modify,delete}.

Ok, Is XML's gzipped size or parsing speed a bottleneck for storing or
processing changes? I'd be happy to offer suggestions on the protocol buffer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100930/b72b299d/attachment-0001.html>

More information about the dev mailing list