[OSM-dev] pbf2osm development has started [code to test!]

Scott Crosby scrosby at cs.rice.edu
Thu Sep 30 05:30:51 BST 2010


>
>
> But what I already discussed with Scott, we *need* a good 'has everything'
> PBF file. Something that can test a parser and has expected output.
>
>
I know. I know. I need to hack a custom program that generates the entire
protocol buffer, serializes it, hitting all of the interesting features and
corner cases. The resulting file is the test-case. Doing that requires
writing code that sets every field manually and it may be one or more weeks
until I have a chance to do it.


> For example Osmosis only generates dense nodes, not the 'other notation'.
>
>
Actually, it can also generate normal nodes with a command line flag.


> Also Osmosis currently implements deflate, not bzip2/lzma. (At the time of
> writing I implemented bzip2 as well, but couldn't test it.)
>
>
If I can find a lzma/bzip2 java code, its ready to plug them in. Although,
I'm thinking that with the cost of bzip2 decompression, the only interesting
case is lzma.


>
> There are a few things on the todo before going for the osm2pbf myself:
> - xml escaping
> - mmap input
> - lzma
>

The current planet XML format doesn't include any form of index. I think you
could call osm2pbf complete, and offering significant advantages, without
support for any indexing data structure at all.

I would be happy to help in the design and algorithms of a geometric index,
but I don't have the time to program it. Are you interested in coding this?
If done properly, geometric sorting code can do much much more than make a
planet.osm.pbf indexable.

They can make very cheap tileservers. PrimitiveBlocks are independently
decodable blobs; they can be stored in a database. Given a bbox query,
select out all of the blobs that intersect it, concatenate them together,
add a header, and thats a conforming *.osm.pbf file. Depending on the
precision used and whether metadata is kept, that entire database can be
cached into 6-10GB of RAM.

I would like to see this happen.

This may be a new way to distribute the planet: A set of PrimitiveBlock
tiles sitting in an Sqlite database..... Actually, now that I think of it,
this may be the superior approach compared to using my hooks to hack an
index into the *.osm.pbf format.

- allow the use of the bbox
>

What exactly do you mean by this: ?


>  + based on the index
>

And this: ?


>  + based on geos like solution
>
>
Roeland wanted to go for a library so other tools could call getNextNode()
> etc. without fiddling with the internal structures.
>
> My personal interest is going for output that support output that can be
> used to 'copy into' data into a database. And extend the current protocol
> for diff support. (Hence: replacing osmsucker-ykw)
>

What kinds of extensions are you thinking of?

Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100929/1611bdfd/attachment.html>


More information about the dev mailing list