[OSM-dev] Indexing of PBF files

[Florian Lohoff]

On Sat, Feb 12, 2022 at 11:30:19PM +0100, codesoap--- via dev wrote:
> Hi,
> 
> Nick Stallman wrote:
> > Adding an index [to PBF files] seems like a logical step which would
> > reduce processing times for many common operations drastically.
> 
> I was just thinkgin about the same thing and was wondering if there is
> any progress on this, now that another 3+ years have passed. I haven't
> found any other resources on the topic than this E-Mail; has there been
> some discussion elsewhere, that I didn't find?
> 
> I'm sad to see you only received pushback against your proposal. IMO
> the benefits of an index would far outweigh the drawback of a few
> bytes of indexdata for each BlobHeader. I think such an index could
> lay the foundation for a whole new kind of applications; for example:
> applications that run on the end-user's hardware, are easy to setup and
> use, yet performant and with low disk usage. Applications like this are
> currently just not feasible.

AFAIU the pbf format its not spatially indexed at all. So you have
objects based on the osm id sorted in the blobs. 

Basically you not only need an index but rather "resort" the objects
from beeing organized by osm_id to beeing organized spatially,
and then build and rtree of the bboxes of the protbuf blobs.

There have been applications in the past which did exactly this, not
based on PBF. When we ran tiles at home aka osmarender in the past there
had been one API implementation for the "bbox" call which allows reading
data in the bounding box. 

Look for "TRAPI" which was a perl implementation of a spatially
organized data store for osm data.

https://wiki.openstreetmap.org/wiki/Trapi

Flo
-- 
Florian Lohoff                                                     f at zz.de
  Any sufficiently advanced technology is indistinguishable from magic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20220213/5d84ed5f/attachment.sig>