[OSM-dev] imposm.parser - New Python library for OSM XML and PBF files

Oliver Tonnhofer olt at omniscale.de
Tue Feb 22 11:48:08 GMT 2011


On 22.02.2011, at 12:12, Jochen Topf wrote:
>> It uses C extensions for performance critical parts and automatically
>> distributes the parsing across multiple CPUs/cores (for PBF and XML!).
> 
> Interesting! How do you do the parallel parsing?

For XML? It reads the (uncompressed) XML line by line and writes these lines into mmap'ed files. It splits the files after closing node/way/relation tags. It's not a generic parallel XML parser but it works great for the OSM data. 

https://bitbucket.org/olt/imposm.parser/src/c7e1cd122f00/imposm/parser/xml/multiproc.py#cl-167

>> As the name already suggests, imposm.parser is only a sub-package of a larger
>> project: Imposm. It is an importer for OSM data that we are going to release
>> in a couple of weeks.
> 
> Importer into what?


PostGIS for now, but it could be extended to support SQLite, etc. It is similar to osm2pgsql but it supports more customizable database schemas. For example, separate tables for buildings and landusage, or motorways and minorways. It can also create generalized tables (with lower spatial resolution) and union views. It uses TokyoCabinet to cache the nodes for ways and relations building (you might remember that from Barcelona?). 

Regards,
Oliver

-- 
Oliver Tonnhofer    | Omniscale GmbH & Co KG    | http://omniscale.de
http://mapproxy.org | https://bitbucket.org/olt






More information about the dev mailing list