[Tile-serving] [osm2pgsql] Initial libosmium port (#379)

Sarah Hoffmann notifications at github.com
Thu Jul 9 20:33:52 UTC 2015 

This introduces libosmium as another parser back-end, replacing the pbf and xml parser. The o5m parser back-end remains for the time being and should still be usable. Once o5m support has been added, the libosmium integration can be taken a bit further and the custom OSM data structures should be replaced with libosmium data structures which would avoid quite a bit of copy/malloc.

I've copied the libosmium sources because none of the distros provide a usable version and won't for some time. Maybe somebody has a better idea how to handle that.

Other notable changes include:

- refactor of parse.hpp/cpp
- retire UTFsanitizer and remove sanitize option
- make C++11 compiler mandatory (required by libosmium)
- update pbf proto descriptions
- resolves #213 by sending all data as 'modify' in append mode (going back to the previous behaviour of the C version)

CC @joto 
You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/osm2pgsql/pull/379

-- Commit Summary --

  * replace pbf and xml parser with libosmium
  * correctly parse change files
  * forward input-reader option to libosmium as format parameter

-- File Changes --

    M Makefile.am (50)
    M README.md (8)
    D UTF8sanitizer.cpp (160)
    M configure.ac (44)
    A contrib/libosmium/LICENSE.txt (23)
    A contrib/libosmium/README.contrib (2)
    A contrib/libosmium/README.md (104)
    A contrib/libosmium/osmium/area/assembler.hpp (783)
    A contrib/libosmium/osmium/area/detail/node_ref_segment.hpp (262)
    A contrib/libosmium/osmium/area/detail/proto_ring.hpp (274)
    A contrib/libosmium/osmium/area/detail/segment_list.hpp (216)
    A contrib/libosmium/osmium/area/multipolygon_collector.hpp (218)
    A contrib/libosmium/osmium/area/problem_reporter.hpp (149)
    A contrib/libosmium/osmium/area/problem_reporter_exception.hpp (96)
    A contrib/libosmium/osmium/area/problem_reporter_ogr.hpp (223)
    A contrib/libosmium/osmium/area/problem_reporter_stream.hpp (96)
    A contrib/libosmium/osmium/builder/builder.hpp (223)
    A contrib/libosmium/osmium/builder/builder_helper.hpp (103)
    A contrib/libosmium/osmium/builder/osm_object_builder.hpp (284)
    A contrib/libosmium/osmium/diff_handler.hpp (67)
    A contrib/libosmium/osmium/diff_iterator.hpp (129)
    A contrib/libosmium/osmium/diff_visitor.hpp (104)
    A contrib/libosmium/osmium/dynamic_handler.hpp (195)
    A contrib/libosmium/osmium/experimental/flex_reader.hpp (129)
    A contrib/libosmium/osmium/geom/coordinates.hpp (97)
    A contrib/libosmium/osmium/geom/factory.hpp (418)
    A contrib/libosmium/osmium/geom/geojson.hpp (154)
    A contrib/libosmium/osmium/geom/geos.hpp (231)
    A contrib/libosmium/osmium/geom/haversine.hpp (94)
    A contrib/libosmium/osmium/geom/mercator_projection.hpp (110)
    A contrib/libosmium/osmium/geom/ogr.hpp (206)
    A contrib/libosmium/osmium/geom/projection.hpp (167)
    A contrib/libosmium/osmium/geom/rapid_geojson.hpp (190)
    A contrib/libosmium/osmium/geom/relations.hpp (57)
    A contrib/libosmium/osmium/geom/tile.hpp (101)
    A contrib/libosmium/osmium/geom/util.hpp (75)
    A contrib/libosmium/osmium/geom/wkb.hpp (277)
    A contrib/libosmium/osmium/geom/wkt.hpp (150)
    A contrib/libosmium/osmium/handler.hpp (101)
    A contrib/libosmium/osmium/handler/chain.hpp (128)
    A contrib/libosmium/osmium/handler/disk_store.hpp (111)
    A contrib/libosmium/osmium/handler/dump.hpp (294)
    A contrib/libosmium/osmium/handler/node_locations_for_ways.hpp (180)
    A contrib/libosmium/osmium/handler/object_relations.hpp (106)
    A contrib/libosmium/osmium/index/bool_vector.hpp (83)
    A contrib/libosmium/osmium/index/detail/create_map_with_fd.hpp (73)
    A contrib/libosmium/osmium/index/detail/mmap_vector_anon.hpp (65)
    A contrib/libosmium/osmium/index/detail/mmap_vector_base.hpp (180)
    A contrib/libosmium/osmium/index/detail/mmap_vector_file.hpp (71)
    A contrib/libosmium/osmium/index/detail/tmpfile.hpp (62)
    A contrib/libosmium/osmium/index/detail/vector_map.hpp (246)
    A contrib/libosmium/osmium/index/detail/vector_multimap.hpp (186)
    A contrib/libosmium/osmium/index/index.hpp (100)
    A contrib/libosmium/osmium/index/map.hpp (266)
    A contrib/libosmium/osmium/index/map/all.hpp (46)
    A contrib/libosmium/osmium/index/map/dense_file_array.hpp (67)
    A contrib/libosmium/osmium/index/map/dense_mem_array.hpp (57)
    A contrib/libosmium/osmium/index/map/dense_mmap_array.hpp (60)
    A contrib/libosmium/osmium/index/map/dummy.hpp (88)
    A contrib/libosmium/osmium/index/map/sparse_file_array.hpp (67)
    A contrib/libosmium/osmium/index/map/sparse_mem_array.hpp (60)
    A contrib/libosmium/osmium/index/map/sparse_mem_map.hpp (115)
    A contrib/libosmium/osmium/index/map/sparse_mem_table.hpp (146)
    A contrib/libosmium/osmium/index/map/sparse_mmap_array.hpp (60)
    A contrib/libosmium/osmium/index/multimap.hpp (127)
    A contrib/libosmium/osmium/index/multimap/all.hpp (41)
    A contrib/libosmium/osmium/index/multimap/hybrid.hpp (200)
    A contrib/libosmium/osmium/index/multimap/sparse_file_array.hpp (54)
    A contrib/libosmium/osmium/index/multimap/sparse_mem_array.hpp (58)
    A contrib/libosmium/osmium/index/multimap/sparse_mem_multimap.hpp (151)
    A contrib/libosmium/osmium/index/multimap/sparse_mmap_array.hpp (58)
    A contrib/libosmium/osmium/index/node_locations_map.hpp (70)
    A contrib/libosmium/osmium/io/any_compression.hpp (48)
    A contrib/libosmium/osmium/io/any_input.hpp (51)
    A contrib/libosmium/osmium/io/any_output.hpp (52)
    A contrib/libosmium/osmium/io/bzip2_compression.hpp (289)
    A contrib/libosmium/osmium/io/compression.hpp (281)
    A contrib/libosmium/osmium/io/detail/input_format.hpp (158)
    A contrib/libosmium/osmium/io/detail/opl_output_format.hpp (317)
    A contrib/libosmium/osmium/io/detail/output_format.hpp (156)
    A contrib/libosmium/osmium/io/detail/pbf.hpp (67)
    A contrib/libosmium/osmium/io/detail/pbf_input_format.hpp (263)
    A contrib/libosmium/osmium/io/detail/pbf_output_format.hpp (946)
    A contrib/libosmium/osmium/io/detail/pbf_parser.hpp (456)
    A contrib/libosmium/osmium/io/detail/pbf_stringtable.hpp (218)
    A contrib/libosmium/osmium/io/detail/pbf_type_conv.hpp (73)
    A contrib/libosmium/osmium/io/detail/read_thread.hpp (106)
    A contrib/libosmium/osmium/io/detail/read_write.hpp (160)
    A contrib/libosmium/osmium/io/detail/write_thread.hpp (86)
    A contrib/libosmium/osmium/io/detail/xml_input_format.hpp (737)
    A contrib/libosmium/osmium/io/detail/xml_output_format.hpp (484)
    A contrib/libosmium/osmium/io/detail/zlib.hpp (113)
    A contrib/libosmium/osmium/io/error.hpp (58)
    A contrib/libosmium/osmium/io/file.hpp (344)
    A contrib/libosmium/osmium/io/file_compression.hpp (72)
    A contrib/libosmium/osmium/io/file_format.hpp (78)
    A contrib/libosmium/osmium/io/gzip_compression.hpp (246)
    A contrib/libosmium/osmium/io/header.hpp (122)
    A contrib/libosmium/osmium/io/input_iterator.hpp (140)
    A contrib/libosmium/osmium/io/opl_output.hpp (39)
    A contrib/libosmium/osmium/io/output_iterator.hpp (124)
    A contrib/libosmium/osmium/io/overwrite.hpp (52)
    A contrib/libosmium/osmium/io/pbf_input.hpp (49)
    A contrib/libosmium/osmium/io/pbf_output.hpp (49)
    A contrib/libosmium/osmium/io/reader.hpp (312)
    A contrib/libosmium/osmium/io/reader_iterator.hpp (51)
    A contrib/libosmium/osmium/io/writer.hpp (145)
    A contrib/libosmium/osmium/io/xml_input.hpp (48)
    A contrib/libosmium/osmium/io/xml_output.hpp (47)
    A contrib/libosmium/osmium/memory/buffer.hpp (553)
    A contrib/libosmium/osmium/memory/collection.hpp (154)
    A contrib/libosmium/osmium/memory/item.hpp (178)
    A contrib/libosmium/osmium/memory/item_iterator.hpp (234)
    A contrib/libosmium/osmium/object_pointer_collection.hpp (112)
    A contrib/libosmium/osmium/osm.hpp (48)
    A contrib/libosmium/osmium/osm/area.hpp (214)
    A contrib/libosmium/osmium/osm/box.hpp (250)
    A contrib/libosmium/osmium/osm/changeset.hpp (336)
    A contrib/libosmium/osmium/osm/diff_object.hpp (183)
    A contrib/libosmium/osmium/osm/entity.hpp (79)
    A contrib/libosmium/osmium/osm/entity_bits.hpp (105)
    A contrib/libosmium/osmium/osm/item_type.hpp (193)
    A contrib/libosmium/osmium/osm/location.hpp (285)
    A contrib/libosmium/osmium/osm/node.hpp (76)
    A contrib/libosmium/osmium/osm/node_ref.hpp (173)
    A contrib/libosmium/osmium/osm/node_ref_list.hpp (184)
    A contrib/libosmium/osmium/osm/object.hpp (438)
    A contrib/libosmium/osmium/osm/object_comparisons.hpp (110)
    A contrib/libosmium/osmium/osm/relation.hpp (194)
    A contrib/libosmium/osmium/osm/segment.hpp (105)
    A contrib/libosmium/osmium/osm/tag.hpp (140)
    A contrib/libosmium/osmium/osm/timestamp.hpp (186)
    A contrib/libosmium/osmium/osm/types.hpp (62)
    A contrib/libosmium/osmium/osm/types_from_string.hpp (116)
    A contrib/libosmium/osmium/osm/undirected_segment.hpp (100)
    A contrib/libosmium/osmium/osm/way.hpp (117)
    A contrib/libosmium/osmium/relations/collector.hpp (544)
    A contrib/libosmium/osmium/relations/detail/member_meta.hpp (158)
    A contrib/libosmium/osmium/relations/detail/relation_meta.hpp (136)
    A contrib/libosmium/osmium/tags/filter.hpp (162)
    A contrib/libosmium/osmium/tags/regex_filter.hpp (58)
    A contrib/libosmium/osmium/tags/taglist.hpp (67)
    A contrib/libosmium/osmium/thread/function_wrapper.hpp (111)
    A contrib/libosmium/osmium/thread/pool.hpp (181)
    A contrib/libosmium/osmium/thread/queue.hpp (194)
    A contrib/libosmium/osmium/thread/sorted_queue.hpp (159)
    A contrib/libosmium/osmium/thread/util.hpp (87)
    A contrib/libosmium/osmium/util/cast.hpp (103)
    A contrib/libosmium/osmium/util/compatibility.hpp (47)
    A contrib/libosmium/osmium/util/config.hpp (72)
    A contrib/libosmium/osmium/util/data_file.hpp (192)
    A contrib/libosmium/osmium/util/delta.hpp (97)
    A contrib/libosmium/osmium/util/double.hpp (93)
    A contrib/libosmium/osmium/util/file.hpp (118)
    A contrib/libosmium/osmium/util/memory_mapping.hpp (723)
    A contrib/libosmium/osmium/util/minmax.hpp (123)
    A contrib/libosmium/osmium/util/options.hpp (155)
    A contrib/libosmium/osmium/util/string.hpp (102)
    A contrib/libosmium/osmium/util/verbose_output.hpp (139)
    A contrib/libosmium/osmium/visitor.hpp (255)
    A contrib/protobuf/fileformat.proto (54)
    R contrib/protobuf/osmformat.proto (82)
    A contrib/protobuf/osmpbf/osmpbf.h (26)
    M docs/osm2pgsql.1 (9)
    M docs/usage.md (2)
    D input.cpp (226)
    D input.hpp (14)
    A m4/ax_lib_expat.m4 (283)
    D m4/ax_lib_protobuf_c.m4 (257)
    D m4/ax_lib_xml2.m4 (150)
    M options.cpp (16)
    M options.hpp (1)
    M osm2pgsql.cpp (13)
    M osmtypes.hpp (4)
    M parse-o5m.cpp (73)
    M parse-o5m.hpp (8)
    A parse-osmium.cpp (0)
    A parse-osmium.hpp (0)
    D parse-pbf.cpp (0)
    D parse-pbf.hpp (0)
    D parse-xml2.cpp (0)
    D parse-xml2.hpp (0)
    I parse.cpp (0)
    I parse.hpp (0)
    D protobuf/fileformat.proto (0)
    I reprojection.cpp (0)
    I reprojection.hpp (0)
    D sanitizer.hpp (0)
    I tests/regression-test.py (0)
    I tests/test-output-multi-line-storage.cpp (0)
    I tests/test-output-multi-line.cpp (0)
    I tests/test-output-multi-point-multi-table.cpp (0)
    I tests/test-output-multi-point.cpp (0)
    I tests/test-output-multi-poly-trivial.cpp (0)
    I tests/test-output-multi-polygon.cpp (0)
    I tests/test-output-multi-tags.cpp (0)
    I tests/test-output-pgsql-tablespace.cpp (0)
    I tests/test-output-pgsql.cpp (0)
    I tests/test-parse-options.cpp (0)
    I tests/test-parse-xml2.cpp (0)

-- Patch Links --

https://github.com/openstreetmap/osm2pgsql/pull/379.patch
https://github.com/openstreetmap/osm2pgsql/pull/379.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/379
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150709/3d58fdea/attachment-0001.html>

More information about the Tile-serving mailing list