[Tile-serving] [openstreetmap/osm2pgsql] Flex output returns PBF error: invalid BlobHeader size (#1348)

[Ryan Lambert]

I am attempting to use the new flex output and am running into this error:

```
ERROR: PBF error: invalid BlobHeader size (> max_blob_header_size)
```

Version used.

```
osm2pgsql --version
2020-11-30 20:50:40  osm2pgsql version 1.3.0 (1.3.0-162-g0a23889)
Compiled using the following library versions:
Libosmium 2.15.6
Proj [API 4] Rel. 6.3.1, February 10th, 2020
Lua 5.3.3
```

Downloaded data from geofabrik and verified the md5sum.  The PBF and MD5 files are both in this zip file: 
[osm-dc-20201130.zip](https://github.com/openstreetmap/osm2pgsql/files/5620219/osm-dc-20201130.zip)


```
wget https://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.pbf
wget https://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.pbf.md5
```


First attempt fails, suggests using `-r`.

```
osm2pgsql -s ~/git/osm2pgsql/flex-config/generic.lua  -d pgosm_flex  ./district-of-columbia-latest.osm.pbf 
2020-11-30 20:53:22  osm2pgsql version 1.3.0 (1.3.0-162-g0a23889)
2020-11-30 20:53:22  Database version: 13.1 (Ubuntu 13.1-1.pgdg20.04+1)
2020-11-30 20:53:22  ERROR: Cannot detect file format. Try using -r.
```


Add the `-r pbf`, now getting the invalid BlogHeader size error.

```
osm2pgsql -s ~/git/osm2pgsql/flex-config/simple.lua -r pbf -d pgosm_flex  ./district-of-columbia-latest.osm.pbf
2020-11-30 20:37:03  osm2pgsql version 1.3.0 (1.3.0-162-g0a23889)
2020-11-30 20:37:03  Database version: 13.1 (Ubuntu 13.1-1.pgdg20.04+1)
2020-11-30 20:37:03  Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
2020-11-30 20:37:03  Setting up table: planet_osm_point
2020-11-30 20:37:03  Setting up table: planet_osm_line
2020-11-30 20:37:03  Setting up table: planet_osm_polygon
2020-11-30 20:37:03  Setting up table: planet_osm_roads
2020-11-30 20:37:03  node cache: stored: 0(0.00%), storage efficiency: 0.00% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
2020-11-30 20:37:03  ERROR: PBF error: invalid BlobHeader size (> max_blob_header_size)
```

I attempted the flex import with these with files from `flex-config/` all with same error:

* compatible.lua
* generic.lua
* simple.lua
* data-types.lua


I was able to load the file using my normal command.  I did not use `-r pbf` here and did not have an error.

```
osm2pgsql --create --slim --drop --hstore --multi-geometry -d pgosm2 district-of-columbia-latest.osm.pbf
2020-11-30 20:36:36  osm2pgsql version 1.3.0 (1.3.0-162-g0a23889)
2020-11-30 20:36:36  Database version: 13.1 (Ubuntu 13.1-1.pgdg20.04+1)
2020-11-30 20:36:36  Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
2020-11-30 20:36:36  Setting up table: planet_osm_point
2020-11-30 20:36:36  Setting up table: planet_osm_line
2020-11-30 20:36:36  Setting up table: planet_osm_polygon
2020-11-30 20:36:36  Setting up table: planet_osm_roads
Processing: Node(1725k 1725.0k/s) Way(212k 106.00k/s) Relation(3914 3914.0/s)  parse time: 4s
2020-11-30 20:36:41  Node stats: total(1725572), max(8181118446) in 1s
2020-11-30 20:36:41  Way stats: total(212069), max(879374048) in 2s
2020-11-30 20:36:41  Relation stats: total(3914), max(11961674) in 1s
2020-11-30 20:36:41  Dropping table: planet_osm_nodes
2020-11-30 20:36:41  Done postprocessing table: planet_osm_nodes in 0s
2020-11-30 20:36:41  Dropping table: planet_osm_ways
2020-11-30 20:36:41  Done postprocessing table: planet_osm_ways in 0s
2020-11-30 20:36:41  Dropping table: planet_osm_rels
2020-11-30 20:36:41  Done postprocessing table: planet_osm_rels in 0s
2020-11-30 20:36:41  Sorting data and creating indexes for planet_osm_line
2020-11-30 20:36:41  Sorting data and creating indexes for planet_osm_roads
2020-11-30 20:36:41  Sorting data and creating indexes for planet_osm_polygon
2020-11-30 20:36:41  Sorting data and creating indexes for planet_osm_point
2020-11-30 20:36:41  Creating geometry index on planet_osm_roads
2020-11-30 20:36:41  Analyzing table 'planet_osm_roads'...
2020-11-30 20:36:41  All indexes on planet_osm_roads created in 0s
2020-11-30 20:36:41  Creating geometry index on planet_osm_point
2020-11-30 20:36:41  Creating geometry index on planet_osm_line
2020-11-30 20:36:41  Analyzing table 'planet_osm_point'...
2020-11-30 20:36:41  Analyzing table 'planet_osm_line'...
2020-11-30 20:36:41  All indexes on planet_osm_point created in 0s
2020-11-30 20:36:41  All indexes on planet_osm_line created in 0s
2020-11-30 20:36:42  Creating geometry index on planet_osm_polygon
2020-11-30 20:36:43  Analyzing table 'planet_osm_polygon'...
2020-11-30 20:36:43  All indexes on planet_osm_polygon created in 2s
2020-11-30 20:36:43  Osm2pgsql took 7s overall
2020-11-30 20:36:43  node cache: stored: 1725572(100.00%), storage efficiency: 51.37% (dense blocks: 50, sparse nodes: 1474787), hit rate: 100.00%
```

I double checked the downloaded PBF file appears valid, and repeated the test with Kansas and Colorado files.  In each case the import using flex failed where the old format works normally.

Let me know what other details I can provide!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1348
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20201130/12bdd22a/attachment-0001.htm>