[Tile-serving] [openstreetmap/osm2pgsql] bad result during COPY (#159)

Fri Sep 27 11:12:48 UTC 2019

Found this issue googling for `"ERROR: unexpected message type 0x58 during COPY from stdin"`.

I also experiencing some issue with this error.

osm2pgsql logs:

```
osm2pgsql version 0.96.0 (64 bit id space)

Using built-in tag processing pipeline
Using projection SRS 4326 (Latlong)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=3

Reading in file: /tmp/tmp97xxauxj
Using XML parser.
  parse time: 0s
Node stats: total(3), max(428204965) in 0s
Way stats: total(0), max(0) in 0s
Relation stats: total(0), max(0) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
Using built-in tag processing pipeline
node cache: stored: 3(100.00%), storage efficiency: 0.04% (dense blocks: 1, sparse nodes: 0), hit rate: -nan%
Osm2pgsql failed due to ERROR: SET synchronous_commit TO off; failed: ERROR:  query_wait_timeout
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
```

PostgreSQL logs:

```
CONTEXT:  COPY planet_osm_point, line 1
STATEMENT:  COPY planet_osm_point (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","capital","construction","covered","culvert","cutting","denomination","disused","ele","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tunnel","water","waterway","wetland","width","wood","z_order",tags,way) FROM STDIN
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_line, line 1
STATEMENT:  COPY planet_osm_line (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_polygon, line 1
STATEMENT:  COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_point, line 1
STATEMENT:  COPY planet_osm_point (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","capital","construction","covered","culvert","cutting","denomination","disused","ele","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tunnel","water","waterway","wetland","width","wood","z_order",tags,way) FROM STDIN
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_line, line 1
STATEMENT:  COPY planet_osm_line (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_polygon, line 1
STATEMENT:  COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
LOG:  could not send data to client: Broken pipe
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_roads, line 1
STATEMENT:  COPY planet_osm_roads (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
FATAL:  connection to client lost
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
ERROR:  unexpected message type 0x58 during COPY from stdin
...
```

Full log here: https://gist.github.com/and-semakin/46270e9289f1bb46a39bc879f465cbcd

I don't know if it matters or not but I'm connecting to PostgreSQL through `pgbouncer` in session pooling mode.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/159#issuecomment-535897028
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20190927/187a23ff/attachment.html>