[Tile-serving] [openstreetmap/osm2pgsql] bad result during COPY (#159)

[Alexander Naydenko]

The same problem encountered during import of 2GB pbf.
Default postgres 9.5 settings, host Ubuntu Xenial, Openstack VM with 48GB RAM, SSD storage and 12 cores.

Error log:
```
2018-10-21 21:48:03 == Import data
osm2pgsql version 0.93.0-dev (64 bit id space)

Using projection SRS 4326 (Latlong)
NOTICE:  table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=7467MB, maxblocks=119472*65536, allocation method=11
Mid: pgsql, cache=7467
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /data/russia-latest.osm.pbf
Using PBF parser.
Processing: Node(7530k 134.5k/s) Way(0k 0.00k/s) Relation(0 0.00/s)node cache: stored: 7616000(100.00%), storage efficiency: 50.51% (dense blocks: 122, sparse nodes: 7039215), hit rate: -nan%
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_rels, line 1
STATEMENT:  COPY planet_osm_rels FROM STDIN;

ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_ways, line 1
STATEMENT:  COPY planet_osm_ways FROM STDIN;

LOG:  could not send data to client: Broken pipe
STATEMENT:  COPY planet_osm_rels FROM STDIN;

LOG:  could not send data to client: Broken pipe
STATEMENT:  COPY planet_osm_ways FROM STDIN;

FATAL:  connection to client lost
FATAL:  connection to client lost
Osm2pgsql failed due to ERROR: PBF error: truncated data (EOF encountered)
LOG:  incomplete message from client
CONTEXT:  COPY place, line 48838
STATEMENT:  COPY place (osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry) FROM STDIN
ERROR:  unexpected EOF on client connection with an open transaction
CONTEXT:  COPY place, line 48838
STATEMENT:  COPY place (osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry) FROM STDIN
LOG:  could not send data to client: Broken pipe
STATEMENT:  COPY place (osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry) FROM STDIN
ERROR: Error executing external command: /app/src/build/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore --number-processes 1 -C 7467 -P 5432 -d nominatim /data/russia-latest.osm.pbf
Error executing external command: /app/src/build/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore --number-processes 1 -C 7467 -P 5432 -d nominatim /data/russia-latest.osm.pbf
FATAL:  terminating connection because protocol synchronization was lost
ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY planet_osm_nodes, line 7616001
STATEMENT:  COPY planet_osm_nodes FROM STDIN;

LOG:  could not send data to client: Broken pipe
STATEMENT:  COPY planet_osm_nodes FROM STDIN;

waiting for server to shut down....LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
FATAL:  terminating connection due to administrator command
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/159#issuecomment-431707332
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20181021/4369f98b/attachment.html>