[Tile-serving] [osm2pgsql] Import failing whilst processing relations (#329)

[Zsolt Ero]

I had the exact thing happen to me. 

log:
```
[1001]: [4-1] ERROR:  unexpected EOF on client connection with an open transaction
[1003]: [4-1] ERROR:  unexpected EOF on client connection with an open transaction
[999]: [4-1] ERROR:  unexpected EOF on client connection with an open transaction
[998]: [4-1] ERROR:  unexpected EOF on client connection with an open transaction
[999]: [4-2] CONTEXT:  COPY planet_osm_line, line 36811394
[1003]: [4-2] CONTEXT:  COPY planet_osm_ways, line 82155938
[998]: [4-2] CONTEXT:  COPY planet_osm_point, line 76244780
[1003]: [4-3] STATEMENT:  COPY planet_osm_ways FROM STDIN;
[999]: [4-3] STATEMENT:  COPY planet_osm_line (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
[1003]: [4-4] 	
[998]: [4-3] STATEMENT:  COPY planet_osm_point (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","capital","construction","covered","culvert","cutting","denomination","disused","ele","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","poi","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tunnel","water","waterway","wetland","width","wood","z_order",tags,way) FROM STDIN
[1001]: [4-2] CONTEXT:  COPY planet_osm_roads, line 3524268
[1001]: [4-3] STATEMENT:  COPY planet_osm_roads (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
[1000]: [3-1] ERROR:  unexpected EOF on client connection with an open transaction
[1002]: [3-1] ERROR:  unexpected EOF on client connection with an open transaction
[1004]: [3-1] ERROR:  unexpected EOF on client connection with an open transaction
[1000]: [3-2] CONTEXT:  COPY planet_osm_polygon, line 1
[1002]: [3-2] CONTEXT:  COPY planet_osm_nodes, line 1
[1004]: [3-2] CONTEXT:  COPY planet_osm_rels, line 1
[1002]: [3-3] STATEMENT:  COPY planet_osm_nodes FROM STDIN;
[1004]: [3-3] STATEMENT:  COPY planet_osm_rels FROM STDIN;
[1000]: [3-3] STATEMENT:  COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",tags,way) FROM STDIN
[1002]: [3-4] 	
[1004]: [3-4] 	
[1002]: [5-1] FATAL:  terminating connection because protocol sync was lost
[1004]: [5-1] FATAL:  terminating connection because protocol sync was lost
[1000]: [5-1] FATAL:  terminating connection because protocol sync was lost
[999]: [6-1] FATAL:  terminating connection because protocol sync was lost
[1001]: [6-1] FATAL:  terminating connection because protocol sync was lost
[998]: [6-1] FATAL:  terminating connection because protocol sync was lost
[1003]: [6-1] FATAL:  terminating connection because protocol sync was lost
```

config
```
# default parts

datestyle = 'iso, mdy'
default_text_search_config = 'pg_catalog.english'
lc_messages = 'C'
lc_monetary = 'C'
lc_numeric = 'C'
lc_time = 'C'
log_destination = 'syslog'
log_timezone = 'UTC'
max_connections = 100
timezone = 'UTC'
update_process_title = off


# changed parts

autovacuum = off
fsync = off
synchronous_commit = off
full_page_writes = off

shared_buffers = 512MB
work_mem = 64MB
maintenance_work_mem = 1024MB
wal_buffers = 16MB

checkpoint_segments = 64
checkpoint_completion_target = 0.9
random_page_cost = 2.0
cpu_tuple_cost = 0.05

effective_io_concurrency = 2
```

command
```
osm2pgsql --create --slim --drop --flat-nodes flat.bin --cache 14000 --number-processes 6 --hstore --database $DB_NAME $PBF_FILE
```

config: FreeBSD 10.2, PostgreSQL 9.3.9, PostGIS 2.1.7, osm2pgsql 0.87.2, default ZFS

Note: I disabled swap on purpose as I prefer a failed import over one which slows down because of swapping. On the other hand, still in the processing node-way-relation part I've not seen over 7 GB of memory usage with the above settings.

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/329#issuecomment-143639482
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150927/e6f4a353/attachment-0001.html>