[Tile-serving] [openstreetmap/osm2pgsql] Long import of Europe data file (#676)

[Marcos Dione]

On Fri, Jan 27, 2017 at 10:52:34AM -0800, Koles500 wrote:
> "Relation" is done, it is continuing , but some strange exceptions were throws. Could you please say something more about these exceptions ?
> 
> `PS C:\OSM\rendering> osm2pgsql -U postgres -m -d osm -p osm -E 3857 -s -S C:\OSM\osm2pgsql\default.style C:\OSM\Data\eur
> ope-latest.osm.pbf
> osm2pgsql version 0.92.0 (64 bit id space)
> 
> Using built-in tag processing pipeline
> Using projection SRS 3857 (Spherical Mercator)
> Setting up table: osm_point
> Setting up table: osm_line
> Setting up table: osm_polygon
> Setting up table: osm_roads
> Allocating memory for sparse node cache
> Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=9
> Mid: pgsql, scale=100 cache=800
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> 
> Reading in file: C:\OSM\Data\europe-latest.osm.pbf
> Using PBF parser.
> Processing: Node(1827652k 131.7k/s) Way(223763k 0.78k/s) Relation(3148140 15.84/s)
> Standard exception processing relation id=6540291: TopologyException: side location conflict at -85245.029999999999 6666
> 499.7699999996
> Processing: Node(1827652k 131.7k/s) Way(223763k 0.78k/s) Relation(3254340 15.98/s)
> Standard exception processing relation id=6752256: TopologyException: side location conflict at 538047.93000000005 56717
> 07.1900000004
> Processing: Node(1827652k 131.7k/s) Way(223763k 0.78k/s) Relation(3316870 16.07/s)  parse time: 507324s
> Node stats: total(1827652739), max(4604487172) in 13878s
> Way stats: total(223763746), max(465494091) in 287105s
> Relation stats: total(3316871), max(6875737) in 206341s
> Committing transaction for osm_point
> Committing transaction for osm_line
> Committing transaction for osm_polygon
> Committing transaction for osm_roads
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> Setting up table: osm_nodes
> Setting up table: osm_ways
> Setting up table: osm_rels
> Using built-in tag processing pipeline
> 
> Going over pending ways...
>         165944299 ways are pending
> 
> Using 8 helper-processes`

    I don't see anything wrong, it's just 8 processes printing the same
message.

    if just importing the data took almost 6 days (!), I have very bad
news for you: you just passed the mostly sequential part of the import.
Like Harmut said, then comes the index processing and that will *kill*
your current performance. Are you *sure* you want the whole Europe? I
would definitely make all efforts to buy the cheapest, even 2nd hand 500
MiB SSD I could if I were really, *really* pressed to finish soon.

-- 
(Not so) Random fortune:
Terrorism isn't a crime against people or property. It's a crime against
our minds, using the death of innocents and destruction of property to
make us fearful.
	    -- Bruce Schneier