[Tile-serving] [osm2pgsql] Regression tests failing (#419)
Paul Norman
notifications at github.com
Fri Jul 31 10:41:14 UTC 2015
building with DEBUG_PGSQL, I get a lot of info.
processing phase
```
Reading in file: diff.osc
Using XML parser.
ExecPrepared: delete_way -- delete from planet_osm_ways
ExecPrepared: insert_way -- insert into planet_osm_ways
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 66
Executing: DELETE FROM planet_osm_line WHERE osm_id = 66
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 66
ExecPrepared: mark_rels_by_way -- get relations using way, which are then set as marked in rels_pending_tracker in middle
ExecPrepared: delete_node -- delete from planet_osm_nodes
ExecPrepared: insert_node -- insert into planet_osm_nodes
Executing: DELETE FROM planet_osm_point WHERE osm_id = 7
ExecPrepared: mark_ways_by_node -- get ways using node, which are then set as marked in middle
ExecPrepared: mark_rels_by_node -- get rels using node, which are then set as marked in middle
ExecPrepared: delete_way -- delete from planet_osm_ways
ExecPrepared: insert_way -- insert into planet_osm_ways
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_line WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 101420
ExecPrepared: mark_rels_by_way -- get relations using way, which are then set as marked in rels_pending_tracker in middle
ExecPrepared: delete_rel -- delete from planet_osm_rels
ExecPrepared: mark_ways_by_rel -- get member ways of rel
ExecPrepared: insert_rel -- insert into planet_osm_rels
Executing: DELETE FROM planet_osm_roads WHERE osm_id = -14
Executing: DELETE FROM planet_osm_line WHERE osm_id = -14
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = -14
ExecPrepared: get_way_list -- get way info for some IDs, presumably three by the below statements. probably 66, 101420, and 6
ExecPrepared: get_node_list -- get a list of nodes
ExecPrepared: get_node_list -- get a list of nodes
ExecPrepared: get_node_list -- get a list of nodes
-- here it looks like it's creating polygon -14 at the same time as deleting its constituant *outer* ways
Executing: COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 66
Executing: DELETE FROM planet_osm_line WHERE osm_id = 66
planet_osm_polygon>>> -14 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N Name_way5 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 0 12893.2 SRID=900913;POLYGON ((-487.3500000000000227 -454.2900000000000205, -331.7900000000000205 -449.4200000000000159, -325.1600000000000250 -556.8099999999999454, -470.5600000000000023 -559.4700000000000273, -487.3500000000000227 -454.2900000000000205), (-433.4399999999999977 -530.7400000000000091, -364.9300000000000068 -526.7699999999999818, -365.3700000000000045 -480.8000000000000114, -431.2200000000000273 -483.0099999999999909, -433.4399999999999977 -530.7400000000000091))
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 66
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_line WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 101420
ExecPrepared: mark_rels
parse time: 0s
Node stats: total(1), max(7) in 0s
Way stats: total(2), max(101420) in 0s
Relation stats: total(1), max(14) in 0s
Executing: COMMIT;
Executing: COMMIT;
Executing: COMMIT;
Committing transaction for planet_osm_point
Executing: COMMIT
Committing transaction for planet_osm_line
Executing: COMMIT
Committing transaction for planet_osm_polygon
Executing: COMMIT
Committing transaction for planet_osm_roads
Executing: COMMIT
```
At this point, it looks like -14 has been deleted and recreated, so the count should be 1.
Tables are now set up again, with statements prepared.
```
Going over pending ways...
2 ways are pending
Using 4 helper-processes
ExecPrepared: get_way -- get way info for one ID
ExecPrepared: get_way -- get way info for one ID
ExecPrepared: get_node_list -- get list of nodes
ExecPrepared: get_node_list -- get list of nodes
-- way 1 uses node 7, so was pending
-- way 6 is inner of rel 14, so presumably needs re-processing
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 1
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 6
Executing: DELETE FROM planet_osm_line WHERE osm_id = 6
Executing: DELETE FROM planet_osm_line WHERE osm_id = 1
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 6
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 1
ExecPrepared: rels_using_way -- find relations using a way
ExecPrepared: rels_using_way -- find relations using a way
Executing: COPY planet_osm_line (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Executing: COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Finished processing 2 ways in 0 sec
-- gets messy to read here, but way 6 is a highway and put in line table, way 1 is a landuse and put in polygon
Committing transaction for planet_osm_point
Executing: COMMIT
planet_osm_line>>> 6 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N \N \N \N \N \N Name_way6 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 33 \N SRID=900913;LINESTRING (-431.2200000000000273 -483.0099999999999909, -365.3700000000000045 -480.8000000000000114, -364.9300000000000068 -526.7699999999999818, -433.4399999999999977 -530.7400000000000091, -431.2200000000000273 -483.0099999999999909)
Committing transaction for planet_osm_line
Executing: COMMIT
Committing transaction for planet_osm_polygon
Executing: COMMIT
Committing transaction for planet_osm_roads
Executing: COMMIT
Committing transaction for planet_osm_point
Executing: COMMIT
Committing transaction for planet_osm_line
Executing: COMMIT
planet_osm_polygon>>> 1 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N Name_way \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 0 27893.2 SRID=900913;POLYGON ((-564.0199999999999818 -232.6500000000000057, -353.5299999999999727 -116.3400000000000034, -401.8299999999999841 -335.1899999999999977, -547.2400000000000091 -337.8299999999999841, -564.0199999999999818 -232.6500000000000057))
Committing transaction for planet_osm_polygon
Executing: COMMIT
Committing transaction for planet_osm_roads
Executing: COMMIT
Committing transaction for planet_osm_point
Executing: COMMIT
Committing transaction for planet_osm_line
Executing: COMMIT
Committing transaction for planet_osm_polygon
Executing: COMMIT
Committing transaction for planet_osm_roads
Executing: COMMIT
Committing transaction for planet_osm_point
Executing: COMMIT
Committing transaction for planet_osm_line
Executing: COMMIT
Committing transaction for planet_osm_polygon
Executing: COMMIT
Committing transaction for planet_osm_roads
Executing: COMMIT
```
Everything is okay here, now we get to pending relations
```
Going over pending relations...
4 relations are pending -- 14, 15 (parent of w1), 1, and ?
-- not clear why relation 1 is in this list, as it references w71 w74, neither which are related to diff
-- but way 1 should be in the list
Using 4 helper-processes
ExecPrepared: get_rel -- get relations
ExecPrepared: get_rel
ExecPrepared: get_rel
ExecPrepared: get_rel
Executing: DELETE FROM planet_osm_roads WHERE osm_id = -15
Executing: DELETE FROM planet_osm_roads WHERE osm_id = -14
Executing: DELETE FROM planet_osm_line WHERE osm_id = -14
Executing: DELETE FROM planet_osm_line WHERE osm_id = -15
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = -15
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = -14
Executing: DELETE FROM planet_osm_roads WHERE osm_id = -14
Executing: DELETE FROM planet_osm_roads WHERE osm_id = -1
ExecPrepared: get_way_list -- get multiple ways
ExecPrepared: get_way_list
ExecPrepared: get_node_list -- get multiple lists of nodes
ExecPrepared: get_node_list
ExecPrepared: get_node_list
ExecPrepared: get_node_list
ExecPrepared: get_node_list
Executing: DELETE FROM planet_osm_line WHERE osm_id = -14
Executing: DELETE FROM planet_osm_line WHERE osm_id = -1
Executing: COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 1 -- not clear why this is being touched in pending relations
Executing: COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = -14
Executing: DELETE FROM planet_osm_line WHERE osm_id = 1
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 66
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = -1
-- put rel 15 into polygon table
planet_osm_polygon>>> -15 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N Name_way \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 0 24749.5 SRID=900913;POLYGON ((-564.0199999999999818 -232.6500000000000057, -353.5299999999999727 -116.3400000000000034, -401.8299999999999841 -335.1899999999999977, -547.2400000000000091 -337.8299999999999841, -564.0199999999999818 -232.6500000000000057), (-510.1100000000000136 -309.1000000000000227, -441.6100000000000136 -305.1299999999999955, -442.0500000000000114 -259.1700000000000159, -507.8999999999999773 -261.3799999999999955, -510.1100000000000136 -309.1000000000000227))
Executing: DELETE FROM planet_osm_line WHERE osm_id = 66
-- put rel 14 into polygon table. should only exist once, as was just deleted above
planet_osm_polygon>>> -14 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N Name_way5 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 0 12893.2 SRID=900913;POLYGON ((-487.3500000000000227 -454.2900000000000205, -331.7900000000000205 -449.4200000000000159, -325.1600000000000250 -556.8099999999999454, -470.5600000000000023 -559.4700000000000273, -487.3500000000000227 -454.2900000000000205), (-433.4399999999999977 -530.7400000000000091, -364.9300000000000068 -526.7699999999999818, -365.3700000000000045 -480.8000000000000114, -431.2200000000000273 -483.0099999999999909, -433.4399999999999977 -530.7400000000000091))
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 66
ExecPrepared: get_way_list
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 1
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 62
ExecPrepared: get_way_list
Executing: DELETE FROM planet_osm_line WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_line WHERE osm_id = 62
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 62
ExecPrepared: get_node_list
ExecPrepared: get_node_list
ExecPrepared: get_node_list
ExecPrepared: get_node_list
ExecPrepared: get_node_list
Executing: COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Executing: COPY planet_osm_polygon (osm_id,"access","addr:housename","addr:housenumber","addr:interpolation","admin_level","aerialway","aeroway","amenity","area","barrier","bicycle","brand","bridge","boundary","building","construction","covered","culvert","cutting","denomination","disused","embankment","foot","generator:source","harbour","highway","historic","horse","intermittent","junction","landuse","layer","leisure","lock","man_made","military","motorcar","name","natural","office","oneway","operator","place","population","power","power_source","public_transport","railway","ref","religion","route","service","shop","sport","surface","toll","tourism","tower:type","tracktype","tunnel","water","waterway","wetland","width","wood","z_order","way_area",way) FROM STDIN
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 66
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 71
Executing: DELETE FROM planet_osm_line WHERE osm_id = 66
Executing: DELETE FROM planet_osm_line WHERE osm_id = 71
-- put rel -14 into poly table.
planet_osm_polygon>>> -14 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N Name_way5 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 0 12893.2 SRID=900913;POLYGON ((-487.3500000000000227 -454.2900000000000205, -331.7900000000000205 -449.4200000000000159, -325.1600000000000250 -556.8099999999999454, -470.5600000000000023 -559.4700000000000273, -487.3500000000000227 -454.2900000000000205), (-433.4399999999999977 -530.7400000000000091, -364.9300000000000068 -526.7699999999999818, -365.3700000000000045 -480.8000000000000114, -431.2200000000000273 -483.0099999999999909, -433.4399999999999977 -530.7400000000000091))
planet_osm_polygon>>> -1 \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N residential \N \N \N \N \N \N Name_rel \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N \N 0 12895 SRID=900913;POLYGON ((-377.3100000000000023 -324.3500000000000227, -221.7500000000000000 -319.4900000000000091, -215.1200000000000045 -426.8899999999999864, -360.5099999999999909 -429.5400000000000205, -377.3100000000000023 -324.3500000000000227), (-323.3999999999999773 -400.8100000000000023, -254.8899999999999864 -396.8299999999999841, -255.3400000000000034 -350.8700000000000045, -321.1800000000000068 -353.0899999999999750, -323.3999999999999773 -400.8100000000000023))
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 66
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 71
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_roads WHERE osm_id = 74
Executing: DELETE FROM planet_osm_line WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_line WHERE osm_id = 74
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 101420
Executing: DELETE FROM planet_osm_polygon WHERE osm_id = 74
Finished processing 4 relations in 0 sec
Committing transaction for planet_osm_point
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_point
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_line
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_polygon
Executing: COMMIT
WARNING: there is no transaction in progress
Committing transaction for planet_osm_roads
Executing: COMMIT
WARNING: there is no transaction in progress
Completed planet_osm_point
Stopping table: planet_osm_nodes
Completed planet_osm_line
Stopping table: planet_osm_ways
Completed planet_osm_polygon
Stopping table: planet_osm_rels
Completed planet_osm_roads
Stopped table: planet_osm_ways in 0s
Stopped table: planet_osm_nodes in 0s
Stopped table: planet_osm_rels in 0s
node cache: stored: 1(100.00%), storage efficiency: 0.10% (dense blocks: 1, sparse nodes: 0), hit rate: 3.17%
Osm2pgsql took 1s overall
```
Here is the problem. In order, `DELETE FROM planet_osm_polygon WHERE osm_id = -14` is executed twice, then put into planet_osm_polygon twice with COPY. My guess is that there are two threads, each which deletes and copies, and some of the time they delete at the same time then both copy.
So, why would two threads both work on relation 14?
One option is that id_tracker is not thread safe.
The underlying implementation of id_tracker has a vector of uint32_t, and each uint32 is used to store a block of bits. This does mean that size scales with max(id), but for ways this is only 43MB at current numbers.
pop_min is what both threads would be calling. Given it has no mutexes, I see no reason why two threads couldn't call it at the same time and get the same result.
---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/419#issuecomment-126644507
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150731/5d7dbcd8/attachment-0001.html>
More information about the Tile-serving
mailing list