[OSM-talk] problem when trying to use osmconvert / osmupdate / osmfilter to keep a filtered planet extract up to date

[Martijn van Exel]

Hi,

I thought I had found a nice way to keep a filtered planet extract (by
geographic extent *and* specific tags, in my case bridges in the US)
up-to-date by using osmconvert[1], osmupdate[2] and osmfilter[3]. This
was going to be my workflow:

Setup steps:
A1. Grab a planet file
A2. Derive the extract using osmconvert
bzcat /osm/planet/planet-latest.osm.bz2 | ./osmconvert -
-B=/osm/poly/us.poly  -o=/osm/planet/us.o5m
A3. Filter the desired features into a new file
osmfilter /osm/planet/us.o5m --keep= --keep-ways="bridge=" --out-o5m >
/osm/planet/bridges.o5m
A4. Use osmosis to import the initial filtered planet extract into PostGIS.

Replication steps:
B1. Update the planet extract:
osmupdate /osm/planet/us.o5m /osm/planet/us-new.o5m -B=/osm/poly/us.poly
B2. Filter the desired features into a second file, analogous to step A3
B3. Derive an osc file from the two filter files using osmconvert.
osmconvert bridges.o5m bridges-new.o5m --diff -o=diffbridges.osc
B4. Using osmosis --rxc --wpc, update the PostGIS database
B5. Clean up

I started testing this and quite soon I hit an error in step B4:
SEVERE: Thread for task 1-rxc failed
java.lang.NullPointerException

Trying to load the osc file[4] into JOSM I got the same error.

I can't be sure but I think this has to do with referential integrity
issues caused by missing nodes outside the bounding polygon. The
--complete-ways option in osmconvert is supposed to solve that, but is
not available either when the input is STDIN (step A2) or when
osmconvert is called from osmupdate (step B1). The first case could be
solved by just getting the initial file via some other means. I don't
know how to solve the second case within the context of this set of
tools. Is that possible? Suggestions for an alternative toolchain to
accomplish this? Back to osmosis for the entire process, sacrificing
speed? Would I run into different problems?

[1] https://wiki.openstreetmap.org/wiki/Osmconvert
[2] https://wiki.openstreetmap.org/wiki/Osmupdate
[3] https://wiki.openstreetmap.org/wiki/Osmfilter
[4] This is the file that was created when testing the process and
gave the errors: https://gist.github.com/8001978303cb1077d7c2


-- 
martijn van exel
geospatial omnivore
1109 1st ave #2
salt lake city, ut 84103
801-550-5815
http://oegeo.wordpress.com