[osmosis-dev] osmosis pipelining troubles

[MERIGHI Marcus]

Hello Andrew, 

first of all thanks for the hint to docs I have read - but not fully,
sorry.

andrew at fastmail.net (Andrew Byrd), 2011.12.07 (Wed) 13:34 (CET):
> Using tee followed by merge can trigger deadlock, and I would guess
> this is what's happening in your case:
> 
> http://lists.openstreetmap.org/pipermail/dev/2009-June/015945.html
> 
> I usually get around the problem by using separate read-xml tasks
> for each sub-pipeline that will be merged at the end of the
> pipeline. See the last example in this section of the Osmosis wiki:
> http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.38#--tag-filter_.28--tf.29

Unfortunately I depend on results of parts of the pipe within the pipe;
thus I (think I) have to write intermediary files to prevent --tee. 
BUT the example below does _not_ work, same error...

osmosis \
  -verbose 5 \
  --read-xml file="${_src}" outPipe.0=source \
  --sort inPipe.0=source outPipe.0=sourcesort \
  --simplify-ways epsilonMeters="${_err}" inPipe.0=sourcesort outPipe.0=norm \
  --sort inPipe.0=norm outPipe.0=normsort \
  --write-pbf file="${_fnnorm}" inPipe.0=normsort \
  --read-pbf file="${_fnnorm}" outPipe.0=normsortf \
  \
  --used-node inPipe.0=normsortf outPipe.0=usedonly \
  --sort inPipe.0=usedonly outPipe.0=usedonlysort \
  \
  --write-pbf file="${_fnused}" inPipe.0=usedonlysort \
  --read-pbf file="${_fnused}" outPipe.0=usedonly \
  --read-pbf file="${_fnnorm}" outPipe.0=norm \
  \
  --derive-change inPipe.0=usedonly inPipe.1=norm outPipe.0=diff \
  --sort-change inPipe.0=diff outPipe.0=diffsort \
  --read-empty outPipe.0=empty \
  --apply-change inPipe.0=empty inPipe.1=diffsort outPipe.0=diffosm \
  --tag-transform file="${HOME}/.nonemptynodes.xml" inPipe.0=diffosm outPipe.0=tt \
  --node-key keyList="is_poi" inPipe.0=tt outPipe.0=pois \
  --write-pbf file="${_fnpois}" inPipe.0=pois \
  \
  --read-pbf file="${_fnused}" outPipe.0=usedonly \
  --read-pbf file="${_fnpois}" outPipe.0=pois \
  --merge inPipe.0=usedonly inPipe.1=pois outPipe.0=merged \
  --write-xml inPipe.0=merged file="${_tgt}" 

Bye, Marcus

> On 12/07/2011 10:00 AM, MERIGHI Marcus wrote:
> >Hello,
> >
> >while trying to integrate various processing steps into a rather complex
> >pipeline I have run into troubles. Each step alone seems to work, even
> >more complex arrangements, but not all of them. (Go to bottom of message
> >(3) to see an working example with multiple invocations of osmosis.)
> >
> >The osmosis commands below (1) stop at "FINE: Waiting for task
> >1-read-xml to complete.", probably forever. CPU utilization goes up
> >first and then slowly down to zero, fstat does not show any read
> >activity on the input file (anymore).
> >I have tried to pin down at what point the processing hangs; everything
> >before "--derive-change" works, tested with "--write-null[-change]".
> >
> >The questions are:
> >1) Am I getting something fundamentally wrong about the way osmosis
> >    pipes work?
> >2) Is there any obvious mistake in my parameters?
> >3) Have I hit a bug?
> >
> >(BTW, basicly I am after simplifying ways; but --simplify-ways leaves
> >(now) unused nodes behind; getting rid of them by means of --used-node
> >removes poi-type nodes as well (not used in ways and relations).
> >Therefore I have to go the long way of diffing before- and after
> >--used-node, parse that for "used" (i.e., poi-type) nodes and merge.)
> >
> >Thanks in advance for any pointers!