[OSM-dev] osmosis advice

Igor Podolskiy igor.podolskiy at vwi-stuttgart.de
Tue Apr 26 10:58:23 BST 2011


Pierre-Alain,

as far as I understand you are trying to filter out all objects that 
have man_made=surveillance set on it, be it nodes, ways, or relations. 
Please correct me if I got this wrong.

Even if it seems that it is a single task, it is really helpful to think 
of it as multiple tasks:

1. Get all nodes with man_made=surveillance
2. Get all ways with man_made=surveillance and all nodes used by them
3. Get all relations with man_made=surveillance and all ways used by 
them and all nodes used by the relations and the ways.
4. Merge all of this into a single data set.

Osmosis works like a pipeline by passing the data from the source along 
the tasks; it's all about data streams. It's actually quite simple once 
you wrap your mind around it ;). So this "get all X" stuff actually 
works by "throwing unneeded objects away" from the big source data 
stream rather than by directly selecting the things you need (at least 
when you do this with a PBF source). If you do the stuff "in order" in a 
single pipeline like in your first example, you may be throwing away 
things you would later need (like the untagged nodes which comprise a 
man_made=surveillance way).

So the solution is to treat the separate tasks I mentioned earlier like, 
well, _separate_ tasks :)

For the nodes (1.) you do something along the tune of:

--tf accept-nodes man_made=surveillance reject-ways reject-relations

Easy and simple.

With the ways (2.), this gets a little bit trickier:

--tf --accept-ways man_made=surveillance reject-relations --used-node

This leaves all nodes untouched by the --tf task: they will get filtered 
in --used-node later when we actually know what to filter.

With the relations, this gets even tricker because relations can contain 
nodes _and_ ways.

--tf --accept-relations man_made=surveillance --used-way --used-node

This is basically the same as with the ways but with the added 
--used-way task.

You can run these filters in different Osmosis processes if you want to 
and write them to disk and run a merge later which will look something 
like this:

--rb sv-nodes.pbf outPipe.0=N --rb sv-ways.pbf outPipe.0=W --rb 
sv-rels.pbf outPipe.0=R --merge inPipe.0=N inPipe.1=W outPipe.0=NW 
--merge inPipe.0=NW inPipe.1=R --wx final.xml

Or you can do it all in a single process with a --tee 3 task at the 
beginning.

I suggest you give OSMembrane a try (http://osmembrane.de) for 
developing your pipeline - it's a graphical frontend for Osmosis which 
makes those long and weird command lines somewhat simpler to create. But 
then, I'm OSMembrane's current maintainer, so I may be biased :)

As to your memory problem - try giving Osmosis a little more memory in 
-Xmx, or try to pre-split the data file geographically. But I don't 
think you will need it - man_made=surveillance doesn't seem to be a very 
common tag to me. I suspect Osmosis ran out of memory in your last 
example because you did keep all the ways in your data stream and then 
--used-node had to operate on all ways in France, which are quite a few :)

Hope that helps,
(and ask again if I got something wrong)
Igor




More information about the dev mailing list