[OSM-dev] osmosis advice
Igor Podolskiy
igor.podolskiy at vwi-stuttgart.de
Tue Apr 26 10:58:23 BST 2011
Pierre-Alain,
as far as I understand you are trying to filter out all objects that
have man_made=surveillance set on it, be it nodes, ways, or relations.
Please correct me if I got this wrong.
Even if it seems that it is a single task, it is really helpful to think
of it as multiple tasks:
1. Get all nodes with man_made=surveillance
2. Get all ways with man_made=surveillance and all nodes used by them
3. Get all relations with man_made=surveillance and all ways used by
them and all nodes used by the relations and the ways.
4. Merge all of this into a single data set.
Osmosis works like a pipeline by passing the data from the source along
the tasks; it's all about data streams. It's actually quite simple once
you wrap your mind around it ;). So this "get all X" stuff actually
works by "throwing unneeded objects away" from the big source data
stream rather than by directly selecting the things you need (at least
when you do this with a PBF source). If you do the stuff "in order" in a
single pipeline like in your first example, you may be throwing away
things you would later need (like the untagged nodes which comprise a
man_made=surveillance way).
So the solution is to treat the separate tasks I mentioned earlier like,
well, _separate_ tasks :)
For the nodes (1.) you do something along the tune of:
--tf accept-nodes man_made=surveillance reject-ways reject-relations
Easy and simple.
With the ways (2.), this gets a little bit trickier:
--tf --accept-ways man_made=surveillance reject-relations --used-node
This leaves all nodes untouched by the --tf task: they will get filtered
in --used-node later when we actually know what to filter.
With the relations, this gets even tricker because relations can contain
nodes _and_ ways.
--tf --accept-relations man_made=surveillance --used-way --used-node
This is basically the same as with the ways but with the added
--used-way task.
You can run these filters in different Osmosis processes if you want to
and write them to disk and run a merge later which will look something
like this:
--rb sv-nodes.pbf outPipe.0=N --rb sv-ways.pbf outPipe.0=W --rb
sv-rels.pbf outPipe.0=R --merge inPipe.0=N inPipe.1=W outPipe.0=NW
--merge inPipe.0=NW inPipe.1=R --wx final.xml
Or you can do it all in a single process with a --tee 3 task at the
beginning.
I suggest you give OSMembrane a try (http://osmembrane.de) for
developing your pipeline - it's a graphical frontend for Osmosis which
makes those long and weird command lines somewhat simpler to create. But
then, I'm OSMembrane's current maintainer, so I may be biased :)
As to your memory problem - try giving Osmosis a little more memory in
-Xmx, or try to pre-split the data file geographically. But I don't
think you will need it - man_made=surveillance doesn't seem to be a very
common tag to me. I suspect Osmosis ran out of memory in your last
example because you did keep all the ways in your data stream and then
--used-node had to operate on all ways in France, which are quite a few :)
Hope that helps,
(and ask again if I got something wrong)
Igor
More information about the dev
mailing list