[OSM-dev] Storing node locations on ways

[Jochen Topf]

On Mi, Apr 20, 2016 at 06:59:35 +0200, Christoph Hormann wrote:
> On Wednesday 20 April 2016, Jochen Topf wrote:
> >
> > Adding node locations to ways is a pain (and needs more than 32 GB
> > RAM these days for a full planet if you want to get it done in
> > reasonable time). So I experimented with adding the node locations to
> > the ways once and storing it in the OSM files. Works for XML, PBF,
> > and OPL. See my blog post at
> > https://blog.jochentopf.com/2016-04-20-node-locations-on-ways.html
> > for the full story.
> 
> From perspective of database design this of course introduces redundancy 
> in the data which in turn bears the risk of introducing 
> inconsistencies.  For storing data temporarily during processing this 
> is not generally a big deal because the processes writing and reading 
> this data are well defined but as a general data exchange and storage 
> format this would be highly problematic - depending on how a program 
> reads the file it will potentially get different results if there are 
> inner inconsistencies in the file.

Well, there are always ways of screwing up the data...

> On a more general note: for the purpose of storing the OSM planet data 
> in a form that is efficient to use and update wouldn't it make sense to 
> store the node locations in a flat file separately from the rest 
> instead of storing everything in a dynamically structured format like 
> XML or PBF that can essentially only be read sequentially?

Sure. That would also make sense for some application. But doesn't solve
the problem I am trying to solve here, that is allowing you to work with
OSM data without much RAM.

> Another thing: you say you throw away untagged nodes that are not member 
> of any way - don't forget nodes can also be relation members.  I don't 
> know any practical case where an untagged node is a relation member but 
> in principle this is possible and could make sense.

Thats true. I thought about adding the node location also to the relation
member, but, as you say, in most cases you'll need the tags of the member,
too, which would make this a whole lot more complicated. So I can't help
with that use case currently.

Jochen
-- 
Jochen Topf  jochen at remote.org  http://www.jochentopf.com/  +49-351-31778688