[OSM-talk] Another point in support of ways as ordered lists

[Joerg Ostertag (OSM Munich/Germany)]

On Wednesday 20 September 2006 18:08, Wollschaf wrote:
> I have started to write a program in c that might later do those things,
> and implemented a rudimentary osm parser (optimized on speed, not fault
> tolerance; tailored for osm data). Parsing all nodes and counting
> how many are in a given area (no storing), and reading the rest of the
> file but not doing anything with it (yet) takes around five minutes (a64 @
> 1000Mhz, reading file over nfs);

If I don't have to store them in a different format (This means keeping all 
node Tags in Memory or file) its fast too. If I have enough memory 16GB for 
this one process to store all the relevant nodes it take less than 40 minutes 
all together. This would include:
 - bunzip2 of the data
 - parsing with XML::Parser
 - resorting all the nodes/segments/ways
 - combining this all together
 - filtering
 - writing the results to 3 separate files
I think you can speed this up (a little) if you write it in C, but you'll need 
a lot of memory for this task too. So I don't really see this much of a 
benefit. We currently have to deal with 3GB of Data and the program must be 
written in a way which can handle 300GB of Source-Data later too.

The problem is: how do I take 
 node 11 ...lat/lon=a/b tag=x
 node 12 ...lat/lon=c/d tag=y
 segment 9 ...from=11 to=12 tag=w
 way ... seg=...,9,... tag=u
into something like
 segment 9 ... lat/lon1=a/b lat/lon2=c/d tag=u,w,x,y,
with the complete planet.osm with 
 - low memory consumption (<1GB) 
 - and fast (<10 hours)?

-- 
Jörg (Germany, Munich)

http://www.ostertag.name/
TeamSpeak2: ts2.ostertag.name, user: tweety, Channel: "GPS Drive"
irc://irc.oftc.net/#osm
Tel.: +49 89 420950304