[OSM-dev] planet_osm2txt with planet-060818?

[Dean Earley]

>>> Was anyone successfull with 
>>> ./utils/osm-pdf-atlas/planet_osm2txt.pl planet-060818-a.osm.bz2 
>>> Any hints on how to do this on a machine with only 2GB of Memory?
>>> 
>> 
>> I guess the parsers will need to be rewritten to use SAX now rather
>> than the DOM (read all and iterate) method. I happily parsed a
>> 2.6GB planet.osm in about 2 minutes using MSXML (in VB)
> 
> Would you mind sharing so more info on this approach. I'd be
> interested in taking a look at some of the data in the planet file.

Basically, it is an event based parser that keeps as little in memory as 
necessary.

It notifies my code whenever a complete tag (start of and end of) has 
been found and any textual content.

This is a sample of the VB code I used:

Private Sub IVBSAXContentHandler_startElement(strNamespaceURI As String, 
strLocalName As String, strQName As String, ByVal attributes As 
MSXML2.IVBSAXAttributes)
Dim ID As Long

Dim Lat As Long
Dim Lon As Long

Dim ToNode As Long
Dim FromNode As Long

Static LastID As Long
Static LastType As String

   Select Case strLocalName
   Case "osm"
   Case "node"
     ID = Val(attributes.getValueFromName("", "id"))
     Lat = Val(attributes.getValueFromName("", "lat")) * 100000
     Lon = Val(attributes.getValueFromName("", "lon")) * 100000

     DB.Execute "INSERT INTO node (ID, Latitude, Longitude) VALUES (" & 
CStr(ID) & ", " & CStr(Lat) & ", " & CStr(Lon) & ");"

     LastType = "node"
     LastID = ID

   Case "segment"
     ID = Val(attributes.getValueFromName("", "id"))
     ToNode = Val(attributes.getValueFromName("", "to"))
     FromNode = Val(attributes.getValueFromName("", "from"))

     DB.Execute "INSERT INTO segment (ID, 'To', 'From') VALUES (" & 
CStr(ID) & ", " & CStr(ToNode) & ", " & CStr(FromNode) & ");"

     LastType = "segment"
     LastID = ID

   Case "way"
     ID = Val(attributes.getValueFromName("", "id"))

     DB.Execute "INSERT INTO way (ID) VALUES (" & CStr(ID) & ");"

     LastType = "way"
     LastID = ID

   Case "seg"
     DB.Execute "INSERT INTO waysegments (WayID, SegmentID) VALUES (" & 
CStr(LastID) & ", " & attributes.getValueFromName("", "id") & ");"

   Case "tag"
     DB.Execute "INSERT INTO tags (ID, Parent, Key, Value) VALUES (" & 
CStr(LastID) & ", """ & LastType & """, """ & 
Replace(attributes.getValueFromName("", "k"), """", "\""") & """, """ & 
Replace(attributes.getValueFromName("", "v"), """", "\""") & """);"

   Case Else
     'Stop on unrecognised tags
     Stop
   End Select
End Sub

It does require a syntactically valid XML file and unfortunately, this 
never completed as I ran out of disk space for the target db :)

/me runs off to his flame proof bunker...

-- 
Dean Earley, Dee (dean at earlsoft.co.uk)

irc:    irc://irc.blitzed.org/
web:    http://personal.earlsoft.co.uk
phone:  +44 (0)780 8369596