[OSM-dev] Osmosis crash for large map import to osmbin format

Curt Nowak cnowak at bwl.uni-hildesheim.de
Wed May 27 16:18:52 BST 2009


Hi all,

when trying to import a large map (Germany) into the osmbin format[1]
using osmosis, the program crashes with the output shown at the end of
this mail. I'd like to use the map of Germany for simple routing
problems with the help of libosm[2]. So the osmbin format seemed like
the best idea.
I had already filtered the map before, running the appropriate osmosis
tasks on the map. Still, the "Size exceeds Integer.MAX_VALUE" part makes
me wonder whether this is a permanent issue for large maps (see also
[3]).
I am not sure where to go next? Is there a possible fix? Should I switch
to a Postgres DB for the map data? Is there a way to make libosm work
with Postgres?
Any hints are most welcome,

Curt



[1] http://wiki.openstreetmap.org/wiki/OSMbin(file_format)
[2] http://wiki.openstreetmap.org/wiki/Traveling_salesman#LibOSM
[3]
https://apps.sourceforge.net/phpbb/travelingsales/viewtopic.php?f=4&t=78

---------------
This is the entire java call:
java -Xss1m -Xmx1048m -XX:MaxDirectMemorySize=256M -cp
C:\Programme\OSM\LibOSM\libosm-2.5.0_plugin_for_osmosis-0.30.2.zip;trave
ling_salesman-1.0.1.signed.jar org.openstreetmap.osmosis.core.Osmosis
-verbose -plugin org.openstreetmap.osm.data.osmbin.WriteOSMBinPlugin
--read-xml-0.6 file=C:\Programme\OSM\germany\germany_filtered.osm
outPipe.0="firstPipe" --write-osmbin-0.6 inPipe.0="firstPipe"
dir=c:\programme\osm\Germany

---------------
This is the last part of the output:
27.05.2009 16:40:09 org.openstreetmap.osm.data.osmbin.IDIndexFile
findFreeRecord
INFO: Growing an Index-File... from 133120001 to 138240001 records
27.05.2009 16:40:09 org.openstreetmap.osm.data.osmbin.FixedRecordFile
growFile
INFO: Growing File - forced Garbage collection done after 31ms...
27.05.2009 16:40:09 org.openstreetmap.osm.data.osmbin.FixedRecordFile
growFile
INFO: Growing File - old memory collected, mapping new memory...
27.05.2009 16:40:09
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinDataSetV10 addNode
SCHWERWIEGEND: [Exception] Problem in
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinDataSetV10
java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:707)
	at
org.openstreetmap.osm.data.osmbin.FixedRecordFile.growFile(FixedRecordFi
le.java:689)
	at
org.openstreetmap.osm.data.osmbin.IDIndexFile.findFreeRecord(IDIndexFile
.java:275)
	at
org.openstreetmap.osm.data.osmbin.IDIndexFile.put(IDIndexFile.java:399)
	at
org.openstreetmap.osm.data.osmbin.v1_0.NodesFile.writeNode(NodesFile.jav
a:455)
	at
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinDataSetV10.addNode(OsmBinDa
taSetV10.java:335)
	at
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10Writer.process(OsmBinV10
Writer.java:45)
	at
org.openstreetmap.osmosis.core.xml.v0_6.impl.NodeElementProcessor.end(No
deElementProcessor.java:106)
	at
org.openstreetmap.osmosis.core.xml.v0_6.impl.OsmHandler.endElement(OsmHa
ndler.java:108)
	at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(
AbstractSAXParser.java:601)
	at
com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emp
tyElement(AbstractXMLDocumentParser.java:180)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.s
canStartElement(XMLDocumentFragmentScannerImpl.java:1339)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$F
ragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2747)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLD
ocumentScannerImpl.java:648)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.s
canDocument(XMLDocumentFragmentScannerImpl.java:510)
	at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML1
1Configuration.java:807)
	at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML1
1Configuration.java:737)
	at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.jav
a:107)
	at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Abstr
actSAXParser.java:1205)
	at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.pars
e(SAXParserImpl.java:522)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
	at
org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:108
)
	at java.lang.Thread.run(Thread.java:619)
27.05.2009 16:40:09
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
waitForCompletion
SCHWERWIEGEND: Thread for task 1-read-xml-0.6 failed
java.lang.IllegalStateException: Size exceeds Integer.MAX_VALUE
	at
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinDataSetV10.addNode(OsmBinDa
taSetV10.java:340)
	at
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinV10Writer.process(OsmBinV10
Writer.java:45)
	at
org.openstreetmap.osmosis.core.xml.v0_6.impl.NodeElementProcessor.end(No
deElementProcessor.java:106)
	at
org.openstreetmap.osmosis.core.xml.v0_6.impl.OsmHandler.endElement(OsmHa
ndler.java:108)
	at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(
AbstractSAXParser.java:601)
	at
com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emp
tyElement(AbstractXMLDocumentParser.java:180)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.s
canStartElement(XMLDocumentFragmentScannerImpl.java:1339)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$F
ragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2747)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLD
ocumentScannerImpl.java:648)
	at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.s
canDocument(XMLDocumentFragmentScannerImpl.java:510)
	at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML1
1Configuration.java:807)
	at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML1
1Configuration.java:737)
	at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.jav
a:107)
	at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Abstr
actSAXParser.java:1205)
	at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.pars
e(SAXParserImpl.java:522)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
	at
org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:108
)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Size exceeds
Integer.MAX_VALUE
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:707)
	at
org.openstreetmap.osm.data.osmbin.FixedRecordFile.growFile(FixedRecordFi
le.java:689)
	at
org.openstreetmap.osm.data.osmbin.IDIndexFile.findFreeRecord(IDIndexFile
.java:275)
	at
org.openstreetmap.osm.data.osmbin.IDIndexFile.put(IDIndexFile.java:399)
	at
org.openstreetmap.osm.data.osmbin.v1_0.NodesFile.writeNode(NodesFile.jav
a:455)
	at
org.openstreetmap.osm.data.osmbin.v1_0.OsmBinDataSetV10.addNode(OsmBinDa
taSetV10.java:335)
	... 18 more
27.05.2009 16:40:09
org.openstreetmap.osmosis.core.pipeline.common.PassiveTaskManager
waitForCompletion
FEIN: Task 2-write-osmbin-0.6 is passive, no completion wait required.
27.05.2009 16:40:09 org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more
tasks failed.
	at
org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletio
n(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30)




More information about the dev mailing list