[OSM-dev] Osmosis crashing due to lack of memory

Karl Newman siliconfiend at gmail.com
Thu Feb 26 16:26:14 GMT 2009


(cc'ing to osmosis-dev)

Clearly it's the ListIdTracker in the Used Node Filter task that is causing
the problem. The ListIdTracker is good for smallish data sets, but for
processing planet-wide data sets a BitSetIdTracker is more memory-efficient.
So, try adding idTrackerType=BitSet to the --used-node task (i.e., java
-Xmx1024m -jar e:\osmosis-0.30\osmosis.jar --read-xml
file="planet-090218.osm" enableDateParsing="no" --log-progress interval="10"
--way-key-value keyValueList="..." --used-node idTrackerType=BitSet
--migrate --write-xml-0.6 file="E:\world\dump\roads\roads.osm"

Karl

On Thu, Feb 26, 2009 at 5:48 AM, Emilie Laffray <emilie.laffray at gmail.com>wrote:

> >
> > Are you sure to be using DMA for HDD-access?
> > It should be IO-bound for such simple tasks.
> >
> > What do you mean with "maxed out"? "99% user" or
> > just just "0% idle"?
> > IO-processes (Windows: [x]show kernel times) use
> > cpu-time too.
> >
> > (It's sad that Java-NIO does not get async file-io until
> >  Java7.)
> >
>
> Well, whether it is IO bound or not in this case doesn't really
> matter. Osmosis just crashed again on me, while I increased the VM to
> 1024MB.
> The exact message that i am getting is this:
>
> INFO: Processing Way 15717214.
> 26-Feb-2009 12:45:54
> com.bretth.osmosis.core.progress.v0_5.EntityProgressLogger
> process
> INFO: Processing Way 15815970.
> 26-Feb-2009 12:46:00
> com.bretth.osmosis.core.pipeline.common.ActiveTaskManager w
> aitForCompletion
> SEVERE: Thread for task 1-read-xml failed
> java.lang.OutOfMemoryError: Java heap space
>        at
> com.bretth.osmosis.core.filter.common.ListIdTracker.extendIdList(List
> IdTracker.java:66)
>        at
> com.bretth.osmosis.core.filter.common.ListIdTracker.set(ListIdTracker
> .java:117)
>        at
> com.bretth.osmosis.core.filter.v0_5.UsedNodeFilter.process(UsedNodeFi
> lter.java:89)
>        at
> com.bretth.osmosis.core.container.v0_5.WayContainer.process(WayContai
> ner.java:60)
>        at
> com.bretth.osmosis.core.filter.v0_5.UsedNodeFilter.process(UsedNodeFi
> lter.java:58)
>        at
> com.bretth.osmosis.core.filter.v0_5.WayKeyValueFilter.process(WayKeyV
> alueFilter.java:90)
>        at
> com.bretth.osmosis.core.container.v0_5.WayContainer.process(WayContai
> ner.java:60)
>        at
> com.bretth.osmosis.core.filter.v0_5.WayKeyValueFilter.process(WayKeyV
> alueFilter.java:53)
>        at
> com.bretth.osmosis.core.progress.v0_5.EntityProgressLogger.process(En
> tityProgressLogger.java:84)
>        at
> com.bretth.osmosis.core.xml.v0_5.impl.WayElementProcessor.end(WayElem
> entProcessor.java:103)
>        at
> com.bretth.osmosis.core.xml.v0_5.impl.OsmHandler.endElement(OsmHandle
> r.java:109)
>        at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endEleme
> nt(Unknown Source)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
> l.scanEndElement(Unknown Source)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
> l$FragmentContentDriver.next(Unknown Source)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(U
> nknown Source)
>        at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
> l.scanDocument(Unknown Source)
>        at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
> nknown Source)
>        at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
> nknown Source)
>        at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown So
> urce)
>        at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Un
> known Source)
>        at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.p
> arse(Unknown Source)
>        at javax.xml.parsers.SAXParser.parse(Unknown Source)
>        at javax.xml.parsers.SAXParser.parse(Unknown Source)
>        at
> com.bretth.osmosis.core.xml.v0_5.XmlReader.run(XmlReader.java:109)
>        at java.lang.Thread.run(Unknown Source)
> 26-Feb-2009 12:46:00 com.bretth.osmosis.core.Osmosis main
> SEVERE: Execution aborted.
> com.bretth.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
>        at
> com.bretth.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pi
> peline.java:146)
>        at com.bretth.osmosis.core.Osmosis.run(Osmosis.java:85)
>        at com.bretth.osmosis.core.Osmosis.main(Osmosis.java:30)
>
> The command line was:
> java -Xmx1024m -jar e:\osmosis-0.30\osmosis.jar --read-xml
> file="planet-090218.osm" enableDateParsing="no" --log-progress
> interval="10" --way-key-value
>
> keyValueList="highway.motorway,highway.motorway_link,highway.trunk,highway.trunk_link,highway.primary,highway.primary_link,highway.secondary,highway.tertiary,highway.unclassified,highway.road,highway.residential,highway.living_street,highway.service,highway.track,highway.pedestrian,highway.bus_guideway,highway.path,highway.cycleway,highway.footway,highway.bridleway,highway.byway,highway.steps,highway.toll_booth,highway.ford,cycleway.lane,cycleway.track,cycleway.opposite_lane,opposite_track,cycleway.opposite,tracktype.grade1,tracktype.grade2,tracktype.grade3,tracktype.grade4,tracktype.grade5,waterway.stream,waterway.river,waterway.riverbank,waterway.canal,waterway.dock,waterway.weir,waterway.dam"
> --used-node --migrate --write-xml-0.6
> file="E:\world\dump\roads\roads.osm"
>
> I am a bit at a loss on why it is failing here.
>
> Emilie Laffray
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20090226/60ecf1ba/attachment.html>


More information about the dev mailing list