[osmosis-dev] Can't load osm file to PostGRESql

Ibrahim Bouchrika ibrahim_bouchrika at hotmail.com
Tue May 4 14:32:46 BST 2010 

Your idea seams nifty, although it's giving an error on the --rx command. I don't know Osmosis well enough to figure out where the error comes from. Is it just an error in the syntax or is it because of the osm-file?


C:\osmosis-0.35\bin>osmosis --rx amsterdam.osm --rx empty.osm --dc --sort-change
-0.6 --simc --wxc amsterdam_fixed.osc
4-mei-2010 13:29:22 org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.35
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskMan
ager waitForCompletion
SEVERE: Thread for task 1-rx failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An output error has occu
rred, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForOutputErrors
(DataPostbox.java:76)
        at org.openstreetmap.osmosis.core.store.DataPostbox.populateCentralQueue
(DataPostbox.java:127)
        at org.openstreetmap.osmosis.core.store.DataPostbox.put(DataPostbox.java
:182)
        at org.openstreetmap.osmosis.core.merge.v0_6.impl.DataPostboxSink.proces
s(DataPostboxSink.java:34)
        at org.openstreetmap.osmosis.core.xml.v0_6.impl.NodeElementProcessor.end
(NodeElementProcessor.java:117)
        at org.openstreetmap.osmosis.core.xml.v0_6.impl.OsmHandler.endElement(Os
mHandler.java:107)
        at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source
)
        at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unkn
own Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElemen
t(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContent
Dispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Un
known Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Sour
ce)
        at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
        at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
        at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:
108)
        at java.lang.Thread.run(Thread.java:619)
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskMan
ager waitForCompletion
SEVERE: Thread for task 2-rx failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file
 empty.osm.  publicId=(null), systemId=(null), lineNumber=1, columnNumber=1.
        at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:
113)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog.
        at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Un
known Source)
        at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)

        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispat
ch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Un
known Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Sour
ce)
        at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
        at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
        at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:
108)
        ... 1 more
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskMan
ager waitForCompletion
SEVERE: Thread for task 3-dc failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: An input error has occur
red, aborting.
        at org.openstreetmap.osmosis.core.store.DataPostbox.checkForInputErrors(
DataPostbox.java:88)
        at org.openstreetmap.osmosis.core.store.DataPostbox.consumeCentralQueue(
DataPostbox.java:158)
        at org.openstreetmap.osmosis.core.store.DataPostbox.hasNext(DataPostbox.
java:235)
        at org.openstreetmap.osmosis.core.change.v0_6.ChangeDeriver.run(ChangeDe
river.java:100)
        at java.lang.Thread.run(Thread.java:619)
4-mei-2010 13:29:23 org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed
.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForComple
tion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Laun
cher.java:329)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:239)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
352)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:31)


Date: Tue, 4 May 2010 22:56:18 +1000
Subject: Re: [osmosis-dev] Can't load osm file to PostGRESql
From: brett at bretth.com
To: ibrahim_bouchrika at hotmail.com
CC: osmosis-dev at openstreetmap.org

On Tue, May 4, 2010 at 10:50 PM, Brett Henderson <brett at bretth.com> wrote:

On Tue, May 4, 2010 at 5:56 PM, Ibrahim Bouchrika <ibrahim_bouchrika at hotmail.com> wrote:







The database i'm using is a fresh made one, so the error must be duplicate entries in the osm. I have the same problem with a duplicate relation in the new york city file i extracted from a cloudmade map.



Could the duplicate values be a result of a bug while extracting a bbox with osmosis?



I doubt it, extracting a bbox should never create multiple instances of the same id.  It's more likely to be caused by incorrect use of the --apply-change task when applying minute or hourly diff files.  It was possible with older versions of osmosis to apply a full history diff containing multiple changes for a single entity which would result in multiple versions of the same entity being written to the result file.


 
Is there a way around those duplicate entries, like perhaps overwriting duplicate data? Going through all those entries manually seems almost impossible.



Hmm, I'm not sure about the best way to tackle this ...

I seem to remember Frederik having a clever way of removing duplicates.  I think he did something like the following:


1. Convert the entire file to an osc file by wrapping the entire contents of the file in osmChange element with an action of create or modify.
2. Feed the osc file through the --simplify-change task.
3. Create an empty osm file, then use the --apply-change task to add the entire change file to it.


Found it.  His steps were:

<quote>
Just for laughs - and wonder if you can come up with something better?
- this is what I did to remove the duplicate objects from an existing
OSM file named "faulty.osm":



echo "<osm></osm>" > empty.osm



osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc good.osc



osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm



(The two osmosis steps could be written as one but that would probably make it more confusing.)
</quote>

Can you try these steps on your cloudmade file and see if that fixes the problem?  If it does, Cloudmade has a problem with duplicate data in their files.  If it doesn't fix it, then we'll have to dig further :-)


Brett

 		 	   		  
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20100504/c4178021/attachment.html>

More information about the osmosis-dev mailing list