[OSM-dev] Osmosis Failing to Read Particular Replicate Files

Brett Henderson brett at bretth.com
Thu Apr 8 13:45:24 BST 2010 

Hi Aaron,

Did you get any further with this?  The error is unlikely to be caused by
Osmosis.  The following line in the stack trace appears to indicate a
network problem:
Caused by: java.net.SocketException: Connection reset

I've tried the same command on the planet server itself (downloading from
the local Apache server) and it appears to have downloaded all files without
an issue.  I started from replication file 3216 which I believe is the one
you started from.  It's actually still running because it takes a while to
sort the data after the download but it's completed the download portion of
the execution okay.

I'm also running the command locally at home.  It's taking a long time to
run due to the time taken to download a large number of hourly files.  It
hasn't reached the same point you encountered errors yet, but so far so
good.  I'll let you know if I see the same problem.

There are a few things you can try.

   - I never set the maxInterval to 0.  It is worth setting to a smaller
   time period so that if encounter problems you don't have to start from the
   start.  You've only downloaded about 30 files which is just over a day when
   encountering the error though so this isn't a huge number.  But the more
   files you download, the greater the chances of tripping over a network
   issue.  I'd suggest setting it to one day (ie. 86400) and running it several
   times until you catch up.  If you're importing into a database then there's
   no reason to try to get it all in one hit.  Patching a planet file is more
   time consuming, but if you're doing that you probably should be using daily
   diffs anyway.
   - Have you tried using wget or curl to download the file?  It would be
   worth verifying that you can download it outside of Osmosis.

Hope that helps.
Brett

On Wed, Apr 7, 2010 at 3:43 AM, Aaron Schaefer <aaron at elasticdog.com> wrote:

> So I'm trying to set up downloads of the hour-replicate diffs and
> osmosis seems to be having issues reading certain files from the
> server. I did a full planet download and dump into my database from
> the 03-Apr-2010 03:38 planet-latest.osm.bz2.
>
> I created a working directory for the diffs and ran 'osmosis --rri
> workingDirectory=.' to initialize the necessary files and then
> downloaded the Fri Apr 02 23:02:09 UTC 2010 state.txt file (a few
> hours earlier than my previous full import). I updated
> configuration.txt to have the proper baseUrl and set the maxInterval =
> 0. Now when I try to get the diffs, this is what happens:
>
> --------
> $  osmosis --rri workingDirectory=. --wxc diffs.osc.gz
> Apr 6, 2010 12:21:07 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Osmosis Version 0.34
> Apr 6, 2010 12:21:08 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Preparing pipeline.
> Apr 6, 2010 12:21:08 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Launching pipeline execution.
> Apr 6, 2010 12:21:08 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Pipeline executing, waiting for completion.
> Apr 6, 2010 12:25:26 PM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion
> SEVERE: Thread for task 1-rri failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to read
> the changeset file 000/003/246.osc.gz from t
> he server.
>        at
> org.openstreetmap.osmosis.core.merge.v0_6.BaseReplicationDownloader.downloadReplicationFile(BaseReplicat
> ionDownloader.java:128)
>        at
> org.openstreetmap.osmosis.core.merge.v0_6.BaseReplicationDownloader.download(BaseReplicationDownloader.j
> ava:236)
>        at
> org.openstreetmap.osmosis.core.merge.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.ja
> va:275)
>        at
> org.openstreetmap.osmosis.core.merge.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:3
> 45)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.net.SocketException: Connection reset
>        at java.net.SocketInputStream.read(Unknown Source)
>        at java.io.BufferedInputStream.read1(Unknown Source)
>        at java.io.BufferedInputStream.read(Unknown Source)
>        at sun.net.www.MeteredStream.read(Unknown Source)
>        at java.io.FilterInputStream.read(Unknown Source)
>        at
> sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown
> Source)
>        at java.io.BufferedInputStream.read1(Unknown Source)
>        at java.io.BufferedInputStream.read(Unknown Source)
>        at java.io.FilterInputStream.read(Unknown Source)
>        at
> org.openstreetmap.osmosis.core.merge.v0_6.BaseReplicationDownloader.downloadReplicationFile(BaseReplicationDownloader.java:114)
>        ... 4 more
> Apr 6, 2010 12:25:26 PM org.openstreetmap.osmosis.core.Osmosis main
> SEVERE: Execution aborted.
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more
> tasks failed.
>        at
> org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
>        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85)
>        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>        at java.lang.reflect.Method.invoke(Unknown Source)
>        at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
>        at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
>        at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
>        at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
>        at org.codehaus.classworlds.Launcher.main(Launcher.java:31)
> --------
>
> It always fails on the 000/003/246.osc.gz file, and I've verified that
> it does exist on the server. I tried using both
> planet.openstreetmap.org and the heanet mirror with the same results.
> I had actually tried this process previously with an older full planet
> dump, and it was giving me an error on the 000/003/014.osc.gz file
> (which was also present on the server)...I figured maybe it was just a
> fluke and that particular file had been corrupted, but now I'm seeing
> it error out on the more recent diff as well.
>
> Is anyone else seeing these errors when trying to generate a diff file?
>
> --
> Aaron Schaefer
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100408/d440e5f6/attachment.html>

More information about the dev mailing list