[osmosis-dev] Osmosis error

[Brett Henderson]

Hi Laura,

On 15 October 2012 19:59, Laura Passigni <laurapassigni at yahoo.it> wrote:

>
>   Hi,
> I have a database openstreetmap like and often I used osmosis to retrieve
> data and update tiles.
>
> Recently I tried to export from the database with a command like the
> following:
>
> osmosis --rdc host='xxx' database='xxx' user='xxx' password="xxx"
> validateSchemaVersion="no" \
> intervalBegin="2012-10-09_11:43:00" intervalEnd="2012-10-11_13:55:00"
> --write-xml-change file="export.osc"
> But I received the following error, after about 8 hours:
>
> Oct 12, 2012 7:24:42 AM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Pipeline executing, waiting for completion.
> Oct 12, 2012 3:37:12 PM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion
> SEVERE: Thread for task 1-rdc failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to create
> object stream writing to temporary file null.
>         at
> org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:100)
>         at
> org.openstreetmap.osmosis.core.store.PersistentIterator.initialize(PersistentIterator.java:56)
>         at
> org.openstreetmap.osmosis.core.store.PersistentIterator.hasNext(PersistentIterator.java:72)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iteratePersisted(FileBasedSort.java:159)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:241)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:245)
>         at
> org.openstreetmap.osmosis.core.sort.common.FileBasedSort.iterate(FileBasedSort.java:277)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.impl.WayDao.getWayNodeHistory(WayDao.java:121)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.impl.WayDao.getFeatureHistoryPopulators(WayDao.java:152)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.impl.EntityDao.getEntityHistory(EntityDao.java:259)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.impl.EntityDao.getChangeHistory(EntityDao.java:281)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.impl.EntityDao.getHistory(EntityDao.java:496)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.impl.AllEntityDao.getHistory(AllEntityDao.java:90)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.ApidbChangeReader.runImpl(ApidbChangeReader.java:84)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.ApidbChangeReader$1.doInTransactionWithoutResult(ApidbChangeReader.java:117)
>         at
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
>         at
> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
>         at
> org.openstreetmap.osmosis.apidb.common.DatabaseContext2.executeWithinTransaction(DatabaseContext2.java:102)
>         at
> org.openstreetmap.osmosis.apidb.v0_6.ApidbChangeReader.run(ApidbChangeReader.java:112)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: Too many open files
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.checkAndCreate(File.java:1717)
>         at java.io.File.createTempFile0(File.java:1738)
>         at java.io.File.createTempFile(File.java:1850)
>         at
> org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:82)
>         ... 25 more
> Oct 12, 2012 3:37:12 PM org.openstreetmap.osmosis.core.Osmosis main
> SEVERE: Execution aborted.
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks
> failed.
>         at
> org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
>         at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
>         at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
>         at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
>         at org.codehaus.classworlds.Launcher.main(Launcher.java:31)
>
>
>
>
>
>
>
> Could you please help me?
> Please note that I increased also the number of open files to 32758
> (before it was 1024), but I received the same error.
> Thank you,
> Laura Passigni
>

There's possibly a bug lurking in here somewhere, although the
FileBasedSort code is quite old so I'm surprised to see this for the first
time..  Can you tell me how large your database is?  In particular how many
records exist in the waynode table?  The FileBasedSort can use a fair
number of file descriptors, although I thought it would limit itself to a
sensible number.  I'll need to dig into the code again.  It's also possible
that open files are leaking (ie. not being closed correctly).

In short, I don't have a quick fix, but if you can provide me with some
numbers I may be able to reproduce the problem.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20121019/a96a1356/attachment.html>