[OSM-dev] Slow Osmosis
Andreas Kalsch
andreaskalsch at gmx.de
Sun Nov 30 14:38:18 GMT 2008
OK,
1) does not work
- Osmosis needs the InnoDB tables:
"Write to database ..
Nov 30, 2008 2:35:09 PM com.bretth.osmosis.core.Osmosis main
INFO: Osmosis Version 0.29
Nov 30, 2008 2:35:09 PM com.bretth.osmosis.core.Osmosis main
INFO: Preparing pipeline.
Nov 30, 2008 2:35:09 PM com.bretth.osmosis.core.Osmosis main
INFO: Launching pipeline execution.
Nov 30, 2008 2:35:10 PM com.bretth.osmosis.core.Osmosis main
INFO: Pipeline executing, waiting for completion.
Nov 30, 2008 2:35:26 PM
com.bretth.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
com.bretth.osmosis.core.OsmosisRuntimeException: Unable to insert a
relation into the database.
at
com.bretth.osmosis.core.mysql.v0_5.MysqlWriter.flushRelations(MysqlWriter.java:783)
at
com.bretth.osmosis.core.mysql.v0_5.MysqlWriter.complete(MysqlWriter.java:885)
at
com.bretth.osmosis.core.xml.v0_5.XmlReader.run(XmlReader.java:111)
at java.lang.Thread.run(Thread.java:619)
Caused by:
com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException:
Duplicate entry '1' for key 1
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1585)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1500)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1485)
at
com.bretth.osmosis.core.mysql.v0_5.MysqlWriter.flushRelations(MysqlWriter.java:781)
... 3 more
Nov 30, 2008 2:35:26 PM com.bretth.osmosis.core.Osmosis main
SEVERE: Execution aborted.
com.bretth.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
at
com.bretth.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:141)
at com.bretth.osmosis.core.Osmosis.main(Osmosis.java:55)"
- Although I use AMD Athlon 64 X2 5600+ Dual Core (in a virtual
environment) pbzip2 does not work. You have to use file which have been
compressed with pbzip2 and not bzip2. Are there dumps created with pbzip2?
> What I will try now:
>
> 1) combine("There is another approach. ALTER all InnoDb tables to
> MyISAM, run
>
> osmosis and then ALTER them back to InnoDb.
> It seems that the INSERT approach scales badly to large InnoDb tables.
>
> Sincerely,
> Joachim
>
> ", "It scales always bad; no exceptions. Some advise on the DB techies
> even include to drop any primary and foreign keys (including
> sequences) at inserting. That is optimal.
>
>
> Stefan", "If you are at it, and have a multi processor:
>
> http://compression.ca/pbzip2/
>
>
> Stefan")
>
>
> If this is still slow, I will try
>
> 2) the raw file approach
>
>
> Thank you for your help! I will send you the results.
>
> Andi
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
>
>
More information about the dev
mailing list