[OSM-dev] Slow Osmosis

Andreas Kalsch andreaskalsch at gmx.de
Sun Nov 30 14:38:18 GMT 2008


OK,

1) does not work
- Osmosis needs the InnoDB tables:

"Write to database ..
Nov 30, 2008 2:35:09 PM com.bretth.osmosis.core.Osmosis main
INFO: Osmosis Version 0.29
Nov 30, 2008 2:35:09 PM com.bretth.osmosis.core.Osmosis main
INFO: Preparing pipeline.
Nov 30, 2008 2:35:09 PM com.bretth.osmosis.core.Osmosis main
INFO: Launching pipeline execution.
Nov 30, 2008 2:35:10 PM com.bretth.osmosis.core.Osmosis main
INFO: Pipeline executing, waiting for completion.
Nov 30, 2008 2:35:26 PM 
com.bretth.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
com.bretth.osmosis.core.OsmosisRuntimeException: Unable to insert a 
relation into the database.
       at 
com.bretth.osmosis.core.mysql.v0_5.MysqlWriter.flushRelations(MysqlWriter.java:783) 

       at 
com.bretth.osmosis.core.mysql.v0_5.MysqlWriter.complete(MysqlWriter.java:885) 

       at 
com.bretth.osmosis.core.xml.v0_5.XmlReader.run(XmlReader.java:111)
       at java.lang.Thread.run(Thread.java:619)
Caused by: 
com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: 
Duplicate entry '1' for key 1
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985)
       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)
       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
       at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)
       at 
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313) 

       at 
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1585)
       at 
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1500)
       at 
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1485)
       at 
com.bretth.osmosis.core.mysql.v0_5.MysqlWriter.flushRelations(MysqlWriter.java:781) 

       ... 3 more
Nov 30, 2008 2:35:26 PM com.bretth.osmosis.core.Osmosis main
SEVERE: Execution aborted.
com.bretth.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
       at 
com.bretth.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:141) 

       at com.bretth.osmosis.core.Osmosis.main(Osmosis.java:55)"

- Although I use AMD Athlon 64 X2 5600+ Dual Core (in a virtual 
environment) pbzip2 does not work. You have to use file which have been 
compressed with pbzip2 and not bzip2. Are there dumps created with pbzip2?

> What I will try now:
>
> 1) combine("There is another approach. ALTER all InnoDb tables to 
> MyISAM, run
>
> osmosis and then ALTER them back to InnoDb.
> It seems that the INSERT approach scales badly to large InnoDb tables.
>
> Sincerely,
> Joachim
>
> ", "It scales always bad; no exceptions. Some advise on the DB techies 
> even include to drop any primary and foreign keys (including 
> sequences) at inserting. That is optimal.
>
>
> Stefan", "If you are at it, and have a multi processor:
>
> http://compression.ca/pbzip2/
>
>
> Stefan")
>
>
> If this is still slow, I will try
>
> 2) the raw file approach
>
>
> Thank you for your help! I will send you the results.
>
> Andi
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
>
>   






More information about the dev mailing list