[OSM-dev] Import planet.osm to rails

[Brett Henderson]

I don't think I've seen that specific error but I get timeout errors 
when trying to dump large data sets.  The --read-mysql task still needs 
work.  I've just imported a full planet into my test databases and am 
planning to get it working over the next couple of days.

Can you use planet.rb to dump the database in the meantime?

Osmosis and planet.rb use a different technique for dumping.  Osmosis 
attempts to generate consistent snapshots of the database at a specific 
point in time which is essential for initialising replication.  I assume 
planet.rb simply reads the "current" tables one after the other which 
leads to inconsistencies between nodes, segments and ways when run 
against a live database.  The snapshot queries run by Osmosis against 
the larger history tables are more difficult to tune, particularly for 
ways where three tables are involved.

If you are the only one accessing the database, planet.rb should (note I 
haven't tested any of this) be suitable.

Osmosis has four tasks for mysql replication; snapshot dumps, snapshot 
loads, change derivation and change application.  The only one not 
working properly is the dump task, it is high on my priority list.

Al Wold wrote:
> Has anyone seen this error?  It seems to happen once I get a decent 
> amount of data into my db.  It worked this morning when I had a small 
> subset of my county's TIGER data, but now I imported the full county 
> and it won't dump successfully.  It also happened with v0.5, so it 
> isn't a new thing.
>
> I definitely think mysql is to blame, but there isn't much on google 
> about this error.
>
> -Al
>
> Exception in thread "Thread-1-read-mysql" 
> com.bretth.osmosis.OsmosisRuntimeException : Unable to move to next 
> record.
>         at 
> com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(EntityReader.java:79)
>         at 
> com.bretth.osmosis.mysql.impl.EntityReader.next(EntityReader.java:144)
>         at 
> com.bretth.osmosis.mysql.MysqlReader.processWays(MysqlReader.java:129)
>         at com.bretth.osmosis.mysql.MysqlReader.run(MysqlReader.java:168)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: com.mysql.jdbc.CommunicationsException : Communications 
> link failure due to underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
> MESSAGE: Can not read response from server. Expected to read 4 bytes, 
> read 1 bytes before connection was unexpectedly lost.
>
> STACKTRACE:
>
> java.io.EOFException: Can not read response from server. Expected to 
> read 4 bytes, read 1 bytes before connection was unexpectedly lost.
>         at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java :1997)
>         at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java :885)
>         at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)
>         at 
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:366)
>         at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:355)
>         at com.mysql.jdbc.ResultSet.next(ResultSet.java:7302)
>         at 
> com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(EntityReader.java:72)
>         at com.bretth.osmosis.mysql.impl.EntityReader.next( 
> EntityReader.java:144)
>         at 
> com.bretth.osmosis.mysql.MysqlReader.processWays(MysqlReader.java:129)
>         at com.bretth.osmosis.mysql.MysqlReader.run(MysqlReader.java:168)
>         at java.lang.Thread.run(Thread.java :619)
>
>
> ** END NESTED EXCEPTION **
>
>
>
> Last packet sent to the server was 435582 ms ago.
>         at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2622)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket (MysqlIO.java:2916)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
>         at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)
>         at 
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java :366)
>         at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:355)
>         at com.mysql.jdbc.ResultSet.next(ResultSet.java:7302)
>         at 
> com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(EntityReader.java 
> :72)
>         ... 4 more
>