[OSM-dev] Import planet.osm to rails

[Al Wold]

Yeah, I will use planet.rb.  I turned on general query logging on mysql, but
it didn't show any errors.  I'll keep poking around and let you know if I
find a fix.

-Al

On 8/2/07, Brett Henderson <brett at bretth.com> wrote:
>
> I don't think I've seen that specific error but I get timeout errors
> when trying to dump large data sets.  The --read-mysql task still needs
> work.  I've just imported a full planet into my test databases and am
> planning to get it working over the next couple of days.
>
> Can you use planet.rb to dump the database in the meantime?
>
> Osmosis and planet.rb use a different technique for dumping.  Osmosis
> attempts to generate consistent snapshots of the database at a specific
> point in time which is essential for initialising replication.  I assume
> planet.rb simply reads the "current" tables one after the other which
> leads to inconsistencies between nodes, segments and ways when run
> against a live database.  The snapshot queries run by Osmosis against
> the larger history tables are more difficult to tune, particularly for
> ways where three tables are involved.
>
> If you are the only one accessing the database, planet.rb should (note I
> haven't tested any of this) be suitable.
>
> Osmosis has four tasks for mysql replication; snapshot dumps, snapshot
> loads, change derivation and change application.  The only one not
> working properly is the dump task, it is high on my priority list.
>
> Al Wold wrote:
> > Has anyone seen this error?  It seems to happen once I get a decent
> > amount of data into my db.  It worked this morning when I had a small
> > subset of my county's TIGER data, but now I imported the full county
> > and it won't dump successfully.  It also happened with v0.5, so it
> > isn't a new thing.
> >
> > I definitely think mysql is to blame, but there isn't much on google
> > about this error.
> >
> > -Al
> >
> > Exception in thread "Thread-1-read-mysql"
> > com.bretth.osmosis.OsmosisRuntimeException : Unable to move to next
> > record.
> >         at
> > com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(
> EntityReader.java:79)
> >         at
> > com.bretth.osmosis.mysql.impl.EntityReader.next(EntityReader.java:144)
> >         at
> > com.bretth.osmosis.mysql.MysqlReader.processWays(MysqlReader.java:129)
> >         at com.bretth.osmosis.mysql.MysqlReader.run(MysqlReader.java
> :168)
> >         at java.lang.Thread.run(Thread.java:619)
> > Caused by: com.mysql.jdbc.CommunicationsException : Communications
> > link failure due to underlying exception:
> >
> > ** BEGIN NESTED EXCEPTION **
> >
> > java.io.EOFException
> > MESSAGE: Can not read response from server. Expected to read 4 bytes,
> > read 1 bytes before connection was unexpectedly lost.
> >
> > STACKTRACE:
> >
> > java.io.EOFException: Can not read response from server. Expected to
> > read 4 bytes, read 1 bytes before connection was unexpectedly lost.
> >         at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java :1997)
> >         at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)
> >         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
> >         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java :885)
> >         at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)
> >         at
> > com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:366)
> >         at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:355)
> >         at com.mysql.jdbc.ResultSet.next(ResultSet.java:7302)
> >         at
> > com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(
> EntityReader.java:72)
> >         at com.bretth.osmosis.mysql.impl.EntityReader.next(
> > EntityReader.java:144)
> >         at
> > com.bretth.osmosis.mysql.MysqlReader.processWays(MysqlReader.java:129)
> >         at com.bretth.osmosis.mysql.MysqlReader.run(MysqlReader.java
> :168)
> >         at java.lang.Thread.run(Thread.java :619)
> >
> >
> > ** END NESTED EXCEPTION **
> >
> >
> >
> > Last packet sent to the server was 435582 ms ago.
> >         at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2622)
> >         at com.mysql.jdbc.MysqlIO.checkErrorPacket (MysqlIO.java:2916)
> >         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
> >         at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)
> >         at
> > com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java :366)
> >         at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:355)
> >         at com.mysql.jdbc.ResultSet.next(ResultSet.java:7302)
> >         at
> > com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(
> EntityReader.java
> > :72)
> >         ... 4 more
> >
>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20070802/5cddb9bd/attachment.html>