[OSM-dev] Import planet.osm to rails

[Al Wold]

Has anyone seen this error?  It seems to happen once I get a decent amount
of data into my db.  It worked this morning when I had a small subset of my
county's TIGER data, but now I imported the full county and it won't dump
successfully.  It also happened with v0.5, so it isn't a new thing.

I definitely think mysql is to blame, but there isn't much on google about
this error.

-Al

Exception in thread "Thread-1-read-mysql"
com.bretth.osmosis.OsmosisRuntimeException: Unable to move to next record.
        at com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(
EntityReader.java:79)
        at com.bretth.osmosis.mysql.impl.EntityReader.next(EntityReader.java
:144)
        at com.bretth.osmosis.mysql.MysqlReader.processWays(MysqlReader.java
:129)
        at com.bretth.osmosis.mysql.MysqlReader.run(MysqlReader.java:168)
        at java.lang.Thread.run(Thread.java:619)
Caused by: com.mysql.jdbc.CommunicationsException: Communications link
failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException
MESSAGE: Can not read response from server. Expected to read 4 bytes, read 1
bytes before connection was unexpectedly lost.

STACKTRACE:

java.io.EOFException: Can not read response from server. Expected to read 4
bytes, read 1 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1997)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:366)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:355)
        at com.mysql.jdbc.ResultSet.next(ResultSet.java:7302)
        at com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(
EntityReader.java:72)
        at com.bretth.osmosis.mysql.impl.EntityReader.next(EntityReader.java
:144)
        at com.bretth.osmosis.mysql.MysqlReader.processWays(MysqlReader.java
:129)
        at com.bretth.osmosis.mysql.MysqlReader.run(MysqlReader.java:168)
        at java.lang.Thread.run(Thread.java:619)


** END NESTED EXCEPTION **



Last packet sent to the server was 435582 ms ago.
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2622)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:366)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:355)
        at com.mysql.jdbc.ResultSet.next(ResultSet.java:7302)
        at com.bretth.osmosis.mysql.impl.EntityReader.readNextValue(
EntityReader.java:72)
        ... 4 more


On 8/2/07, Brett Henderson <brett at bretth.com> wrote:
>
> Dave Stubbs wrote:
> > Ah, yes, that makes sense. Thanks.
> >
> > I'm not in any hurry. I can continue using the hacked up "My 1st Rails
> > Hack" map created using JOSM using nodes segments and ways. The "M" is
> > a motorway and the "1" is oneway residential if anyone is
> > interested... I've even underlined the lot with a river. :-)
> >
> Well, that didn't take as long as I thought.
>
> This should do the trick ...
> http://www.bretth.com/osmosis/osmosis-latest.zip
>
> It creates a new user with email address "osmosis at bretth.com" (hardcoded
> for now) which will be used for all edits.
>
> Overview here:
> http://wiki.openstreetmap.org/index.php/Osmosis
>
> Full command line details here (must migrate this to OSM wiki one of
> these days):
> http://www.bretth.com/wiki/Wiki.jsp?page=OpenStreetMap
>
> If you want to import the whole planet though, patience will be
> required.  You're looking at around 4 hours for the import, 3 hours of
> which seem to be populating the "current" tables.  If anybody has any
> relatively straightforward suggestions on increasing the speed of
> copying data into InnoDB tables I'm all ears.  No simple way of
> disabling indexes (that I can find), and I have to chunk into
> "reasonable" sized transactions or nasty runaway rollbacks will occur if
> the process is interrupted.
>
> If you want to limit the size of the import, try using the
> --bounding-box task to extract the area you want (I often use Australia
> for testing, approx 120MB).  If dates aren't important, you can further
> speed things up by applying the enableDateParsing=no option to the
> --read-xml task.
>
> I'm currently running an import on the August 25 planet to test it
> properly but it will take a while to finish.
>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20070802/012bc939/attachment.html>