[osmosis-dev] Crash on processing nodes?

[Brett Henderson]

On Wed, Jan 20, 2010 at 11:24 PM, Simon Nuttall <simon.nuttall at gmail.com>wrote:

> On one of the CycleStreets machines it takes 17.3 hours to extract and
> merge data from the Britain and Ireland GeoFabrik planets using:
>
> osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm
> enableDateParsing=no  --sort-0.6 type="TypeThenId" --read-xml
> data/osm/downloads/ireland.osm enableDateParsing=no  --sort-0.6
> type="TypeThenId" --merge --write-apidb dbType=mysql host=localhost
> database=britainOSM user=import password=xxx validateSchemaVersion=no
>
> (On our main machine the same only takes about 5 hours.)
>
> During that time it builds an 11 GB innodb file in /var/lib/mysql/ibdata1
>
>
> Osmosis fills the britainOSM database, which I then treat as 'read
> only' to build the CycleStreets routing tables in a separate database.
>
> As I'm only filling a database and then treating it as read-only do I
> really need all the transaction processing that is provided by innodb?
> Is there a way of turning those features off to speed up all the
> processing and use less disk space?
>

You could possibly add populateCurrentTables=no to the --write-apidb task
and use the history tables for your queries.  After a planet import they
contain identical data to the current tables.  Are the history tables still
MyISAM or are they InnoDB?  I haven't used MySQL for a while.  If the
history tables are MyISAM and you disable current table population you
should get very large performance speedups.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20100121/d9a0c3a6/attachment.html>