[OSM-dev] Faster loading with scabies

[Sascha Silbe]

On Thu, Oct 16, 2008 at 09:21:54AM +1100, Brett Henderson wrote:

> The biggest problem I found wasn't the actual processing of INSERT
> statements, it was MySQL scaling non-linearly with the number of rows.
> MyISAM tables are very fast to import regardless of number of rows, 
> but
> InnoDB seems to slow down as the number of rows increase.  I'm 
> surprised
> loading with LOAD DATA INFILE fixes that.
Just a quick stab in the dark as I do know almost nothing about osmosis 
and MySQL:
If you create the indices before loading data, they will be constantly 
rebuilt as you add each row => would explain the exponential slow-down. 
The PostgreSQL manual has a section about optimizing database imports 
giving some useful tips & tricks; I'd guess the same is true for the 
MySQL one.

If osmosis already defers index creation (including referential 
constraints which usually build indices implicitly), just ignore this 
mail. :)

CU Sascha

-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20081016/6fc72d66/attachment.pgp>