[OSM-dev] Osmosis multi-task questions

[Brett Henderson]

Frederik Ramm wrote:
> Ah! I would have thought that the second --bp would immediately
> consume the output created by the first --bp in this case.
>   
If that is more intuitive then I should move to a stack implementation 
sooner rather than later ...
>> With a decent machine, I think you'll find file based performance more 
>> than adequate.  A file will be far quicker than using MySQL as raw data 
>> storage without using any indexing.  If bounding boxes could be 
>> extracted directly from the db it may be quicker, it's not something 
>> I've tried.  A new mysql task for reading bounding boxes directly from a 
>> db could be very useful, patches welcome of course :-)
>>     
>
> I'll see if I can come up with something. From past experience I fear
> that even inserting a planet into Mysql might take more time than the
> whole file-based operation, in which case it might not be worth the
> bother. But my setup will give us interesting test scenarios at least.
>   
 From my own awful experience with MySQL I think you're right.  I've 
spent many many hours trying to improve insert performance on MySQL and 
have come to the conclusion that you can either have fast performance 
with MyISAM tables at the expensive of all other enterprise features, or 
decent feature support in InnoDB at the expensive of decent 
performance.  It doesn't seem to matter what I try, as InnoDB tables get 
bigger, they get much slower.  Indexing is possibly the issue but even 
disabling indexes doesn't help greatly.  At one point a planet import 
took 1 hour to import history tables and then another 3 hours to 
populate the current tables but that was with a sub 5GB planet.  I 
suspect a total planet import would now be pushing 20 hours on my 
hardware ...

In your case it might be possible to create a new schema optimised for 
this purpose (ie. no history tables and MyISAM current tables) but 
that's a lot of work.  I'd test out file performance first and see if 
that solves your problem.  Hopefully you can go with the "not bother" 
approach ;-)

Brett