[OSM-dev] Slow Osmosis

[Stefan de Konink]

Brett Henderson wrote:
> Stefan de Konink wrote:
>> Brett Henderson wrote:
>>> If somebody can prove with real numbers on very large files that LOAD 
>>> DATA INFILE is much faster than the current osmosis approach of doing 
>>> normal inserts (ie. not just a few per cent) I'll add it to osmosis.  
>>> I just haven't done so because my efforts have been focused elsewhere.
>>
>> Oki Brett, if we make the benchmark it would basically make your tool 
>> obsolete for the full insert ;) But I'm to provide a CVS file for the 
>> MySQL schema so that some people can eat their heart out.
> Only if people are happy to download a separate tool for every 
> individual purpose.  The advantage of osmosis is that it combines lots 
> of operations into a single tool.  I haven't invented much new 
> functionality, simply looked at what a bunch of individual scripts have 
> done and made them easier to access.

The point being, I guess there should be only one time where importing 
the full set is important. The other times diffs in xml format could be 
just 'pushed' as if they were inserted in the normal way, right?

> Quite seriously, if you can write a tool that is more than 20% faster 
> than osmosis then by all means go for it.  If it suits people's needs 
> I'll eventually roll similar functionality into osmosis.

If you provide a mean so we can benchmark I am happy to do so. As I told 
you my tool doesn't use streams, so the one pro of your tool over mine 
is still that decompression can't be done on the fly, and thus diskspace 
is required.

I'm unhappy with how currently tags are stored. And honestly I'm not 
going to spend time trying to 'break' that. We can do a comparison, but 
not on the tag point.

> Of course I'd be far happier if you could send me a patch to improve 
> osmosis functionality but that's your choice.

...last time I did Java was in May... I can only do it for a pretty girl ;)


Stefan