[OSM-dev] OSM and CouchDB/GeoCouch

Nolan Darilek nolan at thewordnerd.info
Sun Jul 4 18:17:27 BST 2010


Hmm, I just checked out your code. Out of curiosity, what hardware did 
you run it on where importing a planet took several days?

It looks like our schemas are mostly identical, but in my experience, 
MongoDB used more and more time importing the index as the import 
continued. An import of the dataset for TX took several hours, but 
import speeds dropped off markably as the import continued and, 
presumably, as the geoindex grew.

I wondered if perhaps MongoDB's geospatial support wasn't a good fit for 
mapping apps, especially as I experienced some very long query times for 
no reason I could determine, but now I'm wondering if the slow imports 
at least might be due to some hidden detail in my own implementation.

Also, how much disk space did the final import use?

On 07/03/2010 03:43 PM, Ian Dees wrote:
> On Sat, Jul 3, 2010 at 1:17 PM, Nolan Darilek <nolan at thewordnerd.info 
> <mailto:nolan at thewordnerd.info>> wrote:
>
>     On 07/03/2010 01:09 PM, Nolan Darilek wrote:
>
>         On 07/02/2010 01:52 PM, Serge Wroclawski wrote:
>
>             Similarly, Ian Dees and I have written a server using
>             MongoDB, which
>             also provides functionality such as auto-sharding and built in
>             map/reduce.
>
>
>         Is this work available anywhere? How did you find performance
>         to be, and to what uses did you put it?
>
>         I've done some experiments creating a LibOSM MongoDB backend
>         and found its performance fairly bad, but I don't have the
>         most optimal server for it, and probably didn't use MongoDB to
>         its limits. If you experienced good performance for real-time
>         operations then I'd be very interested in seeing how you
>         managed it so I might adopt the techniques and see if I have
>         any better luck.
>
>         It seemed to me that a dump of the entire planet would require
>         a substantial server to serve up, so I abandoned the work, but
>         would very much like to revive it if it's at all workable.
>
>
> The code is here: http://github.com/iandees/mongosm
>
> It took several days to import a planet file. The majority of the CPU 
> time was spent serializing/deserializing BSON in Python and the Mongo 
> server had very little CPU time so if I use a language with a faster 
> BSON implementation it might be faster.
>
> Serge was working on a way to import diff files to maintain mintutely 
> updates.
>
> I was working on an HTTP API interface.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100704/73334523/attachment.html>


More information about the dev mailing list