[OSM-dev] Fwd: Re: OSM and MongoDB

[Nolan Darilek]

Oopse, meant for this to go to the whole list.



-------- Original Message --------
Subject: 	Re: [OSM-dev] OSM and MongoDB
Date: 	Tue, 12 Apr 2011 15:26:41 -0500
From: 	Nolan Darilek <nolan at thewordnerd.info>
To: 	Ian Dees <ian.dees at gmail.com>



I had/am having a somewhat bad experience storing OSM data in MongoDB.

Initially I stored all map data in MongoDB, but queries took ages. The 
same queries that happen in 100-200 MS now often took nearly a second. 
Additionally, some took upwards of 5, and I even found spots on my map 
sparsely populated with points, but which reliably performed the queries 
I need in 30+ seconds.

I filed a thorough bug in their tracker, including a dataset and queries 
that reliably duplicated the issue. It was marked wontfix, I abandoned 
MongoDB, and it was apparently re-opened and fixed several months later. 
So perhaps it's a non-issue now.

I'm still using MongoDB for part of my current project, user POI 
storage. It does indeed use geohashes, and I'm experiencing strange 
accuracy issues. My platform is pedestrian navigation with many small 
distance queries. Points in the non-MongoDB dataset are reliably 
detected in a radius roughly 100 meters around the traveler. Points in 
MongoDB queried with the same bounding boxes don't appear until they're 
within 30-40 meters. I recently updated from an older version to a new 
build of 1.8. The older version widely varied the detection range. Some 
points were detected 100 or so meters out, while others weren't picked 
up until 30 or so. It was always the same points, too. The point for my 
apartment remains reliably visible for ~100 meters or so, while the 
corner store and restaurant didn't appear until I was very close. 1.8 at 
least appears to be consistent, always detecting at 30 meters or so. I 
can only assume that this is a geohash oddity that only appears for very 
small differences, something that works out to rounding error for larger 
values.

I like MongoDB for many things, but not for geospatial data more 
complicated than a series of points. I'm working on migrating user/POI 
storage to a geospatial store.


On 04/12/2011 01:20 PM, Ian Dees wrote:
> Yep, and I think Mongo uses geohashes as their index behind the 
> scenes. One of the problems with that, though, is they have some 
> arbitrary length that they compute the geohash to and when you have 
> lots of points (as OSM data does) the buckets they're searching are 
> very full.
>
> On Tue, Apr 12, 2011 at 1:00 PM, Steve Coast <steve at asklater.com 
> <mailto:steve at asklater.com>> wrote:
>
>     bbox queries using the built in spatial indexing presumably? OSM
>     has it's own magical bitmask for that, that may also be as fast in
>     mongo, who knows.
>
>
>     On 4/11/2011 5:58 PM, Ian Dees wrote:
>>     On Mon, Apr 11, 2011 at 6:36 PM, Sergey Galuzo
>>     <sergal at microsoft.com <mailto:sergal at microsoft.com>> wrote:
>>
>>         Hi,
>>
>>         I am working on evaluation of MongoDB for several storage
>>         solutions at hand. Some of them resemble current OSM editing
>>         database. I have heard that OSM dev is/was evaluating MongoDB
>>         also. I was wondering whether it possible to share the findings?
>>
>>
>>     In my experimentation with MongoDB (seen here:
>>     https://github.com/iandees/mongosm/) I found it to be very slow.
>>     Inserts were speedy, but bounding-box queries took a long time.
>>
>>     The most recent dev version of MongoDB includes "multi-location
>>     documents" support:
>>     http://www.mongodb.org/display/DOCS/Geospatial+Indexing#GeospatialIndexing-MultilocationDocuments
>>
>>     This would allow a single way document to be indexed at multiple
>>     locations and vastly speed up the map query.
>>
>>
>>     _______________________________________________
>>     dev mailing list
>>     dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>>     http://lists.openstreetmap.org/listinfo/dev
>
>     _______________________________________________
>     dev mailing list
>     dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>     http://lists.openstreetmap.org/listinfo/dev
>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20110412/5c37b416/attachment-0001.html>