[OSM-dev] Fwd: Re: OSM and MongoDB
Steve Coast
steve at asklater.com
Tue Apr 12 21:56:58 BST 2011
Interesting.
How efficient is the (big)int indexing and/or masking?
Was this all on a single machine?
On 4/12/2011 1:52 PM, Ian Dees wrote:
> Yep.
>
> On Tue, Apr 12, 2011 at 3:51 PM, Steve Coast <steve at asklater.com
> <mailto:steve at asklater.com>> wrote:
>
> and using the builtin spatial index?
>
>
>
> On 4/12/2011 1:50 PM, Ian Dees wrote:
>> Yes, one document per node/way/relation.
>>
>> On Tue, Apr 12, 2011 at 3:47 PM, Steve Coast <steve at asklater.com
>> <mailto:steve at asklater.com>> wrote:
>>
>> how was the data put in the db though? 1 document per node?
>>
>>
>> On 4/12/2011 1:39 PM, Nolan Darilek wrote:
>>> Oopse, meant for this to go to the whole list.
>>>
>>>
>>>
>>> -------- Original Message --------
>>> Subject: Re: [OSM-dev] OSM and MongoDB
>>> Date: Tue, 12 Apr 2011 15:26:41 -0500
>>> From: Nolan Darilek <nolan at thewordnerd.info>
>>> <mailto:nolan at thewordnerd.info>
>>> To: Ian Dees <ian.dees at gmail.com> <mailto:ian.dees at gmail.com>
>>>
>>>
>>>
>>> I had/am having a somewhat bad experience storing OSM data
>>> in MongoDB.
>>>
>>> Initially I stored all map data in MongoDB, but queries took
>>> ages. The same queries that happen in 100-200 MS now often
>>> took nearly a second. Additionally, some took upwards of 5,
>>> and I even found spots on my map sparsely populated with
>>> points, but which reliably performed the queries I need in
>>> 30+ seconds.
>>>
>>> I filed a thorough bug in their tracker, including a dataset
>>> and queries that reliably duplicated the issue. It was
>>> marked wontfix, I abandoned MongoDB, and it was apparently
>>> re-opened and fixed several months later. So perhaps it's a
>>> non-issue now.
>>>
>>> I'm still using MongoDB for part of my current project, user
>>> POI storage. It does indeed use geohashes, and I'm
>>> experiencing strange accuracy issues. My platform is
>>> pedestrian navigation with many small distance queries.
>>> Points in the non-MongoDB dataset are reliably detected in a
>>> radius roughly 100 meters around the traveler. Points in
>>> MongoDB queried with the same bounding boxes don't appear
>>> until they're within 30-40 meters. I recently updated from
>>> an older version to a new build of 1.8. The older version
>>> widely varied the detection range. Some points were detected
>>> 100 or so meters out, while others weren't picked up until
>>> 30 or so. It was always the same points, too. The point for
>>> my apartment remains reliably visible for ~100 meters or so,
>>> while the corner store and restaurant didn't appear until I
>>> was very close. 1.8 at least appears to be consistent,
>>> always detecting at 30 meters or so. I can only assume that
>>> this is a geohash oddity that only appears for very small
>>> differences, something that works out to rounding error for
>>> larger values.
>>>
>>> I like MongoDB for many things, but not for geospatial data
>>> more complicated than a series of points. I'm working on
>>> migrating user/POI storage to a geospatial store.
>>>
>>>
>>> On 04/12/2011 01:20 PM, Ian Dees wrote:
>>>> Yep, and I think Mongo uses geohashes as their index behind
>>>> the scenes. One of the problems with that, though, is they
>>>> have some arbitrary length that they compute the geohash to
>>>> and when you have lots of points (as OSM data does) the
>>>> buckets they're searching are very full.
>>>>
>>>> On Tue, Apr 12, 2011 at 1:00 PM, Steve Coast
>>>> <steve at asklater.com <mailto:steve at asklater.com>> wrote:
>>>>
>>>> bbox queries using the built in spatial indexing
>>>> presumably? OSM has it's own magical bitmask for that,
>>>> that may also be as fast in mongo, who knows.
>>>>
>>>>
>>>> On 4/11/2011 5:58 PM, Ian Dees wrote:
>>>>> On Mon, Apr 11, 2011 at 6:36 PM, Sergey Galuzo
>>>>> <sergal at microsoft.com <mailto:sergal at microsoft.com>>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am working on evaluation of MongoDB for several
>>>>> storage solutions at hand. Some of them resemble
>>>>> current OSM editing database. I have heard that
>>>>> OSM dev is/was evaluating MongoDB also. I was
>>>>> wondering whether it possible to share the findings?
>>>>>
>>>>>
>>>>> In my experimentation with MongoDB (seen here:
>>>>> https://github.com/iandees/mongosm/) I found it to be
>>>>> very slow. Inserts were speedy, but bounding-box
>>>>> queries took a long time.
>>>>>
>>>>> The most recent dev version of MongoDB includes
>>>>> "multi-location documents" support:
>>>>> http://www.mongodb.org/display/DOCS/Geospatial+Indexing#GeospatialIndexing-MultilocationDocuments
>>>>>
>>>>> This would allow a single way document to be indexed
>>>>> at multiple locations and vastly speed up the map query.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>>>>> http://lists.openstreetmap.org/listinfo/dev
>>>>
>>>> _______________________________________________
>>>> dev mailing list
>>>> dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>>>> http://lists.openstreetmap.org/listinfo/dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> dev mailing list
>>>> dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>>>> http://lists.openstreetmap.org/listinfo/dev
>>>
>>>
>>> _______________________________________________
>>> dev mailing list
>>> dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>>> http://lists.openstreetmap.org/listinfo/dev
>>
>> _______________________________________________
>> dev mailing list
>> dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>> http://lists.openstreetmap.org/listinfo/dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20110412/8990c73f/attachment.html>
More information about the dev
mailing list