[OSM-dev] OSM with Hadoop

[Markus Mayr]

A very, very interesting and necessary project! Hadoop seems to be the
mature choice.
I have done some experiments based on this CouchDB scripts:
http://wiki.openstreetmap.org/wiki/OSMCouch
(there exists also an older modified imposm which can export directly to
CouchDB)
While this kind of storage is not made for edits, a student at the
Technical University of Vienna is currently looking into representing
the OSM data model in a better (more flexible) way with CouchDB.


Am 2015-01-09 um 16:27 schrieb Paweł Paprota:
> Hi Stephen,
>
> I read your document - it sounds like an interesting project! Even more
> so that, as you can see, not much has been done in this area so far.
> Pretty much every approach to OSM data storage/handling I've seen is
> Postgres + some variation of a database schema optimized towards a
> specific data usage (like routing, rendering, some specific types of
> read queries).
>
> I'd be very interested to see completely different approaches to this
> problem as the database size grows, it becomes extremely hard to build a
> service which works with all OSM data, meaning a service with global,
> not just local, coverage. Not to mention a service which works with all
> *historic* OSM data - that's basically a suicide mission at this point
> because (1) you probably won't fit all the data in your local dev
> environment and hence (2) if you only have a small part of the data
> locally then you can't easily extrapolate from that to how (or rather -
> IF) your service will work if you build an infrastructure to support
> full dataset.
>
> That is quite a challenge since it really puts heavy brakes on
> innovation in OSM and new services being offered.
>
> Paweł
>
> On Thu, Jan 8, 2015, at 16:33, Pieren wrote:
>> On Fri, Jan 2, 2015 at 7:38 PM, Stephen Knox <stephenknox73 at gmail.com>
>> wrote:
>>
>>> So firstly I am wondering if I am missing any previous posts / research on
>>> the topic?
>> Here are my findings about hadoop quoted in MLs:
>> https://lists.openstreetmap.org/pipermail/dev/2009-August/016554.html
>> https://lists.openstreetmap.org/pipermail/dev/2010-March/018648.html
>> https://lists.openstreetmap.org/pipermail/dev/2010-July/019964.html
>> https://lists.openstreetmap.org/pipermail/osmosis-dev/2010-August/000682.html
>> https://lists.openstreetmap.org/pipermail/dev/2012-March/024468.html
>> https://lists.openstreetmap.org/pipermail/talk/2012-March/062444.html
>> https://lists.openstreetmap.org/pipermail/talk/2012-November/065171.html
>>
>> Pieren
>>
>> _______________________________________________
>> dev mailing list
>> dev at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev