[OSM-dev] setting up mirror

[Kevin Mcintyre]

Hey Andrew - Thanks for this message.

I'm looking to have a closer to home files that would also be available
publicly.  Specifically I have 1.6T disk on rackspace for hosting OSM files
but the reality is this is not enough to host all the files.  I'll probably
start with latest planet and extracts used by our devs and see where that
gets me.

Thanks all for the replies, am new to OSM...so please forgive ambiguous
questions and my terrible grammar.

Kevin

On Wed, Nov 18, 2015 at 10:44 AM, Andrew Byrd <andrew at fastmail.net> wrote:

> Hello,
>
> I think we should clarify some terms here. The word “mirror” usually
> refers to a secondary copy of some very large or very heavily used
> resource, generally at a different physical location or network link than
> the primary “official” copy. Its purpose is to reduce load on the main
> server and avoid sending huge amounts of data across the world by keeping a
> recent duplicate of the main resource nearer to the final consumers.
>
> When you say “planet osm mirror” people might assume you mean "a copy of
> planet.pbf closer to home”. Creating this kind of mirror is just that
> simple: you’d place a recent copy of planet.pbf on a web or file server of
> some kind, and have people within your organization or geographic area
> fetch planet.pbf from your server instead of the main one (generally to
> reduce traffic crossing out of your local network or traveling long
> distances). Planet.pbf is updated regularly, so you’d probably schedule
> some scripts to keep your mirrored copy in sync by overwriting it once in a
> while or applying OSM diffs.
>
> I am a bit confused by the comments saying there is only one planet
> mirror. There are eight of them listed at
> http://wiki.openstreetmap.org/wiki/Planet.osm and I’m sure there are many
> more undocumented mirrors around the world within organizations that use
> OSM data.
>
> If you only want to work on one geographic region, it’s also possible to
> download a PBF extract of that region and keep it up to date with tools
> like osmupdate or osmosis. This would be much less bandwidth and resource
> intensive than cloning the whole planet.
> http://wiki.openstreetmap.org/wiki/Osmupdate
>
> There is another notion which could also arguably be called “mirroring”
> but is commonly referred to as “replication”: creating a local database
> with a similar schema to the main OSM database and applying diffs to
> maintain a completely up-to-date copy of that main database. This allows
> extensive indexing and thus searching for items within OSM by location, ID,
> tags etc. http://wiki.openstreetmap.org/wiki/Osmosis/Replication
>
> Special-purpose OSM replication databases exist, which you could use
> instead of a general-purpose relational database depending on your use
> case. Overpass is one such project. Instead of using the public overpass
> API server, you can also install Overpass locally to mirror the whole main
> OSM database and keep your mirror in sync:
> http://wiki.openstreetmap.org/wiki/Overpass_API/Installation
>
> I’m also working on a replication system called Vanilla Extract that is
> geared toward performing large, fast geographic extracts. It’s still under
> development but stable enough for us to use it on a daily basis. Our use
> case is routing, so it currently only indexes ways and the nodes referenced
> by them. No free-floating POI nodes (though we’ll eventually add that), and
> no author metadata.
>
> The original version (in C) loads a PBF at a single point in time and does
> no updates.
> https://github.com/conveyal/vanilla-extract
>
> The newer version (in Java using MapDB) is capable of replicating
> planet.osm with a one-line command and performs minutely updates, with the
> option of even pulling the OSM data in over the network without a local
> copy of planet.pbf. However, it’s still a bit slow on large extracts. We’re
> currently migrating it to MapDB2 and expect a performance improvement.
>
> https://github.com/conveyal/osm-lib/blob/master/src/main/java/com/conveyal/osmlib/VanillaExtract.java
>
> Andrew
>
> On 18 Nov 2015, at 18:46, Kevin Mcintyre <kevin at mapjam.com> wrote:
>
> There's some good information here -
> http://download.geofabrik.de/technical.html one companies process at
> least.
>
> Appreciate the response, it's a tricky search because the results are
> generally information on mirrors not how to become a mirror.
>
>
>
> On Wed, Nov 18, 2015 at 9:21 AM, Tom Hughes <tom at compton.nu> wrote:
>
>> On 18/11/15 17:02, Kevin Mcintyre wrote:
>>
>> Hello - I'm seeking information on setting up a planet osm mirror.
>>>
>>
>> I think the reason you're not getting replies is that you're asking for
>> something that doesn't really exist - there isn't any general
>> infrastructure for mirrors in existence.
>>
>> We have one mirror that we redirect most downloads to, and that's it.
>>
>> Tom
>>
>> --
>> Tom Hughes (tom at compton.nu)
>> http://compton.nu/
>>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20151120/aba7364a/attachment.html>