[OSM-dev] Reliability (was Re: New proposed directory layout for planet.openstreetmap.org)

[Martijn van Exel]

Hi,

On Wed, Sep 5, 2012 at 3:30 PM, Roland Olbricht <roland.olbricht at gmx.de> wrote:
> Dear Martijn,
>
>
>
> There are much more tools around reading OSM files, in particular the XML
> format, than just Osmosis.


Yes, you're right. I mentioned a few, actually, that I use a lot:
osmconvert and osmfilter.
The PBF format is well documented and should be easy enough to
implement. New developers should be able to check
>
>
> And even more important: It is easy to write a piece of software that reads
> XML, and that is _because_ XML is human readable. So you really shy off
> potential developers. It may be 20% or 80% of all potential developers; both
> are numbers to get frightened.

Or it may be 1%. Or 95%. Wild guesses like that don't really make an
argument for me.
>
>
> So I strongly oppose to remove any established format, in particular the OSM
> XML. I'm fine with the proposed directory structure.
>
>
>
> On the other hand, what do you gain by not having XML planet files? 25 GB of
> disk space?
>

Yes, about 25 GB. Every two weeks.

>
>> Agreed, but most of what you would want to do with grep is possible
>
>> with other tools like osmosis, osmconvert and osmfilter that work much
>
>> faster on pbf and o5m files.
>
>
>
> I think you miss the point. The argumentation "Don't continue an established
> toolchain when a fancy new one exists" is exactly what killed the Gnome
> project. Look for Linus Torwalds' reply in
>
> https://plus.google.com/115250422803614415116/posts/hMT5kW8LKJk
>
> The money quote is:
>
>
>
> "One of the core kernel rules has always been that we never ever break any
>
> external interfaces."
>
>
>
> Transferred to our situation this means: we shall carry on the XML format
> forever because there are already a plenty of tools that rely on the XML
> format and they are worth protection, and because this is a clear signal to
> developers that we are a reliable partner.
>

Don't get me wrong, I don't mean to ban XML as an interchange format
for OSM data. I am merely suggesting that the bi-weekly planet dump be
done in PBF only. If people really need XML, they can get a smaller
extent from the API, or from one of various mirrors. If they need the
entire world as XML, they can download the PBF and run the conversion.


> To make it even clearer: being cut off the XML planet means that Overpass
> API will starve for some month until I have implemented the quite complex
> file format PBF, and with it some hundred frustrated users, just to mention
> one tool.
>
Overpass is an amazing resource, but I can't believe it relies on a
XML dump of the database being released every two weeks?  How does
that work?
>
> Do you seriously want to loose a huge part of the OSM community to save 25
> GB of disk space?
>

Maybe we should let the statistics speak. How many times is the XML
planet downloaded as opposed to the PBF? If the numbers for the XML
planet are marginal, then and only then should we consider this. Of
course I don't want to scare away developers, or the community at
large. If this is still a useful resource to many, let it live by all
means. I just want to raise the topic now, because it's a time of
transition anyway, and I feel it could be redundant to the community.
Happy so be proven wrong by the numbers though!

Best,
-- 
martijn van exel
http://oegeo.wordpress.com