[OSM-dev] scaling

[Ben Supnik]

Hi Igor,

Igor Brejc wrote:
> I'll play a heretic here, but my feeling is that "openness" in OSM will 
> more and more come under question, and the reason is scaling. Yes, OSM 
> can proclaim the access to its data is open, but in reality only someone 
> (or better some organization/company) with enough HW resources to be 
> able to process planetary OSM data can actually make use of it.

This was my experience, working for a very very small company: while the 
planet file is growing at a (sometimes terrifying) rate, it is still 
very much accessible to an individual...if you have a 32-bit computer 
and a few hundred GB hard drive (and fortunately hard drive space scales 
like crazy) you can go to bat.

I would go further and say that GLOBAL accessibility of OSM is one of 
the aspects of the project that makes it unique among GIS data available 
to small companies, NGOs and individuals.  Here in the US, there are a 
lot of ways to get a small amount of local data (e.g. "my town") from 
the US government, but if you want the whole planet, that's another 
story.  The ability to get "the whole DB period" for OSM is pretty cool.

It seemes to me that issues with accessing the planet as it gets bigger 
are relatively easily solved compared to the DB and net 
infrastructure...that is...

- Mirror the planet download on more servers.
- Provide it in a more compact format as needed.
- Provide a richer set of open source tools to pull it apart.
- Maybe provide more standard tilings/pre-slicings of the planet as part 
of that process.

It seems to me that all of that could be done with infrastructure that 
sits next to (rather than putting load on top of) the core OSM database, 
etc.

When we get a planet file, the first thing we do is chop, slice and dice 
it; no reason that that can't be done by a Linux server with a cron job. 
  (The code we use to cut the planet is not particularly pretty, but it 
gets the job done; it is currently in our public repo and open licensed, 
and if anyone else wants it, I'm happy to help get it into a wider 
context.  But I get the feeling our tool set is "yet another OSM 
slicer." :-)

cheers
Ben

-- 
Scenery Home Page: http://scenery.x-plane.com/
Scenery blog: http://xplanescenery.blogspot.com/
Plugin SDK: http://www.xsquawkbox.net/xpsdk/
X-Plane Wiki: http://wiki.x-plane.com/
Scenery mailing list: x-plane-scenery at yahoogroups.com
Developer mailing list: x-plane-dev at yahoogroups.com