[Openstreetmap] speed, at last

[Erik Johansson]

On 1/12/06, Lars Aronsson <lars at aronsson.se> wrote:
> Mikel Maron wrote:
> > I've been corresponding with the maintainer of the OnEarth WMS.
> > It may soon be possible for OSM to run its own local cache of
> > Landsat. He has produced a tiling system that only takes up
> > 300Gb of space, which is very affordable. This will speed things
> > up considerably, and is generally a very exciting development
>
> How does it work today?  If we're not running our own local cache
> of the Landsat images, where do they come from?  Are they pulled
> live from some remote server?  What is OnEarth and how are they
> involved?

The terrain images are fetched  from an online WMS that onEarth supplies.

The onEarth ppl are the ones that gives every the nice mosaic of the
world, they created this from  landsat 7 imagery, you can find the
description on: http://onearth.jpl.nasa.gov/


> And what are the sizes today, that are bigger than 300
> GB?  It should be possible to keep sizes down by only storing the
> areas that are used, i.e. worldwide coverage in the uppermost zoom
> levels, and only spotty coverage (Western Europe and little else)
> of the finer details.


I don't know for sure but from the onearth XML description, it says
that all the data takes up 1.66TB of space. And Mikel has stated
before that it takes up 4TB to store it localy. There is a ticket
about it about using quadtree vectors for only storing the things that
matter, http://www.openstreetmap.org/trac/ticket/65 , and that would
help.



Mikel:
----
+ Mapserver requests tiles Landsat from JPL. If some of these could be
cached seperately, that would save a lot of time. We need a lot more
disk space for the cache. To host the entirety of Landsat 7 would
require 4 Tb.

---
http://onearth.jpl.nasa.gov/WMS_GMR.xml:

The WMS Global Mosaic dataset was developed at JPL. In the development
environment, the WMS Global Mosaic is stored as three separate
datasets. One holds the six bands from the visual and near IR at 1
arc-second per pixel. The second dataset contains the panchromatic
band at 0.5 arc-second per pixel. The third dataset contains the two
versions of the thermal band data at 2 arc-second per pixel, for high
and low gain respectively. Each dataset covers the globe from W180 to
E180 and from S85 to N85, having 1,296,000x612,000,
2,592,000x1,224,000 and 648,000x306,000 pixels respectively. The image
data is stored in a tiled, compressed format, with a tile size of
512x512 pixels. Each tile contains information from one band, at 8
bits per pixel.

 The compression is achieved by using zlib http://www.zlib.org, a
freeware lossless compression library. For WMS server use, each
dataset contains reduced resolution versions of the base resolution,
using the same storage format. The required amount of storage for the
complete mosaic, including the lower resolution copies is 1.66 TB. The
panchromatic dataset is 686 GB, the multi-spectral dataset is 963 GB
and the thermal is 58GB.

--
/Erik