[Legal-general] Fwd: Proposal For OSMPD Data Repository Folder Structure

Fri Nov 7 18:02:29 GMT 2008

Kari wrote: "I got an idea how to make accessing the
latitute/longitude folders easy, if one doesn't have gps coordinates.
We could have a web page with a map, which is split to 1080 x 720
tiles. The user selects the tile he is interested in. That would open
a web page describing the folder numbers ( W-119-00 / N-60-15 ) and
all the files in that cell. He could then simply download the files
using his web browser.

Instead of 1080x720 tiles, we could have less tiles at the top level
and under that more detailed submaps (a bit like the tiles in OSM
currently...)."

This is a great idea, but what data do we use to make the maps for the
tile index?

Another idea would be to have a web page with a tree index based on
place name. You could start with continent hyperlinks as the braches
of the tree and end with major cites or other small regions like
counties as the leaf hyperlinks. The leaf hyperlinks would link to the
folder as described above. It would look something like this:

North America
  Canada
  Mexico
  United States
     Alabama
     Alaska
     Arizona
     Arkansas
     California
        Amador County
        Calaveras County
        San Joaquin County
           City of Lodi
           City of Manteca
           City of Stockton
           City of Tracy
           West County Rural
           East County Rural
        Stanislaus County

The only advantage of this system is we don't have to draw and tile a
map, which will take some time to set-up.

Kari wrote: "The nice thing about the proposed folder repository is
that it allows people to upload photos, audio files etc. used during
mapping. I don't think it is currently possible to share them in OSM."

I like this too, and I didn't think about audio files.

Kari wrote: "And demand all uploads to be released into public domain. :)"

Absolutely! Everything in the folders should be PD.

Kari wrote: "How do we ensure that two contributors don't modify the
same osm file at the same time? Do we need a file locking mechanism?"

I thought we might have volunteers that would submit data to folders
for each cell. This would avoid the need for a file locking mechanism,
and would help us do a very small amount of quality control. Obviously
this wouldn't be needed as mappers become established, but it wouldn't
hurt to have an experienced contributor making sure that only PD data
gets into the repository.

As I mentioned before, I really think this oversight should be
tailored to the needs of a region. Our needs in the United States
won't be the same as the needs in Egypt. As a general rule, however, I
think we should avoid the use of proprietary (closed) file formats.
Things like SHP, DXF, and PDF would be OK, but I think we should stay
away from stuff like DOC and DWG. (This is only a suggestion, and
could be decided by those working in the cells belonging to a
particular region.)

We could start by splitting up the world amongst our group, and then
will make smaller divisions of the globe as we get more volunteers. I
think this type of voluntary guidance would make OSMPD a great asset
to OSM.

Landon

On Fri, Nov 7, 2008 at 7:34 AM, Kari Pihkala <kari.pihkala at gmail.com> wrote:
> I got an idea how to make accessing the latitute/longitude folders easy, if
> one doesn't have gps coordinates. We could have a web page with a map, which
> is split to 1080 x 720 tiles. The user selects the tile he is interested in.
> That would open a web page describing the folder numbers ( W-119-00 /
> N-60-15 ) and all the files in that cell. He could then simply download the
> files using his web browser.
>
> Instead of 1080x720 tiles, we could have less tiles at the top level and
> under that more detailed submaps (a bit like the tiles in OSM currently...).
>
> Also, when people want to upload something to the server, they could look up
> the folder names using the map.
>
> The nice thing about the proposed folder repository is that it allows people
> to upload photos, audio files etc. used during mapping. I don't think it is
> currently possible to share them in OSM.
>
> And demand all uploads to be released into public domain. :)
>
> - Kari
>
>
> 2008/11/7 Kari Pihkala <kari.pihkala at gmail.com>
>>
>> Sounds good and it is a start. So, we don't need any database to start the
>> PD project. Great!
>>
>> How do we ensure that two contributors don't modify the same osm file at
>> the same time? Do we need a file locking mechanism?
>>
>> I suppose this would be a temporary solution for GPX files and OSM files
>> until we get a real OSM server running?
>>
>> - Kari
>>
>> 2008/11/7 Sunburned Surveyor <sunburned.surveyor at gmail.com>
>>>
>>> Joseph wrote: "Heh - not many people will comment for this reason:
>>> http://www.bikeshed.com/"
>>>
>>> Understood. I was just trying to build some consensus before making
>>> unilateral moves. That got me into trougle with the mailing list idea.
>>> :]
>>>
>>> Joseph wrote: "How many directories does that create in the repository?
>>> 1/4 of a
>>> degree per makes it sound like a lot."
>>>
>>> Since there are 360 degrees of longitude we are talking about (in
>>> theory) <4*360> 1080 longitude folders. Each longitude folder would
>>> have (in theory) <180*4> 720 latitude folders. That would be a total
>>> of 777,600 folders. I agree that is a lot. However, you'd need to
>>> subtract out the folders for the oceans, other large bodies of water
>>> and places like Antartica, where we aren't likely to do any mapping.
>>>
>>> I thought we would only create the directories as they were needed. If
>>> a new PD enthusiast joined the project and wanted to contribute in an
>>> area that wasn't covered by a folder, we'd create it in the repository
>>> so that if fit in the folder structure I defined. This wouldn't be
>>> hard at all.
>>>
>>> Joseph wrote: "Surely more than any human can
>>> manually manage."
>>>
>>> I think a simple software tool could assist with the downloading of
>>> data for a particular area of interest. I'm just looking for a logical
>>> folder structure. There would be some interesting problems, however,
>>> like a road segment that falls within two (2) cells. I'm still
>>> thinking about how to handle that situation. Perhaps we put the road
>>> semment in the cell that contains the majority of the road segment
>>> geometry?
>>>
>>> Joseph wrote: "Another way we could do it is make quad-trees out of
>>> directories. If
>>> you imagine the root directory contains the whole planet, then four
>>> subdirectories each with 1/4 of the world map... and they each contain
>>> 4 directories which quarter the regions again and so on to some
>>> particular detail level."
>>>
>>> That is an interesting concept. The only disadvantabe with this
>>> approach is it couldn't be built incrementally as we expand our
>>> coverage. (At least, I don't think it could.)
>>>
>>> I will inquire about the possibility of hosting a sample OSMPD
>>> repository like the one I described at the OSGeo, and will report back
>>> to the list.
>>>
>>> The Sunburned Surveyor
>>>
>>> On Thu, Nov 6, 2008 at 2:42 PM, Joseph Gentle <josephg at gmail.com> wrote:
>>> > Heh - not many people will comment for this reason:
>>> > http://www.bikeshed.com/
>>> >
>>> > How many directories does that create in the repository? 1/4 of a
>>> > degree per makes it sound like a lot. Surely more than any human can
>>> > manually manage. I guess the trick is that people only need to check
>>> > out a few of the directories dependent on where they live (which is
>>> > kinda nice).
>>> >
>>> > Another way we could do it is make quad-trees out of directories. If
>>> > you imagine the root directory contains the whole planet, then four
>>> > subdirectories each with 1/4 of the world map... and they each contain
>>> > 4 directories which quarter the regions again and so on to some
>>> > particular detail level.
>>> >
>>> > I'm not sure if it buys you much, and again you'll have a lot of
>>> > directories.
>>> >
>>> > -J
>>> >
>>> >
>>> > On Fri, Nov 7, 2008 at 2:33 AM, Sunburned Surveyor
>>> > <sunburned.surveyor at gmail.com> wrote:
>>> >> I've attached a zip file with a proposed OSMPD data repository folder
>>> >> structure. This is only an idea of what we could do. I know there will
>>> >> be lots of other ideas on how we could set something up, or
>>> >> modifications of my proposal. I just want to help move things forward
>>> >> with a productive discusssion. :]
>>> >>
>>> >> Here is a brief explanation of how the folder structure is designed:
>>> >>
>>> >> The top level of the repository is divided into folders that represent
>>> >> areas of the globe that cover 1/4 a degree of longitude. So, for
>>> >> example, the folder W-119-00 contains all of the data between the west
>>> >> longitude of 119-00-00 and 119-15-00. Each of these longitude folders
>>> >> contains subfolders that cover 1/4 a degree of latitude within the
>>> >> strip of longitude covered in the parent folder. In this way the globe
>>> >> is divided into a grid of 1/4 degree "cells".
>>> >>
>>> >> Each cell has the following subfolders:
>>> >>
>>> >> gpx_files
>>> >> history
>>> >> metadata
>>> >> osm_files
>>> >> supplemental_data
>>> >>   raster
>>> >>   vector
>>> >> photos
>>> >>
>>> >> The "gpx_files" folder will contain raw GPS data in GPX format.
>>> >> The "history" folder can contain versioning information and/or
>>> >> information on additions and changes to the data in the cell.
>>> >> The "metadata" folder can contain metadata files for the data in the
>>> >> cell.
>>> >> The "osm_files" folder will contain OSM files produced from the GPX
>>> >> files.
>>> >> The "supplemental_data" folder can contain other PD data that can
>>> >> assist in the mapping efforts. In the United States this would likely
>>> >> include USGS quad maps and TIGER shapefiles, but it might also contain
>>> >> public domain data from other sources. In different places of the
>>> >> world other types of public domain data might be stored here.
>>> >> The "photos" folder would contain digital pictures of the features
>>> >> that had been/were being mapped in the cell.
>>> >>
>>> >> I think only the "Gpx_files" and "osm_files" folders would be required
>>> >> to contain data. The other folders would be for the optional use of
>>> >> those working together on data within a "cell". Perhaps, over time,
>>> >> individual working within the same region could agree on a basic
>>> >> structure for these optional folders and there contents. The content
>>> >> and structure of these cells would vary by region. A naming convention
>>> >> for photos might also be helpful, and I can see having a standard
>>> >> format for a simple text file with photo descriptions.
>>> >>
>>> >> It would be possible to write simple tools to work with this type of
>>> >> folder structure. For example: I could write a plug-in for OpenJUMP
>>> >> that would do the following:
>>> >>
>>> >> - Allow the user to input latitude/longitude coordinates for an area
>>> >> of interest.
>>> >> - Merge the OSM or GPS files in the zones that cover this area into
>>> >> single files for the area of interest.
>>> >> - Load and display these files in the map view on OpenJUMP.
>>> >>
>>> >> I still think it would be a good idea to appoint volunteer "Cell
>>> >> Coordinators" that could over see the contents of each cell folder and
>>> >> help coordinate mapping efforts within the cell. I believe this would
>>> >> help improve the quality and organization of the OSMPD data
>>> >> repository.
>>> >>
>>> >> I am eager to get some type of folder structure determined and put
>>> >> online so I can have a place to put my data.
>>> >>
>>> >> OK. Now you can tear my proposal apart! :]
>>> >>
>>> >>
>>> >> The Sunburned Surveyor
>>> >>
>>> >> P.S. - To view the complete folder structure in the attached zip file
>>> >> you need to click on the W-121-00 longitude folder and then the
>>> >> N-37-00 latitude folder. This folder is the one that covers my home
>>> >> town.
>>> >>
>>> >> _______________________________________________
>>> >> Legal-general mailing list
>>> >> Legal-general at openstreetmap.org
>>> >> http://lists.openstreetmap.org/listinfo/legal-general
>>> >>
>>> >>
>>> >
>>>
>>> _______________________________________________
>>> Legal-general mailing list
>>> Legal-general at openstreetmap.org
>>> http://lists.openstreetmap.org/listinfo/legal-general
>>
>
>
> _______________________________________________
> Legal-general mailing list
> Legal-general at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/legal-general
>
>