[OSM-dev] release of full-history extracts

[Martijn van Exel]

Peter - so cool!

I did some work independently of this thinking about a good way to
store history in a spatial database.
I'm taking the osmosis 0.6 schema as a point of departure.
What I'm thinking now is to have a table for ways similar to the one
that exists now, but then with version and intermediate_version.
The intermediate versions are the ones created by way nodes getting a
new version without affecting the way version.
I'd say there is no need to store all geometries of intermediary ways,
but only the significant ones - i.e. where the node geometry changes
and thus the way geometry.
There would then be a new table way_gemoetries that contains only the
geometries for significant intermediate versions as well as new way
versions. The foreign key would be (id, version,
intermediate_version).
The original ways table would only require a new intermediate_version column.

Alternatively, there could be a database that does not store way
geometries, keeping size down and import performance up, but would
make mosy history queries much slower.

Is this the same line of thought you are having for the DB schema?

Best
Martijn

On Mon, May 30, 2011 at 3:29 PM, Peter Körner <osm-lists at mazdermind.de> wrote:
> Dear OSM-Develpoers,
>
> New extracts are now online, including the requested Amsterdam extract:
>
> <ftp://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110418/hardcut-bbox-pbf/>
>
> They are in a slightly modified PBF-Format that supports the visible Flag
> [1].
>
> They can be read using this modified version of osmium [2] and its tools
> (ie. osmium_convert [3], which allows you to convert the file back to xml).
>
> If you need help reading those files, don't hesitate to ask me.
>
> It has been created using my history-splitter [4] which is build using this
> very cool library.
>
> To generate those dumps I first used osmium_convert to create a pbf-bersion
> of the full-planet-110418-0000.osm.bz2 (I'll publish this file soon). This
> conversion took 8 hours.
>
> From this pbf file the actual split took just 1 hour (compared to the 21
> hours when I was writing to .osm.bz2 files!!) - what an incredible speedup!
>
> Peter
>
>
>
> [1]
> <https://github.com/MaZderMind/OSM-binary/commit/1ddaca6a03afde8d16e4bf7ab3cd065f09c8d9b1>
> [2] <https://github.com/MaZderMind/osmium>
> [3]
> <https://github.com/MaZderMind/osmium/blob/master/examples/osmium_convert.cpp>
> [4] <https://github.com/MaZderMind/osm-history-splitter>
>
>
> Am 17.05.2011 08:30, schrieb Martijn van Exel:
>>
>> Hi Peter,
>>
>> On Thu, May 12, 2011 at 10:46 AM, Peter Körner<osm-lists at mazdermind.de>
>>  wrote:
>>>
>>> Hi
>>>
>>> I'm very proud to announce the release of the first history extracts.
>>> They
>>> have been created from the latest full-experimental-dump [1] using my
>>> history splitter [2], based on a slightly modified version of Jochen
>>> Topfs
>>> really great osmium framework [3]. They contain multiple versions of an
>>> object. If you just want the map-data as it is currently, use the
>>> Geofabrik-Extracts [4].
>>>
>>> The extracts can be downloaded from gwdg:
>>>
>>>
>>> <ftp://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110418/hardcut-bbox-xml/>
>>>
>>>
>>
>> Thanks for this! I haven't had time to look into them but I am eager
>> to start working with them. These will be helpful for the Essen hack
>> weekend as well!
>>
>> I haven't really looked into it, but would it be straightforward to
>> make an Amsterdam extract using your script? When doing analyses it
>> helps to work with an area you know - at least being able to compare
>> to an area you know.
>>
>> Thanks again
>> Martijn
>>
>
>



-- 
Martijn van Exel
http://about.me/mvexel