[OSM-dev] Full History Files
brett at bretth.com
Thu Apr 23 12:35:29 BST 2009
Andy Allan wrote:
> On Thu, Apr 23, 2009 at 2:04 AM, Brett Henderson <brett at bretth.com> wrote:
>> * Create a new tool that dumps into a full history osm format. It would
>> differ from existing planet files by having multiple versions of each
>> entity (rather than just the latest) and each would have the visible
>> flag set.
> A full history dump with all the changeset info would be nice, IMHO.
> In the same way that most people eventually "graduate" from using
> planet files to using diffs, it's a straightforward concept to get
> your head around (everything in one file) so probably lowers the
> barrier to entry.
From which they could then graduate to history files ;-)
I'll take a look. It should be fairly straightforward to get to a
single snapshot file in several steps. I'm not sure when I'll get time
to do it though.
1. Combine the daily files I already have into a larger dataset.
2. Re-sort by entity type, then id, then version (uses temporary disk
space to perform persistent low-memory merge sort).
3. Process the large history file into an osm file with history and the
visible flag set as appropriate.
Step 2 is time and disk consuming because a merge sort of large data
sets requires a lot of temporary disk space.
Of course the other option is to modify the existing planet dumper to do
it instead, the downside being that it requires hitting the database
heavily instead of working offline.
Osmosis is unlikely to ever directly support an OSM file with multiple
versions of a single entity. It requires the addition of the visible
flag which makes no sense for most processing. I'd rather keep that
concept out of the osmosis pipeline because that's what the change
streams are supposed to achieve. I'd just write a custom task for step
3 above which receives a standard change stream and writes it in the
special osm history format.
More information about the dev