[OSM-dev] Full History Files

Brett Henderson brett at bretth.com
Thu Apr 23 12:35:29 BST 2009


Andy Allan wrote:
> On Thu, Apr 23, 2009 at 2:04 AM, Brett Henderson <brett at bretth.com> wrote:
>
>   
>> * Create a new tool that dumps into a full history osm format.  It would
>> differ from existing planet files by having multiple versions of each
>> entity (rather than just the latest) and each would have the visible
>> flag set.
>>     
>
> A full history dump with all the changeset info would be nice, IMHO.
> In the same way that most people eventually "graduate" from using
> planet files to using diffs, it's a straightforward concept to get
> your head around (everything in one file) so probably lowers the
> barrier to entry.
>   
 From which they could then graduate to history files ;-)

I'll take a look.  It should be fairly straightforward to get to a 
single snapshot file in several steps.  I'm not sure when I'll get time 
to do it though.
1. Combine the daily files I already have into a larger dataset.
2. Re-sort by entity type, then id, then version (uses temporary disk 
space to perform persistent low-memory merge sort).
3. Process the large history file into an osm file with history and the 
visible flag set as appropriate.

Step 2 is time and disk consuming because a merge sort of large data 
sets requires a lot of temporary disk space.

Of course the other option is to modify the existing planet dumper to do 
it instead, the downside being that it requires hitting the database 
heavily instead of working offline.

Osmosis is unlikely to ever directly support an OSM file with multiple 
versions of a single entity.  It requires the addition of the visible 
flag which makes no sense for most processing.  I'd rather keep that 
concept out of the osmosis pipeline because that's what the change 
streams are supposed to achieve.  I'd just write a custom task for step 
3 above which receives a standard change stream and writes it in the 
special osm history format.

Brett





More information about the dev mailing list