[OSM-dev] using Osmium to filter osh files

[Peter Körner]

Hi joost, hi Abhishek,

I've mailed with both of you because you asked me questuons on history 
processing. Abhishek is trying to process a much larger amount of data 
that can't easily be imported into a database, like joost did.

See <https://wiki.openstreetmap.org/wiki/OSM_History_Renderer> for some 
numbers on "large" (Abhishek = Dalek2point3).

Mainland USA did, according to Abhishek, not import successfully on a 
120GB-RAM Amazon instance. At this point, importing into a database does 
not seem suitable.

As Abhishek does not need minutely accuracy, jochens advice given on the 
#osm-dev irc channel may make things a lot easier:

use the osmium_range_from_history-tool or the underlying 
RangeFromHistory-handler to extract all objects valid at a given point 
in time and do your analysis on top of that. then continue to the next 
point in time and read the input file again with a new filter.

Reading the files multiple times to avoid the need for massive amounts 
of memory often works well. This technique is also used in the 
history-splitter for example.

Regards, Peter


Am 24.05.2014 23:45, schrieb joost schouppe:
> Hi Abichek,
>
> I've been using Mazderminds History Splitter and Importer to create a
> Postgres database to do similar analysis. See
> https://github.com/MaZderMind/osm-history-renderer/blob/master/TUTORIAL.md
> and my diary for how far I've gotten:
> http://www.openstreetmap.org/user/joost%20schouppe/diary/21826
> I'm a complete newbee, but from what I hear, it sounds like this would
> give you an easier framework than working directly with Osmium?
>
> I'll send you a personal mail too, because I'm working on a somewhat
> similar project.
>
> Joost
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>