[OSM-dev] History statistics

[Peter Körner]

Hi

no it was a mistake.

Basicly atm there is one huge nested map, that holds all required 
information:

	nodemap[Objekt-ID][Timestamp] = (lat, lon)

lat/lon is atm stored as double. The suggested memory layout is as follows:

1. An Array with a 32bit Pointer for each Node-ID
2. A Series of Timestamp-Lat-Lon Triplets, each stored as Integers, all 
together on one row for all versions, ending with a 0-byte

Memory-Usage would be
8*(max. node-id) for 1.
4*(number of used node-ids) + 12*(number of used node-version) for 2.

Maximum memory usage for a complete planet would be ~40GB.
With a normal Array for 1. the minimum memory usage would be ~10GB.

This could be reduced by using a sparsetable
http://google-sparsehash.googlecode.com/svn/trunk/doc/sparsetable.html

Peter


Am 16.10.2012 10:50, schrieb Ab_fab:
> Thanks for your answer Peter.
> You did not answer to the list, was it intentional ?
>
> I forwarded your reply to the French list thread [1], where the initial
> discussion with Aurelien and others took place, back in july. I totally
> miss the skills required to implement anything ...
>
> Anyway, it could be of interest for some people, in particular for stats
> purpose on the evolution of the database contents (not only the number
> of buildings from cadastre ;-) ).
> French community has also (I think) enough material ressources to setup
> such low activity database, once this RAM issue for the import is solved.
>
> Regards
> Fabien
>
> [1] http://lists.openstreetmap.org/pipermail/talk-fr/2012-July/045337.html
> http://lists.openstreetmap.org/pipermail/talk-fr/2012-October/049849.html
>
> 2012/10/16 Peter Körner <peter at mazdermind.de <mailto:peter at mazdermind.de>>
>
>     Hi
>
>     Well, osm-history-renderer is - after all - a proof of concept. It's
>     impossible atm. to import a full country without having tons of RAM.
>
>     If you are interested in optimizing the memory usage I could provide
>     you with plans about a memory layout frederik ramm and I once talked
>     about; I never came around to implement it, while the necessary
>     interfaces are already in place.
>
>     Regards,
>     Peter
>
>     Am 15.10.2012 15:08, schrieb Ab_fab:
>
>         I think the memory problems occured during the
>         osm-history-importer [1]
>         process
>
>         Some people were successfull in building the database for
>         limited areas,
>         but France is extract is huge in comparison.
>         Is there a way to limit RAM usage for such large files during
>         import ?
>
>         [1] https://github.com/MaZderMind/__osm-history-renderer
>         <https://github.com/MaZderMind/osm-history-renderer>
>
>         2012/7/16 Jochen Topf <jochen at remote.org
>         <mailto:jochen at remote.org> <mailto:jochen at remote.org
>         <mailto:jochen at remote.org>>>
>
>
>              On Mon, Jul 16, 2012 at 01:28:50PM +0200, Aurélien FILEZ wrote:
>               > I'm searching history statistics of the french territory
>              contributions in
>               > order to make some graphics about :
>               > - Number of "named roads"
>               > - Numbers of "house numbers"
>               > - Number of kilometers of different types of roads
>         (highway, primary,
>               > secondary, tertiary, residential).
>               >
>               > I tried OSMIUM with the last avalable france.osh.pbf,
>         but data
>              are too many
>               > data and cause bad allocs errors on a 8 Gb memory server.
>
>              What exactly did you do?
>
>              If you use the RangeFromHistory handler to filter out one
>         specific
>              date in the
>              history and then the Statistics handler, you should be able to
>              generate some
>              statistics, though not exactly the kind of statistics you
>         are asking
>              for. But
>              you can use the Statistics handler as a model on how to
>         write your own.
>
>              You'd have to tun the thing several times, each time
>         filtering out a
>              different
>              date with RangeFromHistory.
>
>              Or you write your own handler that directyl works on
>         history data,
>              but thats
>              a bit more tricky.
>
>              In any case it should not need 8GB for all of this. You
>         might have
>              used the
>              MmapAnon node storage instead of SparseTable handler. This
>         explains the
>              different options:
>         http://wiki.openstreetmap.org/__wiki/Osmium/Storing_Node___Positions
>         <http://wiki.openstreetmap.org/wiki/Osmium/Storing_Node_Positions>
>
>              Jochen
>              --
>              Jochen Topf jochen at remote.org <mailto:jochen at remote.org>
>         <mailto:jochen at remote.org <mailto:jochen at remote.org>>
>         http://www.remote.org/jochen/ +49-721-388298
>         <tel:%2B49-721-388298> <tel:%2B49-721-388298>
>
>
>              _________________________________________________
>              dev mailing list
>         dev at openstreetmap.org <mailto:dev at openstreetmap.org>
>         <mailto:dev at openstreetmap.org <mailto:dev at openstreetmap.org>>
>         http://lists.openstreetmap.__org/listinfo/dev
>         <http://lists.openstreetmap.org/listinfo/dev>
>
>
>
>
>         --
>         ab_fab <http://wiki.openstreetmap.__org/wiki/User:Ab_fab
>         <http://wiki.openstreetmap.org/wiki/User:Ab_fab>>
>
>         "Il n'y a pas de pas perdus"
>
>
>
>
>
> --
> ab_fab <http://wiki.openstreetmap.org/wiki/User:Ab_fab>
> "Il n'y a pas de pas perdus"