[OSM-dev] OSM StreetDensityMap
jochen at remote.org
Tue Mar 6 17:10:27 GMT 2012
On Tue, Mar 06, 2012 at 10:44:45AM -0600, Martijn van Exel wrote:
> On Tue, Mar 6, 2012 at 7:42 AM, Jochen Topf <jochen at remote.org> wrote:
> > On Tue, Mar 06, 2012 at 01:31:31PM +0100, npl wrote:
> > > TagInfo-like systems (aggregating big data and creating statistics)
> > > could definitely be built using hadoop/mapreduce.
> > Or you can do what Taginfo does and just write it cleverly, so it just
> > uses one
> > host for an hour instead of 10 hosts for several hours. :-) I do agree that
> > there are many use-cases for Hadoop & Co. But they also create a lot of
> > overhead...
> Just one host for an hour a day? Wow. That's not very much processing time
> at all for what it provides. Awesome.
I just had a look and currently its at about 2h for the main statistics
generation. So thats gone up from the 1h it used to have. Thats because
people keep asking for more features. :-)
It takes about another hour for crawling the wiki etc.
Most days it takes about another 1.5h for updating the planet files, but on
some days thats considerably slower. I should probably try to figure out why
thats the case. Maybe something else runs in parallel on the machine.
All of that on a 800MHz machine using about 6GB RAM. The OSM processing is
mostly CPU bound so on a modern machine it would be faster. One relatively
easy optimization would be to run the planet update and statistics gathering
in one step. But for now I am lazy and let Osmosis do the planet update
Jochen Topf jochen at remote.org http://www.remote.org/jochen/ +49-721-388298
More information about the dev