[OSM-dev] OSM StreetDensityMap

[npl]

Am 06.03.2012 14:42, schrieb Jochen Topf:
> On Tue, Mar 06, 2012 at 01:31:31PM +0100, npl wrote:
>> TagInfo-like systems (aggregating big data and creating statistics)
>> could definitely be built using hadoop/mapreduce.
>
> Or you can do what Taginfo does and just write it cleverly, so it just uses one
> host for an hour instead of 10 hosts for several hours. :-) I do agree that
> there are many use-cases for Hadoop&  Co. But they also create a lot of
> overhead...

You're absolutely right - hadoop(java)/pig/... creates a lot of overhead.

In our case we had some join+sort operations which slowed things 
considerably down (these operations don't utilize the cluster well - I 
haven't figured it out yet why this is the case) and the upload to HBase 
was also very slow.


- npl