[OSM-dev] OSM StreetDensityMap
npl at gmx.de
Tue Mar 6 22:05:44 GMT 2012
Am 06.03.2012 14:42, schrieb Jochen Topf:
> On Tue, Mar 06, 2012 at 01:31:31PM +0100, npl wrote:
>> TagInfo-like systems (aggregating big data and creating statistics)
>> could definitely be built using hadoop/mapreduce.
> Or you can do what Taginfo does and just write it cleverly, so it just uses one
> host for an hour instead of 10 hosts for several hours. :-) I do agree that
> there are many use-cases for Hadoop& Co. But they also create a lot of
You're absolutely right - hadoop(java)/pig/... creates a lot of overhead.
In our case we had some join+sort operations which slowed things
considerably down (these operations don't utilize the cluster well - I
haven't figured it out yet why this is the case) and the upload to HBase
was also very slow.
More information about the dev