[osmosis-dev] Performance of IndexedObjectStore

[Dennis Frostlander]

Hi,

I am using the IndexedObjectStore for storing and then accessing large
amounts of data - around 300 millions objects.
In the maps I am storing the Long's as the key and simple objects with few
properties as values.
The maps are backed up in the file system by 5 files with size ranging from
3 GB to 11 GB.

When I start accessing the data from the collections, I am experiencing
quite slow performance - just to enumerate all objects in the collection it
takes around 15 hours on the 7200 rpm hard drive, with 10G of memory
available to java vm. The java vm runs in the server mode.

I can see that the machine resources - CPU, hard drives are utilized to a
very small amount, the respective performance counters are close to
minimal.
I have tried to perform multi-threaded reads - in each thread I create
separate indexed store readers. But the result is similar - the benefit is
very small.

Could anyone give me any suggestions how I can improve the data access and
utilize the machine resources more efficiently?

Yours sincerely,
Dennis Frostlander,

P.S. on the related topic, I noticed that when the java process runs in the
debug mode and the debugger is attached (either intellij idea or eclipse),
the read operations are a magnitude slower. Not really sure why though...

Yours sincerely,
Dennis Frostlander,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20111207/f8c518f5/attachment.html>