[OSM-dev] Some statistics for file-format -development

[Marcus Wolschon]

2008/11/21 Sascha Silbe <sascha-ml-gis-osm-dev at silbe.org>:
> On Fri, Nov 21, 2008 at 10:59:27AM +0100, Marcus Wolschon wrote:
>
>> I ran into an outOfMemory-Exception memory-mapping the file when importing
>> Baden-Württemberg.
>
> [...]
>>
>> (I did not think this would become an issue for file-sizes of about 100MB)
>
> Did you try it on Windows or a POSIX compatible OS? 32bit or 64bit
> architecture/OS/userspace?
> On 64bit Linux, I can easily mmap() even the whole planet database (~50GB
> total, with largest file being ~30GB) - on 32bit it's less, but still way
> more than a few hundred MBs.

I will try on a Linux/x86-32 later. ATM I only have an XP on an Eeepc
at my hands.
I could raise the limit to a few hundred MB by using an undocumented argument
to the java-runtime but still , reserving 3GB only allows me to memory-map
2 files with a sum of 450MB.
I made sure there are no references to the old mapping when growing the
file and mapping again and even called the garbage-collector manually.

The code is pretty small and easy to understand.
http://travelingsales.svn.sourceforge.net/viewvc/travelingsales/libosm/src/org/openstreetmap/osm/data/osmbin/
https://sourceforge.net/svn/?group_id=203597

(look at nodesfile.java and it's super-class FixedRecordFile.java)

I don't understand how this can be yet but I intend to find out.

Marcus