[OSM-dev] Some statistics for file-format -development

Marcus Wolschon Marcus at wolschon.biz
Mon Nov 24 09:23:31 GMT 2008


2008/11/21 Marcus Wolschon <marcus.wolschon at googlemail.com>:
> 2008/11/21 Sascha Silbe <sascha-ml-gis-osm-dev at silbe.org>:
>> On Fri, Nov 21, 2008 at 10:59:27AM +0100, Marcus Wolschon wrote:
>>
>>> I ran into an outOfMemory-Exception memory-mapping the file when importing
>>> Baden-Württemberg.
>>
>> [...]
>>>
>>> (I did not think this would become an issue for file-sizes of about 100MB)
>>
>> Did you try it on Windows or a POSIX compatible OS? 32bit or 64bit
>> architecture/OS/userspace?
>> On 64bit Linux, I can easily mmap() even the whole planet database (~50GB
>> total, with largest file being ~30GB) - on 32bit it's less, but still way
>> more than a few hundred MBs.
>
> I will try on a Linux/x86-32 later. ATM I only have an XP on an Eeepc
> at my hands.
> I could raise the limit to a few hundred MB by using an undocumented argument
> to the java-runtime but still , reserving 3GB only allows me to memory-map
> 2 files with a sum of 450MB.

Okay, I "found" it.

I ran my program on an eeepc while programming on the train.
Now I wrote a small test-program that would memory-map a
1 GB-file and write to it. It works fine on a 2GB desktop-PC but
fails to aquire the mapping on the eeepc with 2GB.
Both run Windows XP.

I added a fallback to use normal IO for the index-file
if the memory cannot be mapped.
It is much, much slower (about 1/10 the speed since in the index
I change single integers at random locations in a large file)
but it works.

I'll do the large files on my big Linux-laptop now.

Marcus


More information about the dev mailing list