[OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)

[Jon Burgess]

On Tue, 2008-06-10 at 08:46 +0100, Tom Hughes wrote:
> In message <1213039275.14294.15.camel at localhost.localdomain>
>         Jon Burgess <jburgess777 at googlemail.com> wrote:
> 
> > The recent Mapnik builds try to mmap() the shapefiles while
> > rendering[1]. Since the largest shapefile is a few hundred MB this could
> > easily cause some issues if the file was mmap'd multiple times in a
> > single process. This might be a particular problem if you use a 32-bit
> > system in the multi-threaded Apache worker mode.
> 
> Um... no. At least not if it does the sensible thing and maps it
> for shared read. Then it will only exist once in memory not matter
> how many times a process maps it, or indeed how many processes map
> it in.

Yes it'll be once in physical memory. My other email in this thread
shows that that each thread maps it to a different virtual address and
32 bit processes can be quite limited in virtual address space.

The attached test program can only map the 313MB shapefile 8 times
before it fails when run on a plain 32 bit machine:

asus:/tmp$ gcc -o mtest mtest.c
asus:/tmp$ ./mtest
File length: 328204288
Mapping: 0 => 0xa46a3000
Mapping: 1 => 0x90da3000
Mapping: 2 => 0x7d4a3000
Mapping: 3 => 0x69ba3000
Mapping: 4 => 0x562a3000
Mapping: 5 => 0x429a3000
Mapping: 6 => 0x2f0a3000
Mapping: 7 => 0x1b7a3000
Mapping: 8 => Mapping failed at 8
asus:/tmp$ uname -a
Linux asus 2.6.23.12-52.fc7 #1 SMP Tue Dec 18 21:18:02 EST 2007 i686
i686 i386 GNU/Linux


> In fact that is probably the ideal way to access a file like that
> as you effectively have one disk backed cache of the file that is
> shared by all users of it.

Yes, I agree with the mmap() approach in principle. It is much faster
than the old one which did thousands of seeks when reading the files.

	Jon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mtest.c
Type: text/x-csrc
Size: 756 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20080610/63cfa88c/attachment.c>