[Tile-serving] osm2pgsql - Disk usage

[JunkMail]

I have more information, which may be valuable for anybody else who is 
using osm2pgsql with limited disk space.

I noticed that pgsql was holding onto many deleted, but open, files. 
Each was exactly 1GB. Furthermore, the files were all being used by a 
specific pid. I then ran the query "SELECT pg_terminate_backend(<pid of 
the process>)" in psql. This immediately freed up about 130GB of space, 
and did not affect the import.

I believe that these deleted files were representative of tables which 
were used in the import, but were then dropped. Restarting the 
postgresql service would have also fixed the problem, but I didn't want 
to risk screwing up the import.


On 2019-04-21 8:32 p.m., Stephen D wrote:
> Perfect timing! I was at a GB of space left, which would have lasted 
> me 5 or 6 minutes tops. I deleted the file, but it turns out osm2pgsql 
> keeps it open. I used gdb to close the file, but the kernel didn't 
> free up the space until it dropped to 19MB of free space. I now have 
> 11.5GB free(It only freed up part of the space that the file would 
> have left, it seems), and the import is still going.
>
> I hope the import finishes soon; this is very stressful!
>
> Thank you for your detailed response!
>
>
> Stephen
>
> On 2019-04-21 8:02 p.m., Frederik Ramm wrote:
>> Hi,
>>
>> On 21.04.19 21:15, JunkMail wrote:
>>> I have imported all of the nodes, ways, and routes. Currently, it is
>>> "sorting data and creating indexes". However, I am quickly running out
>>> of disk space. I have already deleted the pbf file(will this cause
>>> issues? I assumed it was no longer needed) and only have 15GB 
>>> remaining.
>>> I worry I will run out of space before it completes.
>> 480 GB will be sufficient for the final database but it could indeed
>> happen that you *temporarily* need a bit more.
>>
>>> At this point of the import, am I safe to delete the flat file? Using
>>> fuser, I found that there are no processes accessing it.
>> It is not needed any more. Though I'm not 100% sure if osm2pgsql will
>> not try to delete the file later and then complain that it's not there
>> any more ;)
>>
>>> I have two other(significantly less urgent) questions. First of all, is
>>> it possible to use nginx instead of apache for serving tiles?
>> Not with mod_tile. You can look into using nginx+mapproxy+mapnik for
>> rendering, or nginx+mapserver+mapcache, but in both cases you'll be far
>> away from what everyone else does, and that tends to make things more
>> difficult.
>>
>>> Second of all, I
>>> plan on moving the server from the SSD to an HDD after the import is
>>> complete. Will I experience disk bottlenecks when serving tiles off the
>>> disk? I will not be updating the database in the future.
>> Yes. Of course, if you can find a way to pre-render all the tiles you
>> need, then that will be a non-issue, and you don't need to move the
>> database at all, you can simply stuff the tiles into a huge mbtiles file
>> and serve from that. *That* would then also get rid of mod_tile and you
>> could use any web server you want. Of course that's only possible if you
>> need the world on z0-12 or z0-13 at most, or maybe a few selected areas
>> down to higher zooms.
>>
>> Bye
>> Frederik
>>
>
> _______________________________________________
> Tile-serving mailing list
> Tile-serving at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/tile-serving