[OSM-dev] disk size for planet osm import into PostGIS (on an SSD)?

Paul Norman penorman at mac.com
Thu Jun 27 08:11:10 UTC 2013


> From: Akos Maroy [mailto:akos at maroy.hu] 
> Subject: [OSM-dev] disk size for planet osm import into PostGIS (on an
SSD)?
> 
> Hi,
> 
> I'd like to inquire about the estimated disk size needed to import the
planet osm file into PostGIS?

For osm2pgsql about 240GB for the database plus 17GB for flat-nodes is my
recollection for the final size from my last tests. 
> 
> What I wanted to do is to import planet osm into a PostGIS that is based
on a 512GB SSD (well, formatted and reported size: 459G, 480720592 bytes). I
did the import using osm2pgsql, using:
> 
> /usr/local/bin/osm2pgsql -d osm_world -s -C 5800 --hstore-all -K -v -G -m
planet-130620.osm.bz2
> 
> the end of the import is the following:
> 
> ...
>  failed: ERROR:  could not extend file "base/1602600/4948340.12": No space
left on device
> HINT:  Check free disk space.
> 
> Error occurred, cleaning up

There are a couple of ways to reduce the disk space. The first is to use
flat-nodes. This turns what was a 80GB+ nodes table for a full planet into a
17GB flat file.

One other optimization is if you're not planning on doing updates and don't
need the slim tables you can use --drop to get rid of them.

By default osm2pgsql does indexing and clustering of the tables in parallel.
This is fastest, but it results in a big spike of disk usage while
rearranging and indexing as it happens on all the tables at once. I believe
--disable-parallel-indexing will fix this.

I am quite surprised you ran into problems on a 512GB SSD. I've imported
recent planets on smaller volumes. On the other hand, I don't know anyone
who's done a full planet import without --flat-nodes lately and that
probably helps lots.

Two other tips for the next time you try are to use the .osm.pbf file
instead of .osm.bz2, and that there was a new planet file generated about
two hours ago.

Another general tip is that if your planet file is more than a day or so
old, use osmupdate (https://wiki.openstreetmap.org/wiki/Osmupdate) to update
your planet file before importing. It only takes about an hour even if it's
a week old, and that's way faster than importing diffs after.

Because osm2pgsql has *so* many options the help text will now suggest a
reasonable command for importing data, see
https://github.com/openstreetmap/osm2pgsql/commit/34a30092d6






More information about the dev mailing list