[OSM-dev] disk size for planet osm import into PostGIS (on an SSD)?
penorman at mac.com
Thu Jun 27 08:11:10 UTC 2013
> From: Akos Maroy [mailto:akos at maroy.hu]
> Subject: [OSM-dev] disk size for planet osm import into PostGIS (on an
> I'd like to inquire about the estimated disk size needed to import the
planet osm file into PostGIS?
For osm2pgsql about 240GB for the database plus 17GB for flat-nodes is my
recollection for the final size from my last tests.
> What I wanted to do is to import planet osm into a PostGIS that is based
on a 512GB SSD (well, formatted and reported size: 459G, 480720592 bytes). I
did the import using osm2pgsql, using:
> /usr/local/bin/osm2pgsql -d osm_world -s -C 5800 --hstore-all -K -v -G -m
> the end of the import is the following:
> failed: ERROR: could not extend file "base/1602600/4948340.12": No space
left on device
> HINT: Check free disk space.
> Error occurred, cleaning up
There are a couple of ways to reduce the disk space. The first is to use
flat-nodes. This turns what was a 80GB+ nodes table for a full planet into a
17GB flat file.
One other optimization is if you're not planning on doing updates and don't
need the slim tables you can use --drop to get rid of them.
By default osm2pgsql does indexing and clustering of the tables in parallel.
This is fastest, but it results in a big spike of disk usage while
rearranging and indexing as it happens on all the tables at once. I believe
--disable-parallel-indexing will fix this.
I am quite surprised you ran into problems on a 512GB SSD. I've imported
recent planets on smaller volumes. On the other hand, I don't know anyone
who's done a full planet import without --flat-nodes lately and that
probably helps lots.
Two other tips for the next time you try are to use the .osm.pbf file
instead of .osm.bz2, and that there was a new planet file generated about
two hours ago.
Another general tip is that if your planet file is more than a day or so
old, use osmupdate (https://wiki.openstreetmap.org/wiki/Osmupdate) to update
your planet file before importing. It only takes about an hour even if it's
a week old, and that's way faster than importing diffs after.
Because osm2pgsql has *so* many options the help text will now suggest a
reasonable command for importing data, see
More information about the dev