[OSM-dev] Tiled osm2pgsql import?

[Kai Krueger]

Nick Whitelegg-2 wrote:
> 
> Related to the ongoing discussion on talk, and given the large memory
> requirements of osm2pgsql imports, has anyone published a "tiled" import
> script, where the import area is broken into say degree tiles, and each
> tile imported as a separate osm2pgsql job? If not, I'm inclined to do one
> myself as I'm now having difficulties even importing UK rights of way into
> a postgis database (slim mode will fix it for now, but maybe not for ever)
> 
Isn't that exactly what slim mode is therefore? To handle make sure it works
with limited memory? Theoretically, you should be able to import the whole
planet on e.g. a 512Mb Ram machine in slim mode. It would just take a very
long time! (Unless you have a super fast disk). So it should fix the memory
problem "for ever".

Slim mode does cost some speed and so it is slower to import than in
non-slim mode, particularly once you no longer have enough ram to cache all
the nodes, but I am not sure a tiled import would necessarily help with
speed.

However, there are other reasons why being able to import multiple extracts
is a good idea, e.g. because you want to render across the boarders of two
extracts and moving to the next larger extract containing both regions is
very expensive resource wise. So a tiles import would allow you to more
carefully select the regions you care about. 


Nick Whitelegg-2 wrote:
> 
> One issue which would need to be addressed (very easily I suspect) would
> be to tweak the osm2pgsql code to flag import of duplicate osm_id ways as
> merely a warning (and continue to refuse to add the same way twice, but
> carry on processing the data), rather than abort as an error.
> 
I don't know how easy it would be, but osm2pgsql kind of already has that
mode. With diff imports, it can deal with importing the same diff multiple
times and thus handle adding nodes, ways and relations that already exist in
the database. So you could possibly treat everything as diffs and it might
work.

However, for diff processing you are back to needing slim mode, and diff
imports are way slower than initial imports even in slim mode, as they have
to do extra checks to deal with duplicates.

Kai

--
View this message in context: http://gis.638310.n2.nabble.com/Tiled-osm2pgsql-import-tp6874334p6874421.html
Sent from the Developer Discussion mailing list archive at Nabble.com.