[Imports] ogr2osm updates

[Paul Norman]

There are no hard limits. ogr2osm will happily load up files so large everything gets paged out into swap. As a practical limit I find I can generate .osm files up to 50% larger (25G) than my physical memory without it slowing down. If you’re using a 32bit python that could be the issue.

 

If you plan to open the created files in JOSM then ogr2osm should never be a limitation. JOSM becomes unbearably slow with a .osm file of a couple gigs and >8GB of RAM assigned to it. I can’t imagine with a 25GB file how you’d be able to get anything done with it.

 

I’d suggest cutting the shapefile with ogr2ogr into reasonable sized chunks. You’ll want a smaller area for testing purposes as you develop your translations anyways

 

From: emux [mailto:devemux86 at gmail.com] 
Sent: Tuesday, February 05, 2013 9:24 AM
To: openstreetmap at googlegroups.com
Cc: imports at openstreetmap.org; Andrew Guertin; penorman at mac.com
Subject: Re: [Imports] ogr2osm updates

 

Hi Paul,

I try ogr2osm with a large shp file of a country detailed road network but python stops the process at 'Parsing data' with 'MemoryError'.
The shp file has size: shp is 180MB and dbf is 1.6GB with over 1.1 million rows.
I use python32 at Win7 64bit.

I know it is quite big. I only managed to produce the osm with the help of java shp-to-osm https://github.com/iandees/shp-to-osm which seems to split the process with max nodes per osm file.

At http://wiki.openstreetmap.org/wiki/Ogr2osm I read 'In March 2010 Ivansanchez was reported to be working on a revamped version of ogr2osm that would be much slower (10x) but would hold all the data in a SQLite database instead of in memory. pnorman's ogr2osm will work on very large files, given enough ram'.
Is there a ogr2osm version which works with large files? And what are actually the limits of your version?

Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/imports/attachments/20130205/26fce821/attachment.html>