[Talk-ca] Dealing with huge features

Frank Steggink steggink at steggink.org
Tue Nov 10 03:37:49 GMT 2009


Hi,

While testing out the Python version of canvec-to-osm, I came across a 
couple of huge features. Especially wooded areas have the tendency to 
grow large.

The second file I looked at (NTS tile 021L03), was already 3.1 MB large, 
despite that I set maxnodes to 2000 in shp-to-osm. It was mostly 
occupied by a giant multipolygon, which contains 321 members and more 
than 30k nodes. When I opened this file in JOSM, it was really 
struggling with it. Uploading this would be a real nightmare.

Since the feature occupied less than half of the NTS tile, there would 
even be room for several such features. In this scenario it is easy to 
imagine that the nodes limit for getting data from the OSM server is 
exceeded. I don't think this is a desirable situation, but I don't know 
a clear solution how to deal with this.

Although splitting up the features is not a good idea, it would at least 
provide a means to upload the data in smaller chunks, and be able to 
retrieve a part of the data, provided that the tile doesn't exceed the 
server limit. Hopefully JOSM would also be more performant.

For those curious, I have uploaded this OSM file here: [1]. It is part 
of this area: [2].
Anyways, check the file out for yourself, and please share any ideas how 
we should deal with a situation like this.

In the meantime I noticed that NTS tile 021L10 contains even a 4.1 MB 
large file. If there are roughly 10k nodes per MB, this would mean that 
all multipolygon members would contain at least 40k nodes...

Regarding the Python script: the first version is nearly complete. I 
want to make a couple of small changes to it, and also check if 
everything looks OK in JOSM, and perhaps generate a couple of tiles from 
it (locally). Once that is done, I'll make it available to whoever is 
interested. It takes roughly 35 mins to convert all features (except 
highways and hydro) in NTS tile 021L. Executing shp-to-osm costs most of 
the time. The script also downloads any missing Canvec SHP files, but 
they were already downloaded.

Cheers,

Frank

[1] 
http://www.steggink.org/osm/Canvec_test/021l03_VE_1240009_2_Wooded_area10.osm.zip
[2] 
http://www.openstreetmap.org/?minlon=-71.5&minlat=46&maxlon=-71&maxlat=46.25&box=yes





More information about the Talk-ca mailing list