[OSM-dev] osm2pgsql for 64-bit IDs

[Jukka Rahkonen]

Igor Brejc kirjoitti:
> On Tue, May 24, 2011 at 11:39 AM, Frederik Ramm <frederik at remote.org>
> wrote:
>
>> Hi,
>>
>>
>> On 05/24/11 11:24, Igor Brejc wrote:
>>
>>> This could be a more serious issue. I guess in the history of GIS there
>>> has never been such a large geo database as OSM is now becoming. Maybe
>>> we (as the OSM community) should take a proactive stance and propose a
>>> new version of shapefile format that could cope with 64 bits. Yes, I
>>> know this is daydreaming, but shapefile format is getting old anyway.
>>> Something using protocol buffers could be a new way to go - easier to
>>> write readers and writers and taking less space, too.
>>>
>>
>> I think in the free/open world, "spatiallite" is trying to be shapefile
>> 2.0,
>>
>
> Yes, but is it really? It's a storage format, you need a 3rd party driver
> to
> read it and it's optimized for querying, not for storing high volume of
> data
> in an efficient manner. And it's a database without a standard schema.
>
> I see spatialite as a good way for thick clients to store geo data without
> the need for an extra DB installation, but not as a good way to exchange
> data files (as opposed to osm.pbf, for example).
>
> Or am I missing something here? I'm interested to know because I plan to
> add
> spatialite support in Maperitive.

The .shp part of a shapefile bunch cannot be bigger than 4 gigabytes and
for a good interoperability it is best to keep it below 2 GB. Therefore
shapefiles cannot keep planet wide datasets. Neither can spatialite. I
have successfully created spatialite database files up 4 or 8 gigabytes
but after that it became very slow to add more data into the database.

There is a little tutorial about OSM data, Spatialite and OpenJUMP at
http://sourceforge.net/apps/mediawiki/jump-pilot/index.php?title=OpenJUMP_with_SpatialLite

Spatialitetools 2.4.0 version
(http://www.gaia-gis.it/spatialite-2.4.0-4/binaries.html) have three ready
made tools for importing OSM-data.
Spatialite_osm_map imports .osm data file into database tables which are
suitable to be rendered with for example standard QGis. 
Spatialite_osm_net creates a routable way network, and spatialite_osm_raw
is creating osmosis style tables.  Tools work fine with reasonable sized
datasets. Spatialite_osm_map with the finland.osm dataset from Geofabrik
is faster than osm2pgsql in slim mode with my laptop. I have not studied
spatialite_osm_map thoroughly and I do not know how well it has solved the
mystery of OSM polygons but that is something that should really be solved
somewhere pretty close to the OSM primary database, if not inside it.

For many use cases it might be nice to have ready made Spatialite country
files available, with tables filled with OSM data, with indexes and with a
nice set of views for simple querying. Instead of downloading raw country
files and spending an hour or more for importing data into PostGIS it
could be just to download and open with your favourite software. But just
like shapefiles of delivering OSM data through Web Feature Service, this
channel would be usable for using OSM data but not for editing and sending
edits back to OSM database. Routability with Spatialite sounds interesting
but I do not know how well it works.


-Jukka Rahkonen-