[osmosis-dev] using osmosis to import full history planet

[Martijn van Exel]

Hi Peter and other osmosis users/devs -

We made a few attempts to import a full history planet into a PostGIS 
database and have thus far not been successful. Let me outline what we 
have done (more verbose explanation in the link below):

* Downloaded the latest full history planet from 
http://planet.openstreetmap.org/full-experimental/
* Created an extract for the Amsterdam area first using osmosis and then 
also using Frederik Ramm's history extract perl script[1]. The file 
sizes were not the same (913MB vs 646MB) , but I chalk this up to more 
whitespace trimming by the perl script.
* compiled patched osmosis and history plugin as per instructions on 
[2]. We had to use a db schema from the 0.38 osmosis as the svn version 
shows a schema version of 5, which causes an error (expect version 6).
* executed an osmosis import command on the history extract: 
patched/osmosis --read-xml amsterdam.osm --write-pgsql-history 
user="user" database="database" password="password". The result was an 
osmosis error like this:

SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to 
process COPY file /tmp/copy8882265165846891763rm.
         at 
org.openstreetmap.osmosis.pgsnapshot.common.DatabaseContext.loadCopyFile(DatabaseContext.java:293)
         at 
org.openstreetmap.osmosis.history.v0_6.impl.HistoryCopyFilesetLoader.run(HistoryCopyFilesetLoader.java:91)
         at 
org.openstreetmap.osmosis.history.v0_6.PostgreSqlHistoryCopyWriter.complete(PostgreSqlHistoryCopyWriter.java:130)
         at 
org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:110)
         at java.lang.Thread.run(Thread.java:662)
Caused by: org.postgresql.util.PSQLException: ERROR: null value in 
column "relation_version" violates not-null constraint
  Where: COPY relation_members, line 1: "1613        6292794        W 
          0"

I don't know at this point whether there is something amiss with the DB 
schema or with the planet file. I did see that the relation member that 
causes the error, way 6292794, occurs twice in the planet file, v1 and 
v2, and both versions have 'visible=true'. This can't be right, or can it?

Who can offer any hints as to what to attempt next? Should I be looking 
at the schema or at the full-planet?

Thanks!

Martijn

[1] 
http://trac.openstreetmap.org/browser/applications/utils/osm-extract/history-excerpt.pl
[2] 
http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20100826/31946e8a/attachment-0001.txt
On 3/4/2011 10:18 AM, Brett Henderson wrote:
> Hi Martijn,
>
> The history tasks you're using are part of some experimental work that
> Peter Körner was working on and they're not part of the normal Osmosis
> distribution.
>
> Perhaps Peter can provide some assistance ...
>
> Brett
>
> On Fri, Mar 4, 2011 at 1:09 AM, Martijn van Exel <martijn at geodan.nl
> <mailto:martijn at geodan.nl>> wrote:
>
>     Op 03-03-11 15:07, Martijn van Exel schreef:
>>     Hi all,
>>
>>     We have been trying to import an extract (bbox) of the full
>>     history planet file into a PostGIS database using the osmosis db
>>     schema. Without much success so far. We described the steps we
>>     took here:
>>     https://docs.google.com/document/d/1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o/edit?hl=en
>>
>     ..or the published version of the Google Doc for readability...
>     https://docs.google.com/document/pub?id=1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o
>
>
>
>     --
>     Martijn van Exel
>     Senior Researcher
>     -------------------------------------
>     Geodan S&R
>     President Kennedylaan 1
>     1079 MB Amsterdam (NL)
>     -------------------------------------
>     Tel: +31 (0)20 - 5711 318
>     Fax: +31 (0)20 - 5711 333
>     -------------------------------------
>     E-mail:martijn at geodan.nl  <mailto:martijn at geodan.nl>
>     Website:www.geodan.nl  <http://www.geodan.nl>
>     KvK-nummer: 33 247475
>     Disclaimer:www.geodan.nl/disclaimer  <http://www.geodan.nl/disclaimer>
>     -------------------------------------
>
>
>     _______________________________________________
>     osmosis-dev mailing list
>     osmosis-dev at openstreetmap.org <mailto:osmosis-dev at openstreetmap.org>
>     http://lists.openstreetmap.org/listinfo/osmosis-dev
>
>

-- 
Martijn van Exel
Senior Researcher
-------------------------------------
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-------------------------------------
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-------------------------------------
E-mail: martijn at geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-------------------------------------