[OSM-dev] Is there a way to use simple schema without hstore

[Brett Henderson]

Hi Andreas,

On Sun, Nov 21, 2010 at 12:54 AM, Andreas Kalsch <andreaskalsch at gmx.de>wrote:

>  Hi,
>
> thank you, Brett, for me this is the perfect setup. I hope that others will
> find it useful as well. The names are OK. Two problems:
>
> 1) I read XML or PBF, dump it to CSV and then read the dump, but now my
> feature tables are blank because of these errors:
>
> ERROR:  extra data after last expected column
> CONTEXT:  COPY nodes, line 1: "125799    5    45445    2010-09-30
> 21:23:30+0200    5922698
> 0101000020E610000034034B64D592214088BE164F98894A40"
> ERROR:  extra data after last expected column
> CONTEXT:  COPY ways, line 1: "3999478    17    4473    1970-01-07
> 21:53:09+0100    4957195
> "mapping_status"=>"incomplete","highway"=>"secondar..."
> ERROR:  extra data after last expected column
> CONTEXT:  COPY relations, line 1: "2952    23    29975    1970-01-15
> 10:56:44+0100    5014762    "type"=>"multipolygon""
>
> I use COPY instead of \copy like this:
>
> COPY nodes FROM 'path/to/nodes.txt';
> ...
>
> I don't know why this happens. The number and type of columns of the simple
> schema table and the CSV do fit.
>
> Reading straight into database works (but is much slower).
>

I don't know what the issue could be here.  But there must be a mismatch
between table columns and the dump file.  Perhaps the order is different?


>
> 2) --write-pgsimp-change results in this error log:
>
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Osmosis Version SNAPSHOT-r24310
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Preparing pipeline.
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Launching pipeline execution.
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Pipeline executing, waiting for completion.
> Nov 20, 2010 2:49:23 PM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion
> SEVERE: Thread for task 1-read-replication-interval failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to insert
> action with type=NODE, action=MODIFY and id=996097244.
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ActionDao.addAction(ActionDao.java:80)
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.EntityDao.modifyEntity(EntityDao.java:263)
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.NodeDao.modifyEntity(NodeDao.java:74)
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ChangeWriter.write(ChangeWriter.java:123)
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ActionChangeWriter.process(ActionChangeWriter.java:48)
>     at
> org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.PostgreSqlChangeWriter.process(PostgreSqlChangeWriter.java:71)
>     at
> org.openstreetmap.osmosis.core.sort.v0_6.ChangeSorter.complete(ChangeSorter.java:64)
>     at
> org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloader.processComplete(ReplicationDownloader.java:93)
>     at
> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:284)
>     at
> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:345)
>     at java.lang.Thread.run(Thread.java:619)
> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
> violates unique constraint "pk_actions"
>     at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>     at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>     at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>     at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>     at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
>     at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:321)
>     at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ActionDao.addAction(ActionDao.java:77)
>     ... 11 more
> Nov 20, 2010 2:49:23 PM org.openstreetmap.osmosis.core.Osmosis main
> SEVERE: Execution aborted.
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks
> failed.
>     at
> org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
>     at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
>     at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
>     at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
>     at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
>     at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
>     at org.codehaus.classworlds.Launcher.main(Launcher.java:31)
>
> Is it always guaranteed that a change for datatype/id is unique?
>

If using minutely updates then it is not guaranteed.  But the action table
requires it.  The solution is to pass the change data through the
--simplify-change task before applying it to the database.

Incidentally this is also the reason that I can't support a non-truncating
action table feature without making changes to the actions table definition.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20101124/6af1abee/attachment.html>