[OSM-dev] Is there a way to use simple schema without hstore
Brett Henderson
brett at bretth.com
Wed Nov 24 08:31:38 GMT 2010
Hi Andreas,
On Sun, Nov 21, 2010 at 12:54 AM, Andreas Kalsch <andreaskalsch at gmx.de>wrote:
> Hi,
>
> thank you, Brett, for me this is the perfect setup. I hope that others will
> find it useful as well. The names are OK. Two problems:
>
> 1) I read XML or PBF, dump it to CSV and then read the dump, but now my
> feature tables are blank because of these errors:
>
> ERROR: extra data after last expected column
> CONTEXT: COPY nodes, line 1: "125799 5 45445 2010-09-30
> 21:23:30+0200 5922698
> 0101000020E610000034034B64D592214088BE164F98894A40"
> ERROR: extra data after last expected column
> CONTEXT: COPY ways, line 1: "3999478 17 4473 1970-01-07
> 21:53:09+0100 4957195
> "mapping_status"=>"incomplete","highway"=>"secondar..."
> ERROR: extra data after last expected column
> CONTEXT: COPY relations, line 1: "2952 23 29975 1970-01-15
> 10:56:44+0100 5014762 "type"=>"multipolygon""
>
> I use COPY instead of \copy like this:
>
> COPY nodes FROM 'path/to/nodes.txt';
> ...
>
> I don't know why this happens. The number and type of columns of the simple
> schema table and the CSV do fit.
>
> Reading straight into database works (but is much slower).
>
I don't know what the issue could be here. But there must be a mismatch
between table columns and the dump file. Perhaps the order is different?
>
> 2) --write-pgsimp-change results in this error log:
>
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Osmosis Version SNAPSHOT-r24310
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Preparing pipeline.
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Launching pipeline execution.
> Nov 20, 2010 2:49:20 PM org.openstreetmap.osmosis.core.Osmosis run
> INFO: Pipeline executing, waiting for completion.
> Nov 20, 2010 2:49:23 PM
> org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager
> waitForCompletion
> SEVERE: Thread for task 1-read-replication-interval failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to insert
> action with type=NODE, action=MODIFY and id=996097244.
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ActionDao.addAction(ActionDao.java:80)
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.EntityDao.modifyEntity(EntityDao.java:263)
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.NodeDao.modifyEntity(NodeDao.java:74)
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ChangeWriter.write(ChangeWriter.java:123)
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ActionChangeWriter.process(ActionChangeWriter.java:48)
> at
> org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58)
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.PostgreSqlChangeWriter.process(PostgreSqlChangeWriter.java:71)
> at
> org.openstreetmap.osmosis.core.sort.v0_6.ChangeSorter.complete(ChangeSorter.java:64)
> at
> org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloader.processComplete(ReplicationDownloader.java:93)
> at
> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:284)
> at
> org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:345)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
> violates unique constraint "pk_actions"
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:321)
> at
> org.openstreetmap.osmosis.pgsimple.v0_6.impl.ActionDao.addAction(ActionDao.java:77)
> ... 11 more
> Nov 20, 2010 2:49:23 PM org.openstreetmap.osmosis.core.Osmosis main
> SEVERE: Execution aborted.
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks
> failed.
> at
> org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
> at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
> at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> at org.codehaus.classworlds.Launcher.main(Launcher.java:31)
>
> Is it always guaranteed that a change for datatype/id is unique?
>
If using minutely updates then it is not guaranteed. But the action table
requires it. The solution is to pass the change data through the
--simplify-change task before applying it to the database.
Incidentally this is also the reason that I can't support a non-truncating
action table feature without making changes to the actions table definition.
Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20101124/6af1abee/attachment.html>
More information about the dev
mailing list