[OSM-talk] Issues with OSM import to postgis

Wed Jun 15 11:11:16 BST 2011

Hello

The latest planet file which we have downloaded is OK and we have 
started with the import
Since osm2pgsql does not work for us since there is no sufficient memory 
we have tried with osmosis
Now the reading part is OK since we have 4GB of ram and we also have an 
additional partition over 400GB which is used for osmosis to put the 
temporary file, but we get the following error

F:\osmosis-latest\osmosis-0.39\bin>osmosis --read-xml 
file="D:\planet-latest.osm" --wp user="postgres" database="gisw" 
host="10.1.1.63"
Jun 15, 2011 9:33:26 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.39
Jun 15, 2011 9:33:26 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jun 15, 2011 9:33:26 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Jun 15, 2011 9:33:26 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Jun 15, 2011 12:02:16 PM 
org.springframework.beans.factory.xml.XmlBeanDefinitionReader 
loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource 
[org/springframework/jdbc/support/sql-error-codes.xml]
Jun 15, 2011 12:02:16 PM 
org.springframework.jdbc.support.SQLErrorCodesFactory <init>
INFO: SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, 
MySQL, Oracle, PostgreSQL, Sybase]
Jun 15, 2011 12:02:20 PM 
org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager 
waitForCompletion
SEVERE: Thread for task 1-read-xml failed
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad 
SQL grammar [SELECT version FROM schema_info]; nested exception is 
org.postgresql.util.PSQLExcepti
on: ERROR: relation "schema_info" does not exist
   Position: 21
         at 
org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:233)
         at 
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
         at 
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:406)
         at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:455)
         at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:463)
         at 
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:471)
         at 
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:476)
         at 
org.springframework.jdbc.core.JdbcTemplate.queryForInt(JdbcTemplate.java:485)
         at 
org.springframework.jdbc.core.simple.SimpleJdbcTemplate.queryForInt(SimpleJdbcTemplate.java:113)
         at 
org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateDBVersion(SchemaVersionValidator.java:64)
         at 
org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateVersion(SchemaVersionValidator.java:47)
         at 
org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetLoader.run(CopyFilesetLoader.java:78)
         at 
org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlCopyWriter.complete(PostgreSqlCopyWriter.java:108)
         at 
org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:110)
         at java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException: ERROR: relation 
"schema_info" does not exist
   Position: 21
         at 
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
         at 
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
         at 
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
         at 
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
         at 
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
         at 
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:252)
         at 
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
         at 
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
         at 
org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:440)
         at 
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:395)
         ... 12 more
Jun 15, 2011 12:02:20 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more 
tasks failed.
         at 
org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
         at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
         at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
         at java.lang.reflect.Method.invoke(Unknown Source)
         at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
         at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
         at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
         at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
         at org.codehaus.classworlds.Launcher.main(Launcher.java:31)

F:\osmosis-latest\osmosis-0.39\bin>

The database is postGIS.
Are we missing something ?

On 6/14/2011 12:55 AM, Jon Burgess wrote:
> On Mon, 2011-06-13 at 12:04 +0200, Zolt Egete wrote:
>> As far as other map files are concerned I have downloaded a few ones
>> but this is the only one which I could unpack (have used pbzunzip2,
>> bzip2, bunzip2) but all the time I have got corrupt archive messages.
>> Also the MD5 sum of the downloaded file where not consistent with the
>> md5 hash sum from the servers (I do not yet know the reason why)
>>
> I think this is where your problem is.
>
> With the recurring checksum issues you are seeing I would suspect that
> something in your system is causing random data to be corruption. This
> could be one of a thousand different things: a faulty RAM module, hard
> drive cable, bad PSU, an over clocked or over heating CPU, bad PCI card
> etc. Or it could be a software issue, such as a bad driver in the kernel
> or X11. It can be really hard to identify the real cause but you could
> try running memtest86+ or swapping around pieces of hardware.
>
> I do not believe the files on the server are at fault. If they were then
> I would expect to see many more complaints.
>
> Until you can resolve the underlying data corruption issue then I don't
> believe it is worth you trying different combinations of options with
> osm2pgsql.
>
>     Jon
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20110615/540b704b/attachment-0001.html>