[GraphHopper] Ordnance Survey Loader
Peter
graphhopper at gmx.de
Mon Dec 1 15:57:06 UTC 2014
Thanks for the hint!
Is there a magic-free possibility :) ?
Peter
On 26.11.2014 01:01, me wrote:
> By default geotools does not merge properly into an uber-jar as some
> files override others in the meta-inf directory. This is explained
> somewhere on the geotools site and could be relevant to your problem?
> You can use the maven shade plugin to do some automagic renaming to
> get round this. I do it
> here https://github.com/PGWelch/com.opendoorlogistics/blob/master/odl-geotools-fat-jar/pom.xml
> based on one of their examples.
>
> Phil Welch
>
>
> Sent from Samsung Mobile
>
>
>
> -------- Original message --------
> From: Stuart Adam <engaric at hotmail.com>
> Date:
> To: graphhopper at openstreetmap.org
> Subject: Re: [GraphHopper] Ordnance Survey Loader
>
>
> Hi Peter
>
> Yes as I suspected the uberjar causes problems. The
> /META-INF/services/org.opengis.referencing.crs.CRSAuthorityFactory ,
> /META-INF/services/org.opengis.referencing.crs.CRSFactory,
> /META-INF/services/org.opengis.referencing.datum.DatumAuthorityFactory and
> /META-INF/services/org.opengis.referencing.datum.DatumFactory files
> stored are only the versions from gt-referencing not the ones from
> gt-epsg-hsql (or gt-epsg-wkt if you chose that implementation)
>
> Not sure how we merge the content of these service registry files as I
> believe we really want the content from all supplied versions.
>
> Sincerely
> Stuart Adam
>
> ------------------------------------------------------------------------
> Date: Tue, 25 Nov 2014 22:05:07 +0000
> From: engaric at hotmail.com
> To: graphhopper at openstreetmap.org
> Subject: Re: [GraphHopper] Ordnance Survey Loader
>
> Hi Peter
>
> I think I might have an idea what is happening when you use the
> graphhopper.sh script (I have updated to something closer to the
> version in upstream). The geotools libs use a service provider pattern
> whereby properties files in the META-INF/services folder of the
> implementing jar are read for the name of the implementing class. It
> may be that the repacked jar does not have the right layout for that
> to work. I will try to investigate that a bit further.
>
> Sincerely
> Stuart Adam
>
> Sent from my BlackBerry 10 smartphone on the EE network.
> *From: *Peter
> *Sent: *Tuesday, 25 November 2014 13:27
> *To: *GraphHopper Java routing engine
> *Reply To: *GraphHopper Java routing engine
> *Subject: *Re: [GraphHopper] Ordnance Survey Loader
>
>
> Hi Stuart,
>
> with your latest changes, which already include all dependencies as
> you wrote I still get the exception. Using a clean repo and following
> my instructions in the issue.
>
> Do you have an idea how to fix this? I looked into the created jar
> which the graphhopper.sh script is using (tools/target/"gh-with-deps")
> and the content looks fine: with hsqldb etc
>
> Regards,
> Peter
>
> On 24.11.2014 16:04, Stuart Adam wrote:
>
> Hi Guys
>
> I was using the Import class to pre prep the graph. See itngen.sh
> for a command line. The pom in my fork of core should already
> contain the following dependency info which would ensure the
> correct jars are downloaded to your local mvn repo.
> gotools.version is 12.1 and you may need to add the repo
>
> <repository>
> <id>osgeo</id>
> <name>Open Source Geospatial Foundation Repository</name>
> <url>http://download.osgeo.org/webdav/geotools/</url>
> </repository>
>
>
> Dependencies as follows (you might be able to use gt-epsg-wkt in
> place of gt-epsg-hsql and thereby remove the jdbc dependency as
> well, this worked for me before but wasn't working when I was
> readying to commit hence the more heavyweight dependency)
>
> <dependency>
> <groupId>org.geotools</groupId>
> <artifactId>gt-metadata</artifactId>
> <version>${geotools.version}</version>
> </dependency>
> <dependency>
> <groupId>org.geotools</groupId>
> <artifactId>gt-opengis</artifactId>
> <version>${geotools.version}</version>
> </dependency>
> <dependency>
> <groupId>org.geotools</groupId>
> <artifactId>gt-referencing</artifactId>
> <version>${geotools.version}</version>
> </dependency>
> <dependency>
> <groupId>net.java.dev.jsr-275</groupId>
> <artifactId>jsr-275</artifactId>
> <version>1.0-beta-2</version>
> </dependency>
> <dependency>
> <groupId>java3d</groupId>
> <artifactId>vecmath</artifactId>
> <version>1.3.2</version>
> </dependency>
> <dependency>
> <groupId>org.geotools</groupId>
> <artifactId>gt-epsg-hsql</artifactId>
> <version>${geotools.version}</version>
> </dependency>
> <dependency>
> <groupId>org.hsqldb</groupId>
> <artifactId>hsqldb</artifactId>
> <version>2.3.2</version>
> </dependency>
>
> I have also removed the reflective code.
>
> Sincerely
> Stuart Adam
> ------------------------------------------------------------------------
> Date: Mon, 24 Nov 2014 15:49:27 +0100
> From: graphhopper at gmx.de <mailto:graphhopper at gmx.de>
> To: graphhopper at openstreetmap.org
> <mailto:graphhopper at openstreetmap.org>
> Subject: Re: [GraphHopper] Ordnance Survey Loader
>
> Hi Stuart, hi Philip,
>
> thanks for the insights! Also thanks Philip for the suggestion!
>
> > Since once the import is done the graph is the same as one
> imported from osm the query time code is unchanged nd so should
> still be Java 6 compatible.
>
> I see. Then we should probably just make the module separation and
> put all import stuff into another module or tools. E.g. the web
> module is also java 7
> Also I'm not 100% sure if java 7 is that problematic as e.g.
> Android development should be possible with java7 too I think.
>
>
> > As commented by Philip Welch on here the simplest one's to
> provide at runtime is either the hsqldb (remember to add the
> hsql-jdbc driver lib as well)
> > or the wkt based one.
>
> Do you have a pom snippet at hand :) ?
>
>
> > (OSITN = too much block caps for my liking)
>
> me too but no problem - just wanted to mention it
>
>
> > The sample data actually requires a license agreement so people
> should be directed to the website rather than trying to automate
> via wget.
> > If it were not for that stage I would have placed the sample
> data in git repo. I will try and negotiate for permission to
> include it which would simplify that stage.
>
> We can also do this license agreement on the (bash) shell.
>
>
> > At the moment execution time without contraction hierarchies (in
> order to support turn restrictions)
> > is very slow and so we would be interested in what work is going
> on to combine the two and if there is any way we can help
>
> What do you mean with 'very slow'? With turn restrictions it
> should be like 2 times slower than without.
>
> Combining the both is no magic but requires some time to work out
> the bits (although all the necessary steps are already defined ->
> #270). And of course this is important for us too but no starting
> date defined yet.
>
> Regards,
> Peter
>
>
> On 24.11.2014 14:52, Stuart Adam wrote:
>
> HI Peter
>
> The logging is so heavy as we are still learning our way
> around the api. Absolutely this should be reduced as time
> goes on and there is already a number of test cases
> specifically to do with os data. Unfortunately due to a late
> change in coordinate convertor (ITN uses British National
> Grid) some of these are currently broken. The tests are
> almost certainly not as clean as they could be. Only recently
> found the GraphInfo class that might help some with that :-)
>
> I used reflection as that would enable us to more dynamically
> extend the available readers but if it causes problems with
> the iOS port then absolutely happy to change that. iOS and
> android are important for us. The import code at the moment
> takes advantage of String based switches so is Java 7
> specific. We can work on removing that but for now have been
> relying on using desktop class machines to pre import and then
> making the graph folder available to client machines. Since
> once the import is done the graph is the same as one imported
> from osm the query time code is unchanged nd so should still
> be Java 6 compatible.
>
> Yes we will standardise the naming scheme. Relatively new to
> git so was leaving such refactorings for now but was already
> unhappy with my initial naming (OSITN = too much block caps
> for my liking)
>
> The geotools supplies multiple different implementations of
> the CRS lookup. As commented by Philip Welch on here the
> simplest one's to provide at runtime is either the hsqldb
> (remember to add the hsql-jdbc driver lib as well) or the wkt
> based one. Only provide a single implementation at runtime.
> Given the preferences of deployment teams the actual
> implementation used wasn't hardwired in to the build.
>
> The sample data actually requires a license agreement so
> people should be directed to the website rather than trying to
> automate via wget. If it were not for that stage I would have
> placed the sample data in git repo. I will try and negotiate
> for permission to include it which would simplify that stage.
>
> At the moment execution time without contraction hierarchies
> (in order to support turn restrictions) is very slow and so we
> would be interested in what work is going on to combine the
> two and if there is any way we can help.
>
> Sincerely
> Stuart Adam
> ------------------------------------------------------------------------
> Date: Mon, 24 Nov 2014 12:26:31 +0100
> From: graphhopper at gmx.de <mailto:graphhopper at gmx.de>
> To: graphhopper at openstreetmap.org
> <mailto:graphhopper at openstreetmap.org>
> Subject: Re: [GraphHopper] Ordnance Survey Loader
>
> Hi Stuart,
>
> this is nice - thanks!
>
> I took a closer look at it. There seems to be an old
> graphhopper.sh script? Also I had a problem with the import
> when using the sample data you mentioned**
>
> What I did I described in the 'current instructions' of the
> issue #227 and let us move the discussion to it:
> https://github.com/graphhopper/graphhopper/issues/277
>
> Let me know when you think that this is stable enough, then we
> can work together on improving the abstraction and refactoring
> mentioned in the notes, if you like.
>
> Kind Regards,
> Peter
>
> **
> org.opengis.referencing.NoSuchAuthorityCodeException: No code
> "EPSG:27700" from authority "EPSG" found for object of type
> "EngineeringCRS".
> at
> org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.noSuchAuthorityException(CartesianAuthorityFactory.java:136)
> at
> org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.createEngineeringCRS(CartesianAuthorityFactory.java:130)
> at
> org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.createCoordinateReferenceSystem(CartesianAuthorityFactory.java:121)
> at
> org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:801)
> at
> org.geotools.referencing.factory.ThreadedAuthorityFactory.createCoordinateReferenceSystem(ThreadedAuthorityFactory.java:731)
> at
> org.geotools.referencing.DefaultAuthorityFactory.createCoordinateReferenceSystem(DefaultAuthorityFactory.java:179)
> at org.geotools.referencing.CRS.decode(CRS.java:519)
> at org.geotools.referencing.CRS.decode(CRS.java:447)
> at
> uk.co.ordnancesurvey.api.srs.OpenCoordConverter.<clinit>(OpenCoordConverter.java:18)
> at
> com.graphhopper.reader.osgb.OSITNNode.parseCoordinateString(OSITNNode.java:143)
> at
> com.graphhopper.reader.osgb.OSITNNode.parseCoords(OSITNNode.java:131)
> at
> com.graphhopper.reader.osgb.OSITNElement.handleCoordinates(OSITNElement.java:289)
> at
> com.graphhopper.reader.osgb.OSITNElement.readTags(OSITNElement.java:89)
> at
> com.graphhopper.reader.osgb.OSITNNode.create(OSITNNode.java:55)
> at
> com.graphhopper.reader.osgb.OsItnInputFile.getNextXML(OsItnInputFile.java:209)
> at
> com.graphhopper.reader.osgb.OsItnInputFile.getNext(OsItnInputFile.java:174)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcessSingleFile(OsItnReader.java:294)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcessSingleFile(OsItnReader.java:284)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcessDirOrFile(OsItnReader.java:275)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcess(OsItnReader.java:260)
> at
> com.graphhopper.reader.osgb.OsItnReader.readGraph(OsItnReader.java:244)
> at
> com.graphhopper.GraphHopper.importData(GraphHopper.java:634)
> at com.graphhopper.GraphHopper.process(GraphHopper.java:603)
> at
> com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:576)
> at com.graphhopper.tools.Import.main(Import.java:16)
> Exception in thread "main" java.lang.RuntimeException: Problem
> while parsing file
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcess(OsItnReader.java:262)
> at
> com.graphhopper.reader.osgb.OsItnReader.readGraph(OsItnReader.java:244)
> at
> com.graphhopper.GraphHopper.importData(GraphHopper.java:634)
> at com.graphhopper.GraphHopper.process(GraphHopper.java:603)
> at
> com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:576)
> at com.graphhopper.tools.Import.main(Import.java:16)
> Caused by: java.lang.NullPointerException
> at
> org.geotools.referencing.CRS.findMathTransform(CRS.java:1201)
> at
> uk.co.ordnancesurvey.api.srs.OpenCoordConverter.toWGS84(OpenCoordConverter.java:30)
> at
> com.graphhopper.reader.osgb.OSITNNode.parseCoordinateString(OSITNNode.java:143)
> at
> com.graphhopper.reader.osgb.OSITNNode.parseCoords(OSITNNode.java:131)
> at
> com.graphhopper.reader.osgb.OSITNElement.handleCoordinates(OSITNElement.java:289)
> at
> com.graphhopper.reader.osgb.OSITNElement.readTags(OSITNElement.java:89)
> at
> com.graphhopper.reader.osgb.OSITNNode.create(OSITNNode.java:55)
> at
> com.graphhopper.reader.osgb.OsItnInputFile.getNextXML(OsItnInputFile.java:209)
> at
> com.graphhopper.reader.osgb.OsItnInputFile.getNext(OsItnInputFile.java:174)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcessSingleFile(OsItnReader.java:294)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcessSingleFile(OsItnReader.java:284)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcessDirOrFile(OsItnReader.java:275)
> at
> com.graphhopper.reader.osgb.OsItnReader.preProcess(OsItnReader.java:260)
> ... 5 more
>
>
> On 19.11.2014 00:08, Stuart Adam wrote:
>
> Hi All
>
> The dependency on internal code has been removed from the
> OS ITN data loader by using geotools.
> This means you should be able to try it out now. Some of
> the tests currently fail but the basics work. To build
> you will want to skip the acceptancetesting module, since
> that is an internal test which will probably be moved out
> of this repository as it is data dependant. For now skip
> tests whilst I figure out what has broken the unit tests,
> hopefully tomorrow.
>
> To run graphhopper loading itn files supply the following
> configuration options
>
>
> osmreader.osm=<path to data file as normal but this should
> be either a directory or itn xml file rather than osm>
>
> reader.implementation=com.graphhopper.reader.osgb.OsItnReader
>
>
> the reader.implementation setting changes which DataReader
> implementation GraphHopper uses with
> com.graphhopper.reader.osgb.OsItnReader and
> com.graphhopper.reader.OSMReader now implementing an
> interface DataReader. if reader.implementation is not set
> com.graphhopper.reader.OSMReader is used by default.
>
> osmreader.osm setting should probably change to
> datareader.src with this change but for now I have left
> that alone for compatibility reasons.
>
> Once the data is imported it can be used in a standard
> GraphHopper server without having to switch reader
> implementation which means ingested data should work in a
> GraphHopper instance from the main repo without consuming
> the engaric/graphhopper branch though I believe at the
> moment I need to resync with graphhopper/graphhopper repo
> as I suspect the graph version has been changing rapidly
> recently.
>
> Sincerely
> Stuart Adam
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20141201/2347a31d/attachment.html>
More information about the GraphHopper
mailing list