[Talk-ca] call for help, importing roads

jamesmikedupont at googlemail.com jamesmikedupont at googlemail.com
Fri Mar 19 07:03:23 GMT 2010


On Fri, Mar 19, 2010 at 2:46 AM, Adam Dunn <dunnadam at gmail.com> wrote:

> The reason why it is functionally useless at this point is because the
> id/et_id is not being transferred through the process correctly. Perhaps an
> explanation of the process is in order:
> The SQL stage is basically for reprojection and selecting a bounding box.
> If you already had both the government and the OSM roads converted into a
> Shapefile format, and they both had the same projection, and you were
> willing to work on the entire country at the same time, then you wouldn't
> even need SQL.


Here are here:
http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/LL_Roads_OSM.shp

Here are the osm data (from cloudmade)
http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/LL_Roads_OSM.shp

You generally don't want to do the whole country at a time (trust me, it's
> not something you do in OpenJUMP), and you generally need to do some
> reprojection/format conversion, so we need SQL. What gets spit out of the
> SQL step is two files, OSM.shp and GOV.shp, and they each have geometry
> (where the roads are) and id numbers (internal numbers used by each).
> These two files get loaded into OpenJUMP Roadmatcher, where you find the
> matches, find the standalones, then output the result. This result file has
> a list of the id numbers for the roads that are standalone in the government
> database.
> Then (for Canada) geobase2osm.py will use the Roadmatcher result file as a
> mask to see which roads to copy from the government gml file over to an osm
> file. It does this by looking at the standalone id numbers, and then copying
> only the roads in the gml that have those ids.
> I kind of think of it like a transistor. A transistor takes some voltage
> input and connects it to the output, depending on whether the gate voltage
> is on or not. The input/output voltages can be large or small, but the gate
> voltage only needs to be tiny. The gate voltage itself doesn't get passed
> through to the output. Something like this:
>                gate
>                  |
> input -- transistor -- output
>
> Equivalently, geobase2osm.py connects the Government.gml geometry to the
> Standalone.osm geometry, but only if the Standalone Result ID is on. The
> Result ID itself doesn't get passed through, it's only used as a gate mask.
> So we get something like:
>                                      Result ID#
>                                            |
> GovGeometry.gml -- geobase2osm.py -- StandaloneGeometry.osm
>
> Since I haven't yet gotten the ID numbers to pass through, we can't use
> them as a mask for geobase2osm.py, rendering it useless. We can't generate
> osm files that have no duplicates with what's already on OSM.
>

I see, so the ids are the problem. I can assign them new ids. We can rename
the column?


>
> Let's say you want to do things more manually. You could run the match
> process in Roadmatcher, then visually look and see what roads need to be
> copied over, then go over to JOSM and manually pick out those roads. This is
> silly, as you could do this faster by skipping Roadmatcher, and just
> matching by eye within JOSM.
>

I see, well that is what i hoped roadmatcher would help with.

>
> Even if I do get ID numbers to transfer through, I don't know how portable
> geobase2osm.py will be to the Albanian data. What tools have other imports
> used? Maybe there's something else that uses a slightly different method?
>
>
OK, well we need to look into this in detail.
I was working on buffering the roads to remove duplicate.
http://osmopenlayers.blogspot.com/2010/03/comparison-of-maps-using-buffering.html
http://osmopenlayers.blogspot.com/2010/03/maps-of-albania-comparison-of-roads.html

Thanks
mike

> Adam
>
>
> On Thu, Mar 18, 2010 at 1:38 PM, jamesmikedupont at googlemail.com <
> jamesmikedupont at googlemail.com> wrote:
>
>>
>> The roads shp files  are all the same,, that is true.
>>
>>
>> On Thu, Mar 18, 2010 at 9:19 PM, Adam Dunn <dunnadam at gmail.com> wrote:
>>
>>> So far, I've got the LL_Roads2 files imported in PostgreSQL, and I've
>>> written some functions to export them (modified from the NRN process on
>>> wiki), but it's not working properly. Using pgAdmin on Linux, I can see what
>>> data values there are, and their types:
>>> gid serial NOT NULL,
>>> id numeric,
>>> "type" character varying(50),
>>> category smallint,
>>> cat2 smallint,
>>> shape_leng numeric,
>>> "name" character varying(30),
>>> shape_le_1 numeric,
>>> et_id integer,
>>> the_geom geometry,
>>>
>>> When I export this to shp and open in OpenJUMP, I can see that some of
>>> the values aren't getting exported properly. gid doesn't seem to be exported
>>> at all. id, type, shape_leng, shape_le_1, and the_geom all seem to be fine.
>>> Unfortunately, category, cat2, and et_id are all coming up with the value 0,
>>> even though they had other values in the original dataset. Name I can't
>>> test, since it seems to be null in the original data.
>>> Here's the function I'm using:
>>>
>>>
>>> DROP TYPE albania_gov_data;
>>> CREATE TYPE albania_gov_data AS (
>>>   --Included all Albanian keys, since I don't know what they mean/which
>>> are important
>>>     gid integer,
>>>     the_geom geometry,
>>>     id numeric,
>>>     way_type text,
>>>     category smallint,
>>>     cat2 smallint,
>>>     shape_leng numeric,
>>>     way_name text,
>>>     shape_le_1 numeric,
>>>     et_id integer
>>>
>>> );
>>> CREATE OR REPLACE FUNCTION select_albania_gov_roadtile(coordinates text)
>>> RETURNS SETOF albania_gov_data
>>> AS $$
>>>     SELECT gid,ST_Transform(the_geom,4326),id,"type", --might need 4191
>>> rather than 4326
>>>                 category,cat2,shape_leng,"name",shape_le_1,et_id FROM
>>> albania_roadseg
>>>         WHERE
>>>               ST_Intersects(ST_Transform(the_geom,4326),
>>>                               ST_GeomFromEWKT($1))
>>>
>>>
>>> $$
>>> LANGUAGE SQL;
>>>
>>> CREATE TYPE albania_osm_data AS (
>>>     way geometry,
>>>     osm_id integer,
>>>     name text
>>> );
>>>
>>> CREATE OR REPLACE FUNCTION select_albania_osm_roadtile(coordinates text)
>>> RETURNS SETOF albania_osm_data
>>> AS $$
>>>     SELECT ST_Transform(way,4326),osm_id,substr(name,0,80) --might need
>>> 4191 rather than 4326
>>>                 FROM planet_osm_line WHERE
>>>         ST_Intersects(way,ST_GeomFromEWKT($1))
>>>              AND ((highway is not null)
>>>                OR (railway is not null)) --Albania appears to have
>>> railways in gov_data
>>> $$
>>> LANGUAGE SQL;
>>>
>>>
>>> Anybody got any ideas?
>>>
>> But the shapes are working?
>>
>>
>>>
>>> On the data itself:
>>> What is the difference between all the files on the ftp? There is
>>> LL_Roads, LL_Roads2, LL_Roads_OSM, but a quick check in OpenJUMP, and they
>>> all look the same.
>>>
>>
>> no difference
>>
>>
>>> For my test, I exported an area around Gjirokastra [
>>> http://www.openstreetmap.org/index.html?minlat=40&minlon=20&maxlat=40.2&maxlon=20.2&box=yes&layers=B000FTF].
>>> In this area, the id tag was only partially useful: In downtown Gjirokastra
>>> (the densely roaded area that is not currently visible on OSM, but is approx
>>> bounded by [http://www.openstreetmap.org/browse/way/30455602] and [
>>> http://www.openstreetmap.org/browse/way/36777258]) the id tag seemed to
>>> be missing for roads. The id tag was only available in the outer (rural?)
>>> areas. Shape_leng was mostly 0.0, except for about 20 roads at the south end
>>> of Gjirokastra (out of 2479 roads), in which case it matched up with
>>> shape_le_1. This data also includes Footpath as a type, along with the other
>>> types I mentioned in my last email. A listing of all the different
>>> ways/types/categories in this data would be nice to have.
>>>
>>
>> I can tag them afterwards.
>> I have a tag tool as well to do a report.
>> the most important thing is to get rid of the duplicates
>>
>>
>>
>>>
>>> The data is missing a gml file. This is necessary to run the
>>> geobase2osm.py step. You could probably use ogr2ogr to do it, but you'd need
>>> proper id or et_id, whichever would work better as a unique identifier
>>> (probably not id, since it's not available in urban areas!).
>>>
>>
>> hmmm...
>>
>>>
>>> I did get both the OSM data and the (I assume) government data into
>>> openjump roadmatcher, so it would be possible to roadmatch with what I have,
>>> but not very useful.
>>>
>>
>> why not?
>> thank you for your efforts to you irc? come to irc.freenode.net #flossk
>> or add me on skype : h4ck3rm1k3
>>
>> thanks,
>> mike
>>
>>
>>>
>>> Adam
>>>
>>>
>>> On Wed, Mar 17, 2010 at 2:47 PM, Adam Dunn <dunnadam at gmail.com> wrote:
>>> >
>>> > Hi, I was kind of hoping someone more knowledgeable in GIS programming
>>> would've responded first for the following reasons: geobase2osm.py is an
>>> integral part of the whole roadmatching process for Canada, and it's a very
>>> Canadian-specific script (there are chunks of code that deal with naming of
>>> major trunks on a province-by-province basis using if-else programming).
>>> >
>>> > Also, I don't have much experience programming GIS stuff. For example,
>>> the Canadian way of doing this uses EPSG 3348 for projection [1]. I did some
>>> Google searching, and it looks like you would want to use EPSG 4191 for the
>>> Albania area (see [2]), but 2462 might also the one you want to use ([3],
>>> although it looks like you get weird projected bounds with it). I don't know
>>> why this reprojection is really even necessary. When you look at EPSG 3348
>>> (the Canadian one), the projected bounds are really weird there as well, so
>>> maybe it has to be done just to match up with NRN. If the Albania dataset is
>>> already in Lat/Long and 4191 is in Lat/Long and OSM is in Lat/Long, maybe
>>> you don't need to reproject at all? Someone with more GIS knowledge should
>>> know.
>>> >
>>> > You'll also want a script to automatically convert various attributes
>>> in the shp file to tags that are used by OSM. For example, I opened up the
>>> Roads_OSM.shp file in OpenJUMP (just the way it is, no changes or
>>> reprojections or anything), and I see that one road has the following
>>> properties:
>>> > ID: 175879.0
>>> > Type: Well-Kept Gravel Road
>>> > Category: 2
>>> > Cat2: 20
>>> > Shape_Leng: 0.0
>>> > Name:
>>> > Shape_Le_1: 774.851
>>> > ET_ID: 167693
>>> >
>>> > Most of these you probably won't care about, but Type: Well-Kept Gravel
>>> Road should be converted to
>>> > surface=gravel;
>>> > and highway=unclassified or residential or track=* depending on what
>>> the Category: 2 and Cat2: 20 mean.
>>> >
>>> > Another road has Type: Dwelling Area Road, which would probably be
>>> highway=residential;
>>> >
>>> > I've also seen Type: Village Road, Type: Railway, Type: Seasonal Road,
>>> and Type: National Asphalted Road.
>>> >
>>> > You can see where geobase2osm.py makes similar mappings for Canada, by
>>> looking at lines 63 to 77 in geobase2osm.py [4].
>>> >
>>> > None of the roads I sampled had names associated with them (even the
>>> national highway, the name was blank). You'll probably have to match up road
>>> names to the roads using one of the other data sets?
>>> >
>>> > I would like to help out more, but I'm afraid I don't have the
>>> experience. I could lend a hand in modifying the SQL tables and
>>> geobase2osm.py, but I would be of limited help there, and I haven't a clue
>>> how you would get road names imported. You'll want more expertise help in
>>> getting the proper toolchain.
>>> >
>>> > [1] http://spatialreference.org/ref/epsg/3348/
>>> > [2] http://spatialreference.org/ref/epsg/4191/
>>> > [3] http://spatialreference.org/ref/epsg/2462/
>>> > [4]
>>> http://svn.openstreetmap.org/applications/utils/import/geobase2osm/geobase2osm.py
>>> >
>>> > Adam
>>> >
>>> > On Wed, Mar 17, 2010 at 12:31 PM, Sam Vekemans <
>>> acrosscanadatrails at gmail.com> wrote:
>>> >>
>>> >> Hi Adam,
>>> >> James is looking to use RoadMatcher, i know its on the wiki, but that
>>> >> page needs to be fixed.
>>> >> Would you be able to explain it?
>>> >>
>>> >> Personally, i now favour tracing WMS, but for massive areas
>>> >> RoadMatcher is the way to go :-)
>>> >>
>>> >> Also, Robert (Bob) Shand from PEI is currently Itching to also learn
>>> >> how its done.
>>> >>
>>> >> Cheers,
>>> >> Sam
>>> >>
>>> >> ---------- Forwarded message ----------
>>> >> From: Michael Barabanov <michael.barabanov at gmail.com>
>>> >> Date: Wed, 17 Mar 2010 12:13:39 -0700
>>> >> Subject: Re: [Talk-ca] call for help, importing roads
>>> >> To: "jamesmikedupont at googlemail.com" <jamesmikedupont at googlemail.com>
>>> >> Cc: talk-ca at openstreetmap.org
>>> >>
>>> >> Sounds like you just need to convert SHP to OSM. Then you can upload
>>> from
>>> >> JOSM. There's a script here (though I haven't tried it):
>>> >> http://redmine.yellowbkpk.com/projects/list_files/geo
>>> >>
>>> >> On Mon, Mar 15, 2010 at 12:48 PM, jamesmikedupont at googlemail.com <
>>> >> jamesmikedupont at googlemail.com> wrote:
>>> >>
>>> >> > Hi,
>>> >> > I have been talking to Sam V. who mentioned that you guys are
>>> experts in
>>> >> > openjump for road importing.
>>> >> > before I dig too deep into this myself, let me ask if anyone can
>>> help me
>>> >> > with my current problem:
>>> >> >
>>> >> > I have these files from a public domain source, projected into
>>> Lat/Lon
>>> >> >
>>> >> > http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/
>>> >> >
>>> >> > [image: [ ]] LL_Roads_OSM.dbf<
>>> http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/LL_Roads_OSM.dbf
>>> >05-Mar-2010
>>> >> > 07:48 53M  [image: [ ]]LL_Roads_OSM.prj<
>>> http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/LL_Roads_OSM.prj>05-Mar-2010
>>> 07:48 143
>>> >> >   [image: [ ]]LL_Roads_OSM.shp<
>>> http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/LL_Roads_OSM.shp>05-Mar-2010
>>> 07:48
>>> >> > 35M  [image: [ ]]LL_Roads_OSM.shx<
>>> http://xhema.flossk.org:8080/mapdata/03/MapAction/tpginc/LL_Roads_OSM.shx>05-Mar-2010
>>> 07:48
>>> >> > 1.5M
>>> >> > They are from http://tpginc.net/gis/albania/albania.php
>>> >> >
>>> >> > we would like to find the new roads that are not in OSM and import
>>> them.
>>> >> > Can anyone help? can you tell me exactly what software to install, I
>>> am a
>>> >> > bit confused my the many pages and broken links I found for jump.
>>> >> >
>>> >> > thanks,
>>> >> > mike
>>> >> >
>>> >> > _______________________________________________
>>> >> > Talk-ca mailing list
>>> >> > Talk-ca at openstreetmap.org
>>> >> > http://lists.openstreetmap.org/listinfo/talk-ca
>>> >> >
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Twitter: @Acrosscanada
>>> >> Blog:  http://Acrosscanadatrails.blogspot.com
>>> >> Facebook: http://www.facebook.com/sam.vekemans
>>> >> Skype: samvekemans
>>> >> OpenStreetMap IRC: http://irc.openstreetmap.org
>>> >> @Acrosscanadatrails
>>> >
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20100319/c81a93a4/attachment.html>


More information about the Talk-ca mailing list