[Imports] TMC LCL - help needed

[Sam Vekemans]

On Fri, Oct 9, 2009 at 4:06 AM, <marcus.wolschon at googlemail.com> wrote:

> On Thu, 8 Oct 2009 10:56:27 -0700, Sam Vekemans
> <acrosscanadatrails at gmail.com> wrote:
> > Hi,
> > in the next week, i'll update the wiki 'importing government (bulk
> > provider) data' and list the 3-step approach.
>
> I'm looking forward to it.
> But please don't limit your self too much to shape-files.
>
> The concept is the same.
1. convert
2. share and
3. import manual (josm) / automatic (bulk_upload) / semi-automatic
(road_matcher/automatch)


>  > If i get through on the conference call today, i'll try to explain it,
> > and maybe someone else who is a better wikipedian can fix it. :-)
>
> Sorry. I'll not be on any of the conference-calls at all.
> (
> http://wiki.openstreetmap.org/wiki/Foundation/Import_Support_Working_Group
> )
>
> > Step 1: Data conversion
> > -using eithor 'shp-to-osm.jar' or 'shp2osm.py' or 'mp2osm' or 'other'
> > we make a conversion script
> > -1 person is in charge of editing this script (as users will submit
> > errors) (just as for other programs)
>
> Yes, my parser for the common interchange-format of LCLs for TMC-services
> exists for a while now. No known errors, well structured and commented
> (code quality is very important to me).
>
>
> > Step 2: data sharing
> > -the person that is running the conversion script would make these
> > .osm files available to the community so people can see it.
>
> Well. Guess what I did quite a few days ago. ;)
>
> >
> > 3 -data import
> > eithor a.1 use josm and copy the features over
>
> That's what we do for everything that cannot be automatically imported.
> I had to add even more warnings as despite quite a lot of fat warnings
> people DID upload some of these reference .osm -files directly.
>
> the nature of the beast :).  ... which is why i want to be making the
rivers (geobaseNHN data) available at the same time, and only for those
areas that have roads already imported.  (so to save people work)


> >          3.2 use bulk_import.pl for direct api load (for large areas,
> > if given permission to access the API at non-peak hours, and on que)
>
> "que"?
>

On the status chart i note the progress, and list what is next to load.
With bulk import, you can set it so that it uploaded to the API only at
non-peak hours, the sys admins, can set it so that changesets would get held
back, to make room for the regular OSM traffic flow.


>
> >          3.3 use a postGIS database and use OpenJUMP Automatch to
> > compare datasets.
>
> Never heard of that but the LCL-map if extremely simplified. I don't think
> the complex matching needed can be done by a generic tool.
>
> It just helps by finding exact matches, and highlights where it couldn't
match right based on a set of rules.
.... thats why i prefer having the .osm files available, and letting local
people copy what they like. (avoiding openJUMP all together)


> > The facts that remain are;
> > 1 -the available data is constantly changing
>
> Not much changes to LCLs expected here. But many other countries with
> lots of TMC-providers may follow. That's why I took so much care my tools
> are well written and easy to understand for a developer.
>
>
In Canada it's speratic, and somethings would be 6 months for a partial
update and 1 year for full update.  and sometimes the source features
tagging gets changed too.

Because it's a federal holding house, they provide data for other people
too, not just OSM :) so they want to keep their dataset uptodate also.


>  > utility tool, and having the 'tag matching list' be maintained by 1
> > person, will make it easier for everyone to work with the data.
>
> What do you mean by 'tag matching list'?
>
>
with shp-to-osm the rules.txt lists the tag matches (source=osm) key/value
comparisions.

from the google docs chart. (or wiki chart if it's not that big)


> > By making it available, we can let the users decide what they want to
> > add, or omit. and put the emphasis on simply making all available data
> > available to use when and if we want.
>
> Sounds less like hints for people doing an import then general advice
> for people offering datasets to import.
>
> lost me there, it's learning for everyone IMO :)

> Marcus
>

re: next message
>
>> The "Merge layers" option in JOSM is also not really usable because it
>> can take really long (several days) for two layers with 100k nodes.

>Well I don't think he was talking about single .osm -files with that
many elements.
. The .osm -files I generate (@topic)never have THAT kind of node-count.

Ya actually it was refering to how lakes can be a higher node count, and the
shp-to-osm script has a hard time with that.  So the solution is to manually
split up the big lake to different polygons.

>Your script is VERY "without fancy logic". I wouldn't use that
at all if there is any already existing data in the area you are doing
an import in. Besides there can be elements with the same id in your case.

 You dont need to use it :)  Its designed to only handle this data.  (and im
not an expert programmer eithor)

... i just ask that they make the .osm files available rather than direct
upload so local area mappers can copy in the data.  .. so where elements are
existing, its up to the local area mapper to decide on what to add, or not
to add, and how or if they want to merge it.  That decision shouldn't be my
own to make remotly.  (not having being physically been in the area).

And a last point, it's fine if the converted .osm files remain on a server
for a year before getting added, as long as people know it's there.  I don't
see a point in rushing, 'just to get it copied in' .

Cheers,
Sam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/imports/attachments/20091009/0384979c/attachment.html>