[Talk-us] Comparing Tiger 2017 dataset with OSM in a automatedway.

Badita Florin baditaflorin at gmail.com
Thu Oct 26 09:40:01 UTC 2017


I was able to process 10 states until now

[image: Inline image 2]

Did not had time to look at each individual state, but i will share the
insight for Arizona :
Feel free to look at the other states.
And please, if you want me to run this on any public dataset, just tell me,
I will help you create a translation file, and we can compare any SHP with
OSM, and get the results.

Uploaded on the google drive folder Arizona, the result size is 267 MB XML

https://drive.google.com/drive/folders/0B7aOUf0DFRnLU3hUWFNu
S05JS2c?usp=sharing

There seems to be some new neighborhoods that were constructed/ are being
constructed added to this TIGER 2017 release.
[image: Inline image 1]


[image: Inline image 2]
https://www.openstreetmap.org/#map=16/33.03417609298051/-111.46275813092095

[image: Inline image 3]

https://www.openstreetmap.org/#map=16/33.15199379882382/-111.42954284892059

[image: Inline image 4]
https://www.openstreetmap.org/#map=17/33.21657280495004/-111.57601075495845



On Thu, Oct 12, 2017 at 4:52 PM, Ian Dees <ian.dees at gmail.com> wrote:

> Thanks for posting this Badita. When I look through this in JOSM I see a
> very small portion of TIGER roads that are missing from OSM that should
> actually be there. For example, there are some clumps of roads in
> subdivisions or apartment complexes like this:
>
> [image: Inline image 1]
>
> The vast majority of roads seem to be correctly missing from OSM. For
> example, this section of roads that are placed over the top of a baseball
> stadium:
>
> [image: Inline image 2]
>
> This is the same thing I've seen with my TIGER 2017 delta map I made.
> After a decade of hard work, the OSM community has far surpassed the TIGER
> dataset in terms of quality. I would suggest that it is more useful to run
> CYGNUS comparisons with OSM over data directly from local governments
> (there are several examples in https://github.com/osmlab/centerlines)
> rather than the possibly sub-par data from TIGER. The local government data
> will be more likely to have recently-added or recently-removed roads.
>
> -Ian
>
> On Thu, Oct 12, 2017 at 8:19 AM, Badita Florin <baditaflorin at gmail.com>
> wrote:
>
>> I started processing, State by State, the Tiger 2017 dataset, because i
>> get less errors from overpass this way then if i process in bulk of 10
>> states.
>>
>> The first state is Alabama.It took around 5 hours to complete,and it
>> shows over 20.000 possible ways that are not added in OSM.
>> The total length of the ways is 5042 km, or 3133 miles.
>> The xml size is 36Mb, and can be found in this folder.
>> https://drive.google.com/drive/folders/0B7aOUf0DFRnLU3hUWFNu
>> S05JS2c?usp=sharing
>>
>> There are 5 35x35 miles tiles missing from the data.
>>
>> Be mindful that this is the raw output, and even if Tiger 2017 is better
>> then older versions, there are still a lot of bad data in Tiger.
>>
>> I expect more then 50% of the data to not be good for mapping into OSM.
>> Also, you can filter this data-set and keep just the road with the names,
>> that could be more important roads compared to the ones that don`t have a
>> name.
>>
>> @joe, regarding the dataset for the twin cities, if you can
>> create/provide a translation file for the shapefile, i can run it and post
>> the results in the folder.
>>
>> Some examples here
>>
>> https://github.com/ToeBee/ogr2osm-translations
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Oct 11, 2017 at 6:46 PM, Ian Dees <ian.dees at gmail.com> wrote:
>>
>>> Yes, it'd be great to do this. I started a project to track open road
>>> centerlines like this on GitHub here: https://github.com/osmla
>>> b/centerlines
>>>
>>> In theory, we could download that data and then do the same process
>>> using the presumably higher quality road data.
>>>
>>> On Wed, Oct 11, 2017 at 10:42 AM, Joe Sapletal <joe.sapletal at charter.net
>>> > wrote:
>>>
>>>> This is really cool.  Can I suggest for the Twin Cities metro area
>>>> someone doing something similar with the Metro Regional Centerlines
>>>> Collaborative Local Centerlines (MRCC)?  I know that Dakota County hasn’t
>>>> submitted centerlines to Tiger in a couple of years, but will be for the
>>>> next update.  Not sure about the other counties though.  There very well
>>>> may be areas that the MRCC will be a better source than the Tiger data.
>>>>
>>>>
>>>>
>>>> https://gisdata.mn.gov/dataset/us-mn-state-metrogis-trans-mr
>>>> cc-centerlines
>>>>
>>>>
>>>>
>>>> Joe
>>>>
>>>>
>>>>
>>>> *From: *Ian Dees <ian.dees at gmail.com>
>>>> *Sent: *Wednesday, October 11, 2017 10:25 AM
>>>> *To: *Badita Florin <baditaflorin at gmail.com>
>>>> *Cc: *talk-us at openstreetmap.org Openstreetmap
>>>> <talk-us at openstreetmap.org>
>>>> *Subject: *Re: [Talk-us] Comparing Tiger 2017 dataset with OSM in a
>>>> automatedway.
>>>>
>>>>
>>>>
>>>> It would be interesting to see what differences CYGNUS would turn up.
>>>> What would the output of CYGNUS be?
>>>>
>>>>
>>>>
>>>> I put together the TIGER 2017 layer that's in the editors right now.
>>>> I'll work on writing up how I did it later today.
>>>>
>>>>
>>>>
>>>> Basically: I used tiger-tiles (https://github.com/iandees/tiger-tiles)
>>>> to generate a vector tiles database with expanded road names from TIGER
>>>> 2017. Then I downloaded an osm-qa-tiles (https://osmlab.github.io/osm-
>>>> qa-tiles/) file for the United States and ran osmlint's tigerDelta (
>>>> https://github.com/osmlab/osmlint/tree/master/validators/tigerDelta)
>>>> to find the segments that had different geometry. The output was then ran
>>>> through Tippecanoe to generate a vector tileset and posted to Mapbox as the
>>>> low zoom red features.
>>>>
>>>>
>>>>
>>>> On Wed, Oct 11, 2017 at 4:03 AM, Badita Florin <baditaflorin at gmail.com>
>>>> wrote:
>>>>
>>>> Hi, i wanted to ask if there will be interest around comparing TIGER
>>>> 2017 with what we have in OSM, using CYGNUS, in a automated way.
>>>> http://www.openstreetmap.org/user/mvexel/diary/36746
>>>>
>>>>
>>>>
>>>> On top of cygnus, i have developed shgp2cygnus, were you can place any
>>>> shapefile, any size, you provide a translation file, and, in the end, you
>>>> get a list with all the ways that are in the TIGER dataset, but not in OSM.
>>>>
>>>> This would be something useful for the community ?
>>>>
>>>>
>>>>
>>>> I don`t know if somebody is already doing something similar, or what is
>>>> the status ? Were can i read more ?
>>>>
>>>> I knoiw that the TIGER 2017 Overlay in JOSM shows in red the roads that
>>>> are not in OSM, but are in TIger 2017.
>>>>
>>>> But I don`t know were to read more, and if the data is accessible to
>>>> download directly, not just show as a WMS Layer.
>>>>
>>>>
>>>>
>>>> It will take around 7-14 days to process all USA”
>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 1041870 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 957021 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 48356 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 481245 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 2020604 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 641868 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 523075 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us/attachments/20171026/4f98ebdb/attachment-0013.png>


More information about the Talk-us mailing list