[Imports] Worldwide fuel stations import, 59k objects

[Ilya Zverev]

08.03.2018 10:28, Frederik Ramm пишет:
> Ilya,
> 
> as imports go, yours tend to be the best-planned and I appreciate that
> you do respect our processes even in the face of adversity. I would
> encourage you to also make them the best-documented ;)

Thanks Frederik, I'll try making a good wiki page for these. Though I'm 
inclined to make a single page for every POI import done with OSM 
Conflator, since only tags and dataset providers differ for these, and 
algorithms are mostly the same.

> You say that "we get updates monthly" and later that it is "impossible
> to check each object ... afterwards each month up to eternity". Does
> that mean that you are not planning a one-off import, but a continuous
> data synchronisation process between the NavAds data set and OSM? If
> that is the case, it should be said clearly and mandates a closer look.

Yes, I am receving a json file with ~300k features every month from 
NavAds. This import was made from the March dataset, I also have one 
from February. After a successful first import, I plan to update the 
data with new information. That's why the import sets "ref:navads" tags, 
for easier future conflation.

> I would like to understand more about the different parties in this game
> and their interests. I assume that the fuel station chains pay someone
> (Navads?) to publicize their information, and Navads in turn pays
> someone (you? your employer?) to get this information into OSM. I'm not
> sure, you say "I got my hands on this data set" which sounds a bit like
> you're doing this in a hobby capacity but later you say "we".

Last year, MAPS.ME has signed some kind of a agreement with NavAds and 
some other SEO companies, stating that we will help them with getting 
their data on OpenStreetMap. You have seen the Shell stations import and 
a couple imports by Brandify (Walmarts and Retro Fitness gyms) in US, 
that were the result of this collaboration.

No money switched hands here. We — as in MAPS.ME — are doing this 
because it makes the map that we use richer and more useful to people, 
especially tourists. SEO companies do that because they are paid for it 
by companies.

Finally, you could say I am doing this as a hobby. Because this is not 
directly related to the improvement of the application, nor this in any 
way advertises MAPS.ME. It distracts me from work I need to be doing, 
that have deadlines. I just said that I could help in the past, and I am 
spending my time on this because I myself believe OSM should have the 
most up-to-date information on POIs. Craftmapping helps with that, but 
mappers are not everywhere, and most doesn't bother themselves with 
stopping at every fuel station and noting their opening hours.

> <...> Paying brands will be
> fully covered even in areas with no mapping activity, smaller
> independent stations and non-paying brands will only be covered in areas
> where a mapper happened to add them >
> Now you may reply "but nobody is keeping anyone from adding a small
> independent fuel station" and you are right; but in letting the major
> brands directly manage their representation on OSM you are getting one
> step closer to making OSM unwelcoming for independent contributions
> ("it's no use adding this single shop here, all the retail data in OSM
> is managed by the big brands anyway").

I understand what you are getting at. I am also bothered that not all 
the fuel station owners know about companies like NavAds and use their 
services. That is not big money, so the issue is in visibility. They can 
use something like OnOsm or OSMyBiz for free, remembering to update the 
data. Or they can pay a small fee for companies like Brandify to put 
them on many maps, not limited to OSM. In any case they should be 
interested in that and looking for such opportunities. If they don't, 
they do not consider not being on maps an issue. Which makes the issue 
not theirs, but yours: you allow only 0% or 100% mapping coverage.

> I am also concerned about diversity. In other places we make big words
> about how it is important to attract people from all walks of life, all
> nationalities, all genders to OSM because we believe that this will make
> the map better. But those words ring hollow if at the same time, with an
> import like this, we're essentially replacing the voice of several
> thousand mappers who have last edited a fuel station, by the voice of
> one commercial data provider because we believe that their data is
> somehow better.

Wait, no. We are not replacing anything. The import adds new POI and 
adds or updates a few secondary tags. It does not prevent anybody from 
adding more fuel stations, or changing these already mapped. And there 
is not a single data provider. We in MAPS.ME talked to four, and as a 
member of DWG you know about a dozen more, who want to add businesses on 
our map (which paid for being added to maps), but do not know the way. I 
see that the reaction is mostly "scare them away from OSM" and not "show 
them the correct way to manage the data here", which is what I am trying 
to do.

> What is diversity for, then, when in the end we will
> always assume that commercial data is better?

Better than what? Commercial data is better than no data. Please do keep 
in mind that commercial data = data that is provided by business owners. 
Can you trust them to know where their venues are located and when do 
they open?

> Is diversity just for
> mapping the parts of OSM that are of no interest to business, and for
> everything else we won't hesitate to let our imperfect data made by all
> these imperfect people from all walks of life, be overridden by the
> shiny corporate data set?

Don't you think this is too much? To repeat my answer from the last 
SotM, if you think a hundred thousand points from a commercial provider 
can override and even replace a work by even a single mapper, then OSM 
has failed. To me this is thinking very low of mappers.

A million fuel stations would not override the work of mappers. On the 
contrary, these would validate and improve on that work. Take one worry 
off them, that a in year their mapping would go obsolete and they would 
need to do it again. Unlike highways and buildings, POI change often, 
and seeing that some of my work is looked after would make me glad.

Or is adding tags and looking for data by a commercial entity different 
from the same done by a local mapper? If so, how far it goes — if a 
business owner of 10 shops does it, is it good or bad? 1000 shops? A 
special employee whose job is to keep the map updated? An outside 
company that is provided with a list and knows all the mapping APIs? 
Where do you draw the line?

> Again, not something that would block this particular import, but
> something to keep in mind. This import is one small step on the road
> towards an explicitly non-diverse, corporate-managed OSM and we have to
> be very careful about just how far we're prepared to go down that road
> in return for nice data. Because after the fuel stations there will come
> the restaurant chains, and then the hotel data sets, the retail chains,
> and we might well look back at this import in ten year's time and say:
> This is where the gentrification began in OSM.

Yes, fuel stations are just 1/5 of the full dataset I get from this 
single company. I believe that we can have millions, even tens of 
millions of corporate-managed points of interest if we want it. The 
question is, why do we oppose having the data in batches, in favour of 
mapping every business one by one by ourselves? Does it make our map 
prettier, or more useful, or does adding points from external sources 
diminish work of local mappers in some way?

Ilya