[OSM-talk] Report on the OSMF 2021 Survey after One Week

[Christoph Hormann]

On Sunday 24 January 2021, Mateusz Konieczny via talk wrote:
>
> We do not know how strong bias, even type of it and we can only guess
> how sample of survey-takers is biased compared to general community
> (and there is tradictional problem of people trying to manipulate
> survey by taking it multiple times, strategic refusal to answer some
> questions about demographics or just lying).

Not only that, as explained on my diary there is also the problem that 
different respondents will have different understandings of the 
questions asked and will therefore essentially answer different 
questions - due to in particular the different language versions not 
being identical, due to different cultures of expressing your opinion 
and due to ambiguities and subtext in the questions.

Past OSMF boards when doing surveys upfront acknowledged that in terms 
of the responses these were nothing more than anecdotal sampling of or 
selective brainstorming with parts of the community.  It is not a good 
sign that the current board now indicates upfront that they plan to 
draw quantitative conclusions (like XX percent of the OSM community 
agree/disagree with the decisions of the board from the past year with 
a YY percent confidence possibly?)

If you'd correlate the answers to the questions with the demographics 
data you could very well end up analyzing general cultural 
particularities of the respondents instead of differences in views on 
the subject matter.  Like when people from country X responding 
slightly more positively to question Y that could either mean they have 
a substantially more positive opinion on it or that their cultural 
habits mandate a more positive baseline in formulating an opinion to 
others.  Or that their reading due to translation differences and 
different culture specific connotations being connected to the phrasing 
is substantially different.

As understandable as it is that the board would like to have such 
information simply neglecting the factors that make determining such 
things quantitatively with a survey impossible is not advisable.  A 
solid scientific scepticism beyond the mere mathematics of the numbers 
is important here.

I look forward to studying the results of the survey if they get 
published in a comprehensive and not just a subjectively selective 
form - in particular the free form answers have been insightful in 
previous surveys and certainly could be in this one as well.  But 
personally i much more value actually talking and listening to people 
(or reading what they write in bidirectional communication) to learn 
about their thoughts on things and the reasons for those thoughts in a 
way that a survey will never give you access to.

-- 
Christoph Hormann
https://www.imagico.de/