Brief experiment, 50 silver for participation in a survey versus nothing

[Objectives] To find out how much of an effect on the percentage of players answering a survey a 50 in-game silver incentive can have when given to players agreeing to participate in a Kronos survey.

The secondary objective was to gather data on the latency players from different countries have.

[Methods] 50 players were identified with the first /who query sent to the Kronos I (K1) server on the 8th March 2017 between 12:50 and 13:10 Server Time. They were all sent whispers from my main character, a level 60 undead warlock named "Tekai". The first 25 players in the list were the intervention group and received an invitation to participate in a survey with a 50 silver reward for participation. The next 25 players in the list were the control group and received a standard invitation to participate in a survey. Participant's answers were copy/pasted on this blogpost along with their level and names were removed.

Control group message : "Greetings, I am making a survey to better evaluate how much latency Kronos I players have and I would appreciate your participation. All answers will be kept anonymous."

Intervention group message : "Greetings, I am making a survey to better evaluate how much latency Kronos I players have and I would appreciate your participation. All answers will be kept anonymous and participants will be sent 50 silver by mail."

Follow-up message : "Great! How much latency (ms) do you currently have? Are you in the EU/RU/US/NA/Asia/Africa?"

End message : "Thank you very much for your participation! Have a good day!"

Further details message : "New players often wonder about the latency of the server, this survey would allow them to have an overview. Please send me a in-game mail if you want to look at the results."

I had previously obtained 57% (survey about forum usage) [1] and 39% (survey about world channel) [2] response rates with similar methods (in-game whispers) and expected similar numbers in the control group. This experiment will be underpowered and should be interpretated as such.

[Results] A total of 45 players where whispered, 23 in the intervention group and 22 in the control group. Players in the intervention group had a mean level of 46.38 and players in the control group a mean level of 46.64. Five (5) players were offline when whispered and a total of five (5) players were away from keyboard (AFK) when whispered; I did not keep note of in which group they were. One player in the intervention group probably talked russian "Chto?" and couldn't understand my message.

8 players answered in the intervention (incentive) group (34.8%) and 11 in the control group (no incentive) (50%).

Relative risks can be calculated by blindly using Medcalc [3].  Incentives were associated with a 30% (RR 0.7, 95% CI 0.35 to 1.40) lower chance of getting an answer, although the confidence interval crossed 1, which means there could be no real difference. The absolute difference was -15.2% (34.8%-50%).

In other words sending 100 whispers with an incentive would lead to 35 answers and sending 100 whispers without it would lead to 50 answers.

The reported latencies (ms) were as follows :

• EU : 47ms, 48 ms, 88 ms and 81 ms
• NA : 140-200ms
• US : 120ms
• Oceania : 340 ms
• RU : 129ms and 65ms
• Unspecified location : 20-30ms, 60-65ms, 90-119ms, 420ms, 78ms, 81ms and 46ms
• Latency and location wasn't clearly defined for three (3) other players : "no trouble so far, well maybe sometimes but very punctual moments and didn't last more than 2 or 3 seconds, and it doesn't happen often", "latency is good" and "not sure how low my ms is atm"

I ended up giving out a total of 4 in-game gold (8x50 silver). 16 minutes was the longest amount of time before a player answered my whisper. 30 minutes after the last whisper no further answer was received and I disconnected my character.

[Conclusions] The use of a 50 silver incentive to increase answer rates to a in-game survey on Kronos I was NOT associated with a higher participation. The players NOT offered a reward for participation participated more often (11 answers in the control group vers 8 in the incentive group) but this could be due to chance due to the low sample size.

If three more players had participated in the intervention group, the conclusions could be very different. Perhaps incentives do not work, perhaps it was a too easy to answer topic for incentives to make a difference, perhaps more AFK players were whispered in the intervention group. Or something else.

On a younger server a 50 silver incentives might have a stronger effect as players haven't yet accumulated much gold. The /who queries might not offer truly random players. Players that know me by name might have a higher chance to answer the survey. Players of higher level might be less interested by money (although this didn't appear to have an effect in this experiment).

A handful of players made comments suggesting the amount of money was too low, "lol", "only 50 silver :P". Others seemed enthusiastic "lets do this", "alright", "i am listening" and happy to have participated "=)", "same!", "you too", "Np have a good day".

Acknowledgments

I (Tekai) used to play on the Kronos I server as a level 60 warlock for months in a progression raiding guild, I wrote extensively about it, contributed to bugreports, quality improvements, discussions and have published the (now completed) Kronos WoW Comics (humoristic drawings inspired from the server) on the official Project Kronos Facebook page. I don't have any financial interest in this survey and its results and I am not financed in any way by the Kronos team or anyone else.

This is mostly an exercise of curiosity and for fun (yes my fun is wierd), please don't take it as a scientific study, but go ahead and do refer to it if you find it interesting.

Data :

Intervention group, 23 players whispered :

12:50:05 [W From] <32:_>: EU, 47 ms latency
12:50:16 [W From] <60:_>: 20-30
12:50:24 [W From] <60:_>: Hi sure, im usually playing around 60-65 ping
12:50:29 [W From] <60:_>: only 50 silver :P, go ask, eeeh !@#$ I have addons so i dont see it sec I disable it, i have 48 latency and I am in ___ eu
12:50:42 [W From] <28:_>: Current latency is 119ms thou average is around 90ms
12:51:19 [W From] <51:_>: RU 129
12:56:52 [W From] <60:_>: EU, not sure how low my ms is atm
13:09:35 [W From] <20:_>: oh, i had no trouble so far, well maybe sometimes but very punctual moments and didn't last more than 2 or 3 seconds, and it doesn't happen often

Control group, 22 players whispered

13:00:05 [W From] <60:_>: 420 ms, sometimes i spike to 1337
13:00:10 [W From] <60:_>: 120ms New York, USA
13:00:15 [W From] <60:_>: 65 ms, RU
13:00:36 [W From] <60:_>: 78ms
13:02:48 [W From] <38:_>: ocenia 340ms
13:03:35 [W From] <60:_>: 88MS/EU
13:05:20 [W From] <47:_>: 81
13:05:25 [W From] <9:_>: latency is good
13:05:55 [W From] <55:_>: 81ms right now, i'm in EU
13:07:13 [W From] <4:_>: Just started an hour ago, don't really know if I can do anything, 46ms
13:10:50 [W From] <60:_>: I usually have 140 to 200 MS.. as NA