This change was done because I hypothesized that the order could affect my results and reversing it in the replication would end up balancing the first experiment. For example, there could be more players willing to answer in zones beginning with different alphabetical letters. Should I try this experiment again, I would instead encourage whispering players in a different manner, like this : player 2 on the list, player 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24. And then giving out the second intervention to players 1, 3, 5, etc.
[Results] A total of 44 players were whispered, 24 in the intervention group and 20 in the control group. Players in the intervention group had a mean level of 44.4 and players in the control group a mean level of 44. A total of five (5) players were offline when whispered (1 in the intervention group and 4 in the control group) and a total of four (4) players were away from keyboard (AFK) when whispered (2 in the intervention group and 2 in the control group).
One (1) player in the control group didn't talk english and couldn't participate. Two (2) players, one in each group said I had already whispered them and decline to participate twice. One (1) player in the incentive group didn't want to get the reward twice when I whispered him on his other character, and one (1) player in the intervention group said money wasn't needed "I'm happy to help, No money neccessary, :)".
9 players answered in the intervention group (37.5%) (the player answering twice wasn't counted) and 8 in the control group (33.3%) (the player that didn't speak english wasn't taken into account, neither was the one answering twice).
Relative risks can be calculated by blindly (I know, I know...) using Medcalc [3]. Incentives were associated with a 17% (RR 1.17, 95% CI 0.52 to 2.63) higher chance of getting an answer, although the confidence interval crossed 1, which means there could be no real difference. The absolute difference was +4.2% (37.5%-33.3%).
When results from the previous experiment were combined with those new ones* (this means I assume they are similar, which could be wrong), the incentives were associated with a 7% (RR 0.93, 95% CI 0.54 to 1.63) lower chance to get an answer, but again, the results could be due to chance. The absolute difference was -4% (41%-45%).
The reported latencies (ms) were as follows :
- EU : 99ms, 53ms, 45ms, 167ms, 47ms, 73ms, 50-100ms, 15-30ms
- US : 170ms, 166ms, 100-175ms,
- RU : 50-60ms
- Unspecified location : 18ms, 195ms, 16-31ms
- One (1) player couldn't see his latency due to addons "I cant see latency, sorry" and one (1) gave a narrative answer "hi, today I have experienced some heavy lag spikes, but since tha made the network changes, latency have been good overall" and players added details "its usually good, idk what the average latency is or whatever", "there is always the weird server lag especially if you encounter ppl from NA as European".
Perhaps of interest is that no single player asked me if and where the results could be seen or what I wanted to do with them. The "further details" message was never sent to anyone.
Some players seemed particularly happy to participate and friendly "I'm happy to help", "You are doing gods work :)", "no problem thanks you too", "Greetings sir, ofc i can help out with that". I did not get negative comments about the amount being too low.
[Harms] I had to apologize twice for sending the same whispers to characters from the same players and none complained of this being a problem "hi i already said around 60 on another char, no need to send again :p, no problemo", "I have already answered on another character and answering twice would provide lackluster results, right?, No worries, You are doing gods work :)".
[Conclusions] The overall results suggest that a 50 silver incentive is associated with a lower chance of getting an answer when players are whispered to participate in an online survey. To get a definitive answer I would need a much larger sample, especially considering how the effect, if any, appears to be small.
In the very best possible scenario, with the maximum relative increase of 63% (RR 1.63), and a baseline answer rate in control groups of 34% (33.3% to 34.8% in my experiments), 50 silver incentives would lead to a 55% answer rate (0.34*1.63). This is a 21% absolute difference, which means I would have to pay about 5 players (5x50 silver = 2 gold and 50 silver) to get one more answer. And in the worst case scenario, giving the 50 silver incentive would lower the amount of players answering. I don't think that 50 silver incentives is worth it for surveys about latency. Perhaps results would be different with another kind of survey or question, or wording.
Latency appeared to be stable between experiments, even if the hour of the day and the day of the week weren't the same. I found players to be quite friendly and patient in their answers.
Acknowledgments
I (Tekai) used to play on the Kronos I server as a level 60 warlock for months in a progression raiding guild, I wrote extensively about it, contributed to bugreports, quality improvements, discussions and have published the (now completed) Kronos WoW Comics (humoristic drawings inspired from the server) on the official Project Kronos Facebook page. I don't have any financial interest in this survey and its results and I am not financed in any way by the Kronos team or anyone else.
This is mostly an exercise of curiosity and for fun (yes my fun is wierd), please don't take it as a scientific study, but go ahead and do refer to it if you find it interesting.
Data :
Intervention group, 24 players whispered, from bottom /who to top, 2 AFK, 1 offline
17:17:40 [W From] <49:_>: 99ms in EU (_)
17:18:28 [W From] <57:_>: 53 ms in EU.
17:18:33 [W From] <22:_>: dont know how to check, 45, EU
17:20:14 [W From] <17:_>: hi i already said around 60 on another char, no need to send again :p, no problemo
17:20:10 [W From] <60:_>: USE 170 ms on avg.
17:20:35 [W From] <17:_>: my average ms is 166, I live in the US on the East Coast.
17:22:09 [W From] <15:_>: I'm happy to help, No money neccessary, :), 167 UK/EU
17:23:37 [W From] <60:_>: I'm EU (_) and have 47 MS
17:24:38 [W From] <60:_>: 18ms atm
17:24:23 [W From] <60:_>: idk what to tell u its usually good, idk what the average latency is or whatever, ya i knnow mine 195 right now
Control group, 20 players whispered , 2 AFK, 4 missing
17:26:43 [W From] <60:_>: I cant see latency, sorry
17:28:18 [W From] <60:_>: 100-175ms, usa based
17:30:05 [W From] <60:_>: I have already answered on another character and answering twice would provide lackluster results, right?, No worries, You are doing gods work :)
17:30:34 [W From] <60:_>: hi, today I have experienced some heavy lag spikes, but since tha made the network changes, latency have been good overall
17:30:59 [W From] <60:_>: hey, im from Russia and i have 50 ms atm. Usually its around 60ms
17:33:12 [W From] <2:_>: EU and right now its 73, no problem thanks you too
17:33:28 [W From] <60:_>: all from 50-100ms, but there is always the weird server lag especially if you encounter ppl from NA as European
17:33:48 [W From] <42:_>: 16-31ms
17:35:07 [W From] <8:_>: Greetings sir, ofc i can help out with that, Im in _ and my latency varies from 15ms-30ms, same to you!
17:33:32 [W From] <60:_>: sry i dont speak english
*For a total of 16 answers in the incentive group for 39 whispers and 19 answers in the control group for 42 answers.
Aucun commentaire:
Enregistrer un commentaire