by Assim Kalouaz
Crowdsourcing is a sourcing model that people and organizations use to gather goods, services, opinions and micro-tasks and can be a powerful way to collect data and obtain work from large groups of people. This method is key for online experiments to develop an important ecological validity and heterogeneity from their sample size – given that the experimental design allows to verify data quality (e.g., with verification items in surveys, or monitoring the time taken to complete a task). While this method poses constraints because participants cannot be supervised, it is particularly efficient when data is purely qualitative or when the tasks performed by participants are not too lengthy and can be done from a computer or an app.
Crowdsourcing becomes more complex for experiments that really should be lab-based. Virtual reality (VR) experiments might be considered a good example of such experiments where behavioural (and sometimes physiological data) require the supervision, the equipment, and the physical space of a lab – but perhaps this has changed in the recent years.
According IBC.org (Dams, 2020), 6.4 million VR headsets have been sold during 2020, totalling 16.44 million units since 2019, with an estimation of 34 million sold by 2024 (Alsop, 2021). VR as a consumer product has reached maturity with a critical amount of consumer owned headsets ready to use for experiment participation. The democratization of VR as a consumer product has led researchers to conduct studies exploring the reliability of unsupervised crowdsourced VR data, progressing from smartphone powered handheld head mounted displays (HMDs) (Mottelson & Hornbæk, 2017; Steed et al., 2016) to a mix of different HMDs (Ma et al., 2018) to more specific popularized VR platforms such as VRChat (Saffo et al., 2020). All these studies have shown good data reliability despite a few aberrant responses for which verification items must be submitted, providing evidence that unsupervised VR studies are very much comparable to lab VR studies. Moreover, thanks to the high adoption of newer headsets that embed hand-tracking (Oculus Quest) and eye-tracking (HTC Vive Pro Eye), crowdsourced VR experiments have expanded their behavioural data collection reach beyond proxemics to oculomotor activity and hand gestures. These innovations open new doors to mixed measures experiments, as can be seen in work by Mottelson et al. (2021) who collected hand-tracking data remotely when investigating avatar movement correction as a support for motor learning.
Crowdsourcing VR experiments addresses many commonly highlighted limitations (Mottelson et al., 2021): low sample size, time-consuming studies and novelty effect that has to be taken into account. It also solves health safety concerns recently brought forth by the COVID-19 that has pushed researchers to rethink VR studies. This event has sparked an increase in unsupervised VR studies proposing unsupervised experiments in an iterative fashion, each informing the next. This has also incited the development of crowdsource VR experiment platforms (XRDRN.org), lists of validated HMD-owning crowdworkers (Ma et al., 2018) and new frameworks for remote VR experimental design (Rivu et al., 2021) with the newest studies highlighting the do’s and don’ts, mitigating a range of shortcomings and proposing guidelines that include most recent VR HMDs (Mottelson et al., 2021). Among the most common shortcomings highlighted, personal factors such as available physical space, internet speed and demographics seems to be the most important moderators to consider. Participants using their own VR equipment also grants more flexibility to the study for themselves and experimenters as they can perform tasks on their own time at their discretion, negating the safety concerns, while experimenters become exempted from the logistical and organizational work they would normally do to invite participants in the lab.
This newly validated potential of VR data crowdsourcing is not only tapped into by academics but also by companies such as VRAI who sends VR headsets when they customer don’t have one before having them returned and sanitized using UV sanitizers. In a more traditional fashion, Liminal VR, a Australian-based VR company, collects data using extremely short surveys embedded inside their apps that can be found on VR app platforms like the Oculus Store and Steam, expanding the ecological validity of their emotion-inducing experiences with huge participant pools.
Will unsupervised VR data collection become the new norm for VR experiments? While crowdsourcing comes at the cost of rethinking the experimental design to address procedural and technical limits, as well as considering sample biases, notably gender and socioeconomical disparity, it provides indisputable advantages for newer VR research. I believe that the medium, its adoption as well as the pandemic-induced rethinking of research practices have called for this technique to progressively become the new norm and I look forward to contributing to this discourse for my own research, bolstering my sample size and its heterogeneity while informing future research on the matter.
References:
Alsop, Thomas. “VR Headset Unit Sales Worldwide 2024.” Statista, 19 July 2021, www.statista.com/statistics/677096/vr-headsets-worldwide/.
Dams, Tim. “2020 In Review: Virtual Reality Gets Real.” IBC, 10 Dec. 2020, www.ibc.org/trends/2020-in-review-virtual-reality-gets-real/7106.article.
Ma, X., Cackett, M., Park, L., Chien, E., & Naaman, M. (2018). Web-Based VR Experiments Powered by the Crowd. The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018, 33–43. https://doi.org/10.1145/3178876.3186034
Mottelson, A., & Hornbæk, K. (2017). Virtual reality studies outside the laboratory. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST, Part F1319. https://doi.org/10.1145/3139131.3139141
Mottelson, A., Petersen, G. B., Lilija, K., & Makransky, G. (2021). Conducting Unsupervised Virtual Reality User Studies Online. Frontiers in Virtual Reality, 2(May), 1–12. https://doi.org/10.3389/frvir.2021.681482
Rivu, R., Mäkelä, V., Prange, S., Rodriguez, S. D., Piening, R., Zhou, Y., Köhle, K., Pfeuffer, K., Abdelrahman, Y., Hoppe, M., Schmidt, A., & Alt, F. (2021). Remote VR Studies -- A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs. 1(1). http://arxiv.org/abs/2102.11207
Saffo, D., Yildirim, C., Di Bartolomeo, S., & Dunne, C. (2020). Crowdsourcing virtual reality experiments using VRChat. Conference on Human Factors in Computing Systems - Proceedings, May. https://doi.org/10.1145/3334480.3382829
Steed, A., Frlston, S., Lopez, M. M., Drummond, J., Pan, Y., & Swapp, D. (2016). An “In the Wild” Experiment on Presence and Embodiment using Consumer Virtual Reality Equipment. IEEE Transactions on Visualization and Computer Graphics, 22(4), 1406–1414. https://doi.org/10.1109/TVCG.2016.2518135
Comments