A Comparative Study of Online Remote Proctored Versus Onsite Proctored High-stakes Exams

A Comparative Study of Online Remote Proctored Versus Onsite Proctored High-stakes Exams

Authors

  • John A. Weiner
     PSI Services LLC, 611 N. Brand Blvd. 10th Floor, Glendale, CA 91203
  • Gregory M. Hurtz
     PSI Services LLC, 611 N. Brand Blvd. 10th Floor, Glendale, CA 91203

Keywords:

Remote Proctoring, Remote Online Proctoring, Remote Invigilation, Equivalence of Proctoring Conditions, Test Security, Preventing Online Test Fraud, High-Stakes Testing, Computer-based Testing, Internet-based Testing

Abstract

Advances in technology have spurred innovations in secure assessment delivery. One such innovation, remote online proctoring, has become increasingly sophisticated and is gaining wider consideration for high-stakes testing. However, there is an absence of published research examining remote online proctoring and its effects on test scores and the examine experience. This paper describes a quasi-experimental field study carried out with three professional licensing examinations administered concurrently at different test sites that offered either onsite proctoring in testing centers or remote online proctoring in computer kiosks where the testing was proctored via Internet-connected video communication and surveillance. Results using both classical test theory and item response theory methods revealed substantial reliability and a strong degree of measurement equivalence across proctoring conditions. Candidates revealed slightly less positive reactions to some of the remote proctored testing conditions, but reactions were positive overall and had virtually no relation to test performance. Overall, the results of this study support the equivalence of kiosk-based remote online proctored exams and exams proctored onsite in test centers.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Downloads

Published

2017-04-13

How to Cite

Weiner, J. A., & Hurtz, G. M. (2017). A Comparative Study of Online Remote Proctored Versus Onsite Proctored High-stakes Exams. Journal of Applied Testing Technology, 18(1), 13–20. Retrieved from http://jattjournal.net/index.php/atp/article/view/113061

Issue

Volume 18, Issue 1, 2017

Section

Articles

References

AERA, APA, NCME (2014). Standards for educational and psychological testing. Washington DC: American Educational Research Association.

Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey: Brooks-Cole.

Arthur, W. A, Jr., Glaze, R. M., Villado, A. J., & Taylor, J. E. (2010). The magnitude and extent of cheating and response distortion effects on unproctored internet-based tests of cognitive ability and personality. International Journal of Selection and Assessment, 18, 1–16.

Arvey, R. D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivational components of test taking. Personnel Psychology, 43, 695–716.

Bauer, T. N., Truxillo, D. M., Sanchez, R. J., Craig, J. M., Ferrara, P., & Campion, M. A. (2001). Applicant reactions to selection: Development of the selection procedural justice scale (SPJS). Personnel Psychology, 54, 387–419.

Bauer, T. M., Truxillo, D. M., Tucker, J. S., Weathers, V., Bertolino, M., Erdogan, B., & Campion, M. A. (2006). Selection in the information age: The impact of privacy concerns and computer experience on applicant reactions. Journal of Management, 32, 601–621.

Bedford, W., Gregg, J., & Clinton, S. (2009). Implementing technology to prevent online cheating: A case study at a small southern regional university (SSRU). MERLOT Journal of Online Learning and Teaching, 5, 230–238.

Bedford, D. W., Gregg, J. R., & Clinton, M. S. (2011). Preventing online cheating with technology: A pilot study of remote proctor and an update of its use. Journal of Higher Education Theory and Practice, 11, 41–58. Retrieved August 11, 2016 from http://www.na-businesspress.com/ jhetpopen.html

Berkey, D., & Halfond, J. (2015, Jul 20). Cheating, student authentication and proctoring in online programs. New England Journal of Higher Education.

Brothen, T., & Peterson, G. (2012). Online exam cheating: A natural experiment. International Journal of Instructional Technology and Distance Learning, 9(2), 15–20.

Cluskey, G. R., Jr, Ehlen, C. R., & Raiborn, M. H. (2011). Thwarting online exam cheating without proctor supervision. Journal of Academic and Business Ethics, 4. Retrieved from http://www.aabri.com/jabe.html.

Dunn, T. P., Meine, M. F., & McCarley, J. (2010). The remote proctor: An innovative technological solution for online course integrity. The International Journal of Technology, Knowledge and Society, 1, 1–7.

Gilliland, S. W. (1993). The perceived fairness of selection systems: An organizational justice perspective. Academy of Management Review, 18, 694–734.

Gilliland, S. W. (1994). Effects of procedural and distributive justice on reactions to a selection system. Journal of Applied Psychology, 79, 691–701.

Kantrowitz, T. M., & Dainis, A. M. (2014). How secure are unproctored pre-employment tests? Analysis of inconsistent test scores. Journal of Business and Psychology, 29, 605-616.

Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014). Cheating, reactions, and performance in remotely proctored testing: An exploratory experimental study. Journal of Business and Psychology, 29, 555–572. DOI: 10.1007/ s10869-014-9343-z.

Lievens, F., & Burke, E. (2010). Dealing with the threats inherent in unproctored internet testing of cognitive ability: Results form a large-scale operational test program. Journal of Occupational and Organizational Psychology, 84, 817-824.

Lilley, M., Meere, J., & Barker, T. (2016). Remote live invigilation: A pilot study. Journal of Interactive Media in Education, 1, 1–5. DOI: http://dx.doi.org/10.5334/jime.408.

Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

Luxton, D. D., Pruitt, L. D., & Osenbach, J. E. (2014). Best practices for remote psychological assessment via telehealth technologies. Professional Psychology: Research and Practice, 45, 27–35.

Nielssen, O., Dear, B. F., Staples, L. G., Dear, R., Ryan, K., Purtell, C., & Titov, N. (2015). Procedures for risk management and a review of crisis referrals from the MindSpot Clinic, a national service for the remote assessment and treatment of anxiety and depression. BMC Psychiatry, 15:304, 1–6. DOI: 10.1186/s12888-015-0676-6.

Nye, C. D., Do, B., Drasgow, F., & Fine, S. (2008). Two-step testing in employee selection: Is score inflation a problem? International Journal of Selection and Assessment, 16, 112–120.

Okrainec, A., Vassiliou, M., Jimenez, M. C., Henao, O., Kaneva, P., & Ritter, E. M. (2016). Remote FLS testing in the real world: Ready for “prime time.†Surgical Endoscopy, 30, 2697–2702.

Plaus, K., Boren, L., Brazell, T. P., Wickett, J., & Weber, A. (2015). Remote proctoring test delivery: A report on options and considerations. Washington, DC: Institute for Credentialing Excellence.

Prince, D. J., Fulton, R. A., & Garsombke, T. W. (2009). Comparisons of proctored versus non-proctored testing strategies in graduate distance education curriculum. Journal of College Teaching & Learning, 6(7), 51–62.

Rodchua, S., Yiadom-Boakye, G., & Woolsey, R. (2011). Student verification system for online assessments: Bolstering quality and integrity of distance learning. Journal of Industrial Technology, 27(3), 2–8.

Settle, J. R., Robinson, S. A., Kane, R., Maloni, H. W., &Wallin, M. T. (2015). Remote cognitive assessments for patients with multiple sclerosis: A feasibility study. Multiple Sclerosis Journal, 21, 1072–1079.

Tippins, N. T., Beaty, J., Drasgow, F., Gibson, W., Pearlman, K., Segall, D. O., & Shepherd, W. (2006). Unproctored internet testing in employment settings. Personnel Psychology, 59, 189–225.

Tomasi, L. F., Figiel, V. L., & Widener, M. (2009). I’ve got my virtual eye on you: Remote proctors and academic integrity. Contemporary Issues in Educational Research, 2, 31–35.

Wright, N. A., Meade, A. W., & Gutierrez, S. L. (2014). Using invariance to examine cheating in unproctored ability tests. International Journal of Selection and Assessment, 22, 12–22.