By David Seelow, PhD©
Last Wednesday afternoon (2/21/18) I led a post conference workshop at the Association of Test Publishers annual Innovations in Testing conference held in San Antonio. My basic goal was to bring leaders in test publishing and assessment into a more serious conversation with leaders from the game design and production profession. My hope was that this conversation would push the test publishers into a more active role in supporting the design of simulations and video games as alternatives and options to traditional multiple choice and constructed response exams. I felt the best first or maybe second step- as the conversation has been ongoing at ATP for a few years now- would be to have assessment leaders learn more about how games for assessment can be used, but, even more important, play and experience some of these cutting-edge games on site.
I began by pointing out to the audience the need for the assessment community, like the teaching community, to close the gap between the way this generation and future generations are and will be learning through rapid, highly interactive, multimedia-based experiences, with a past generation’s assessment formats. After all, students do not even bring paper and pencils/pens to class anymore where I teach!
Second, everyone played an ice breaker- the old childhood, but still perennial favorite Rock, Paper, Scissors. A playful attitude- open minded, exploratory and imaginative, helps create conditions that afford engaging test creation. After this minute of active learning, G. Tanner Jackson, a cognitive scientist from the famed Educational Testing Service in Princeton, New Jersey, provided the framework for assessing the effectiveness of Game Based Assessments. Like traditional tests, video game assessments must meet the following criteria: validity, reliability, fairness, and security.
With the stage set, leaders from the game design profession introduced a series of games that were set up on four large round tables. All participants were encouraged to play and talk about any or all these games in a 40- minute play session with the game industry experts at the tables to guide play and answer questions. It was a very hands-on and intimate environment for experiencing the power and potential of video games to assess learning. I reproduce the basic game descriptions below and suggest contacting the respective designers for more extended information about their respective games.
I Victoria VanVoorhis, the founder of Second Avenue Learning in Rochester, New York, and her Director of Learning and Assessment, Ben Paris, presented three education games.
Historical Friction, Martha Madison and Martian Math [Insert screen shot of one of the games]
In Historical Friction, players manage budgets and make strategic decisions to preserve (or tear down) landmarks. In our augmented reality game, Martian Math, players answer traditional math questions but have a game-based motivation: incorrect answers allow flying saucers to abduct your farm produce. These three games represent points on a spectrum of game design: Martha Madison is an immersive environment. Historical Friction in more structured in its navigation and was used to build awareness in addition to measuring skills. Martian Math is closest to a traditional skill development activity with added game-based elements.
II Randy Brown, Vice President and Division Manager at Virtual Heroes, a division of Applied Research Association, in North Carolina, presented a game that assesses patients with Traumatic Brain Injury.
Expedition is a game used to evaluate the severity of Traumatic Brain Injury (TBI) for clinical trials studies. In this PC-based game, the user is presented with three different experiences requiring them to perform tasks such as packing luggage for a trip, working through a metro system to reach a destination, and observe and interact with content in a virtual museum environment. For this workshop, the luggage event will be available with varying durations of the virtual “trip” to prepare for. The users are presented with a virtual environment and a description of their upcoming trip, and they must pack appropriately and in a timely manner; these challenges increase in complexity as the user experiences multiple sessions in their clinical trials, dependent on their results from prior sessions, with 300 total variations created for testing purposes.
III Jennifer McNamara, Vice President of Serious Games for BreakAway Games in Maryland, brought two games, one on assessing multi-patient medical skills in an Emergency Department and another assessing multiple competencies in Occupational Therapy.
Under a Stemmler Grant from the National Board of Medical Examiners, a collaboration was formed between Children’s Hospital LA, USC Keck School of Medicine, and BreakAway Games to develop and validate a serious game platform using a virtual emergency department (ED) to assess multi-patient management skills (MPMS) within pediatric and emergency medicine residents. Ultimately, VitalSigns demonstrates that a validated serious game platform featuring a virtual ED can both assess and improve multi-patient management skills among residents. Attendees briefly played the 2D game focused on cognitive fidelity instead of graphical fidelity and learned how it was used to assess complex multi-patient management skills.
The National Board for Certification in Occupational Therapy Navigator Competency Assessment Platform
The NBCOT Navigator® Competency Assessment Platform is a suite of online tools including mini-practice quizzes, games, and case-based simulations designed to help certificants assess competency across all areas of occupational therapy. Co-developed by The National Board for Certification in Occupational Therapy (NBCOT®) and BreakAway Games, all tools were created based on current practice and evidence-based literature. Completing the tools allows certificants to earn Competency Assessment Units (CAU) toward certification renewal, develop occupational therapy knowledge and skills, and stay current in OT practice.
IV G. Tanner Jackson presented a game designed in cooperation with American University’s Game Lab led by Lindsay Grace that assesses social interaction skills among English Language Learners.
ETS and American University teamed up to create an intentionally awkward gaming experience. In Awkward Annie players have the chance to live out their often-unrealized social dreams by intentionally saying the most inappropriate things to friends and co-workers. In doing so, the players are able to indicate, through omission of the appropriate options, that they understand the appropriate way to handle a given work interaction (e.g., conversations with friends vs strangers, bosses vs co-workers). This educational game targets pragmatics skills and is designed to be used by non-native US English speakers.
V Bobby Corrigan, Team Lead in Test Development Solutions from Prometric in Baltimore, led participants at another table in a conversation about reviewing a job role and catering the content used for its assessment in an alternative form than traditional text-based questions. We will review how task and activity-based statements can be used to frame an assessment situation, with emphasis on overt behaviors that easily lend themselves to being simulated. At the end of this exercise, participants will be able to evaluate their content to determine if their exam can potentially be simulated.
Randy Brown facilitated a debriefing of the play activity, which saw assessment experts actively playing and talking about the onsite games in an enthusiastic and serious fashion.
Following the debrief, Jenn McNamara talked about the game design team, how to select a good partner and some of the cost considerations in designing, producing and distributing a game for assessment purposes. We cannot emphasize enough the importance of partnerships and the need for assessment teams to work with expert game designers and vice versa.
Tory talked about the range of assessment items from basic multiple choice on a scantron to immersive environments. Tory also described, “The Steps to Designing an Assessment Game,” stressing the essential elements of analytics, roles, exploration decisions, and results.
After another round of Rick, Paper, Scissors, but now in a team as opposed to paired environment, Ben Paris facilitated a hands-on exercise where groups brainstormed concept documents for an original game. Tanner and Bobby ended the formal workshop talking about “How Data from Games is Being Used,” broken into two overarching components: Product (answers, outcomes, artifacts, culminating events) and Process (strategies, sequences of events and behaviors that contribute to the product.
What fascinated me about the concept exercise was how many of the competencies, skills, or knowledge sets the participants wanted to “teach” through a game were, in fact, the kind of competencies most resistant to standardized testing formats. The four major categories of competencies that they wanted to measure in a game-based environment were: collaboration, critical thinking communication, and content knowledge/application. Other popular choices included mathematical/abstract reasoning, ethics, creativity/innovation, and strategic thinking.
From those, the three groups wound up choosing to write about critical thinking, collaboration, and strategic thinking.
Either way we measure and report this, it’s clear that the interests of the group reflect higher-order thinking skills that are very much in demand and harder to measure with traditional assessment types. This result lends credibility to the argument video games are best suited for assessing and measuring these complex competencies, which are the very competencies most in demand by our increasingly complex society.
For next steps, I would love to see an Assessment Game Jam in the D.C. area that brings together leaders from the test publishing and assessment field with game designers, game design students and subject matter experts in a day long effort to create some cutting edge and testable assessment games. Once we have more assessment games in place, I am confident that the world of testing and learning will on the road today’s and future generations are looking and needing to take.
If you are interested in further information about any of the ideas or games mentioned in this blog you can contact us directly at the emails below:
David Seelow (Revolutionary Learning), firstname.lastname@example.org
Randy Brown (Virtual Heroes), email@example.com
Bobby Corrigan (Prometric), firstname.lastname@example.org
G. Tanner Jackson (Educational Testing Service), email@example.com
Jennifer McNamara (BreakAway Games), firstname.lastname@example.org
Victoria Van Voorhis (Second Avenue Learning), email@example.com
Ben Paris (Second Avenue Learning), firstname.lastname@example.org