During April 2022 – June 2022, WestEd conducted a classroom feasibility study to determine if the PS app can be implemented within an authentic education setting. WestEd recruited two 3rd grade public school teachers and their classrooms from California to participate in the classroom feasibility study. To be eligible for the study, teachers must have had prior experience teaching 3rd grade English Learners (ELs) and have at least five ELs in their classroom this academic year. The final analytic sample included 37 students who completed both pre- and post-assessments and surveys with 11 students classified as ELs based on their ELPAC scores. Overall, teachers felt that using the PS app helped their students learn new words and exploratory analysis showed the potential of the app in improving student vocabulary acquisition. On average, students scored an additional 1.46 points (p < .001) on the adapted Peabody items following the Pictoword School app implementation compared to their performance prior to implementation. This difference corresponds to an approximate 12% gain (effect size 0.49 standard deviation units). Additionally, students’ confidence in their word knowledge increased on all three VASE-adapted items.

In February 2022, WestEd conducted a student usability study to determine if ELs can navigate through the PS app and to gather feedback on EL enjoyment and challenges with PS app use. Experienced facilitators led a total of five, 45-minute, one-on-one usability sessions over Zoom. Trained researchers recorded notes on students’ feedback and reactions to the PS app, and the audio was recorded and reviewed during analysis. The study found that, overall, students understood how to progress through the PS app screens on their own, students were able to interact with the platform independently, and students enjoyed playing it. Several students wanted to keep playing beyond the allotted session time. Teachers made specific suggestions for improving the platform, such as providing clearer instructions for the puzzles and ensuring translations are accurate and supportive for students.

In November 2021, WestEd conducted a focus group to gather feedback from 3rd grade teachers on the newly developed features of the PS app and teacher dashboard. Experienced facilitators moderated a two-hour focus group session with eight teachers over a video conference call. Trained researchers recorded notes on participant’s feedback and reactions and the audio was recorded and transcribed. The study found that, overall, teachers were able to navigate through the PS app and dashboard with ease, but felt the app would need a number of significant modifications to be impactful in their classrooms. The core feedback from teachers is that the product needs to be more engaging, more reflective of the vocabulary learning expected of students, and more targeted and flexible to the needs of ELs. Kooapps should focus their development on crafting activities and gameplay that is more engaging than other available competing products. Additionally, teachers highlighted that engaging ELs requires more effort and scaffolding than would be sufficient for mainstream students.

During December 2020 – January 2021, WestEd conducted a classroom feasibility study to test whether the PS app is feasible for teachers and students to use in the classroom environment. WestEd recruited four teachers (two 3rd grade classrooms and two 3rd/4th grade combo classrooms), and their 42 ELs participated. During the study, each teacher used the PS app in their classroom for 15-30 minutes per session for a total of 5-8 sessions during the two to three weeks of implementation. Students completed a researcher-developed content assessment and student attitudinal survey at the beginning and end of the study. Pre- and post-tests of students’ vocabulary acquisition were scored for accuracy and analyzed through paired sample t-tests to explore the differences in student vocabulary knowledge that could be associated with use of the PS app. Across all students who had both pre- and post-test data, there was an average 7% gain on the Picture Vocabulary Items from pre- to post- which was statistically significant (p = .036) and had an effect size of .450. The results are early-stage but show promise given the sizable gains over a short implementation period. Researchers did not observe any changes in students' attitudes towards spelling and vocabulary from pre- to post- when analyzing student survey data and triangulating with teacher survey and interview responses. However, teachers praised the high level of engagement students exhibited when using the PS app and 76% of students reported that they liked using the app.

In September 2020, WestEd conducted a focus group to gather early feedback from 3rd grade teachers on the newly developed, student-facing features of the PS app and teacher dashboard. Experienced facilitators moderated one, two-hour focus group session with six 3rd grade teachers over a video conference call. Trained observers recorded notes on participant’s feedback and reactions. Based on the feedback from participants in the focus group, researchers recommended adding more text-to-speech features on the information page and dictionary screen to make the language more accessible to English Learners. For the Teacher Dashboard, researchers recommended streamlining the chart types and providing more explanatory information about metrics within the charts.

In August 2020, WestEd conducted a usability study to gather early feedback from 3rd grade teachers on the newly developed, student-facing features of the PS app. Experienced facilitators moderated six one-hour, one-on-one usability sessions with each teacher over a video conference call. Trained observers recorded notes on participant’s feedback and reactions. Each session was followed by a brief post-session interview and app rating, which consisted of questions and ratings about the ease-of-use, promise for classroom use, and content appropriateness. The study found that, PS app was relatively easy to navigate, and that students would be able to navigate the app well. Many teachers thought the app could be even more user-friendly if developers added in-app tutorials on how to solve the puzzles and explanations of the different features.