Abstract
Background Interest has been growing in the use of augmented reality (AR) based social communication interventions in autism spectrum disorders (ASD), yet little is known about their safety or negative effects, particularly in head-worn digital smartglasses. Research to understand the safety of smartglasses in people with ASD is crucial given that these individuals may have altered sensory sensitivity, impaired verbal and non-verbal communication, and may experience extreme distress in response to changes in routine or environment.
Objective The objective of this report was to assess the safety and negative effects of the Brain Power Autism System (BPAS), a novel AR smartglasses-based social communication aid for children and adults with ASD. BPAS uses emotion-based artificial intelligence and a smartglasses hardware platform that keeps users engaged in the social world by encouraging “heads-up” interaction, unlike tablet- or phone-based apps.
Methods A sequential series of 18 children and adults (mean age 12.2-years-old, range 4.4-21.5-years-old) with clinically diagnosed ASD were given the opportunity to use BPAS to learn emotion recognition, face-directed gaze, and managing transitions. Users and caregivers were interviewed about perceived negative effects of using BPAS, and had an opportunity to highlight any hardware or software design issues.
Results The majority of users were able to wear and use BPAS (n=16, 89%). Caregivers reported no perceived negative effects in users during or after use of BPAS. Two users reported temporary negative effects: eye strain, dizziness, and nasal discomfort due to the smartglasses nose stabilizers. Most users and caregivers did not have any design concerns regarding the smartglasses hardware or software (users 77.8%, caregivers 88.9%). The only reported design concern was that the smartglasses became warm to the touch during extended use.
Conclusions It is important to conduct research to understand the feasibility and safety associated with new emerging technologies for vulnerable populations such as ASD. This report found no significant negative effects in using an AR smartglasses based social communication aid across a wide age range of children and adults with ASD. Further research is needed to explore the efficacy and longer-term effects of such novel interventions.
Introduction
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder affecting 1 in 68 U.S. children (1), and is characterized by social communication impairment and the presence of a restricted and/or repetitive range of interests and behaviors (2). The rising prevalence of ASD has increased demand for educational and behavioral services, often exhausting these limited resources (3, 4). There has been increased focus on the development and study of technology-aided approaches for the social, cognitive, and behavioral challenges related to ASD (5). Technology-aided approaches may be especially suitable for people with ASD given that it has been reported that some of these individuals may have a natural propensity to utilize digital tools (6), display a fondness for electronic media (7), express a preference for standardized and predictable interactions (6), enjoy game-like elements (8) and favor computer-generated speech (9). Technology may also have negative effects in some people with ASD. Individuals may become “addicted” to video games, and can become agitated or disruptive when attempting to disengage from playing video games (10). Anecdotally many caregivers describe meltdowns and other episodes of behavioral dysregulation in children with ASD when attempting to stop them playing smartphone- and/or tablet-based videogames.
Evidence suggests that a broad range of technology-aided interventions, such as those using computer programs and virtual reality (VR), may be effective in people with ASD (5). Technology-based interventions have been found to be beneficial for improving a wide range of skills and behaviors including aiding social and emotional skills (11-13), communication skills (12), academic skills (14), and challenging behaviors (11).
There is particular interest in interventions that help users learn skills while continuing to interact with the people and environment around them. Learning social-emotional skills in real life settings (such as in social skills groups), may increase the chance these skills generalize to the challenges of daily life (15). Augmented reality (AR) is a technology that holds considerable promise in this regard, as it allows for users to see and interact with the real world around them, while virtual objects and audio guidance are provided through a visual overlay and audio speakers. AR incorporates many of the features of VR that are thought to make VR technology well suited to the creation of learning tools for people with ASD (16), including being a primarily visual and auditory experience, being able to individualize the experience, and promoting generalization and decreasing rigidity through subtle gradual modifications in the experience (16). AR experiences can also be easily modified and personalized for each individual, an important consideration given that many people with ASD exhibit intense interest in a restricted range of topics, and may experience extreme distress if changes to their routine/environment occur (2). AR experiences are also not restricted solely to real-world limitations on time, space, and resources. Users may have the opportunity to interact with objects or experiences from historical or fantasy worlds, or a simplified cartoon-like interaction where the sensory and perceptual experiences may be reduced in complexity and/or magnitude.
Most ASD-related research into AR has focused on the use of smartphone- and/or tablet-based apps. While research has been limited, AR apps on smartphones/tablets have been shown to improve selective and sustained attention (17), attention to social cues (18), the ability to understand emotions and facial expressions in storybook characters (18), and navigating the physical world when attempting to find employment opportunities (19). However, smartphone-based AR may carry with it a risk of negative effects, including grip and postural strain, minor falls, and falls leading to major trauma and blindness (20, 21).
It is know that people with ASD face major challenges in coping with transitions, such as being asked to stop playing video games (10), it remains to be seen whether stopping the use of a more distracting/cognitively demanding smartphone/tablet results in more difficulty than encountered when stopping use of smartglasses (22). While there may be important differences between how people with ASD experience and use smartglasses compared to screen-based smartphones and tablets, little comparative research has been conducted. People using smartphones and tablets look directly down at the screen of these devices, and use their hands to interact with touch sensitive displays. These individuals may become distracted from real world events around them, performing less well on attentional tasks that are simultaneously demanded (22), and becoming less aware and engaged with the physical and social world around them (23). It could be conceivable that using smartphones may be an adaptive behavior in some individuals with ASD to minimize the stress and sensory stimuli that they encounter from the real world. Smartglasses may not provide the same level of real world disengagement, as users appear to require less cognitive load to use them, and appear to become less distracted by them compared to smartphones (22). Smartglasses require users to continue to remain heads-up and visually engaged with the real-world around them. Unlike screen-based devices, smartglasses feature a transparent optical display, thereby requiring users to continue to pay attention to the physical environment environment around them as they use the device. Additionally, people with ASD face major challenges in coping with transitions in activities (2) such as being asked to stop playing video games (10), so it remains to be seen whether stopping use of a more distracting/cognitively demanding smartphone/tablet results in more difficulty than encountered when stopping use of smartglasses.
While AR has been investigated as an educational strategy in ASD children for at least a decade (24), minimal research has been conducted into the safety of head-mounted AR in ASD populations. Head-mounted AR, delivered on smartglasses, may have a number of notable advantages compared to smartphone- or tablet-based AR. Despite these gaps in knowledge, it is thought that the adoption of AR smartglasses will be driven by their ability to improve social interactions and relationships, make life more efficient, and provide enjoyment and fun to the user (25). AR smartglasses contain a wide array of sensors that can collect quantitative data that may well help research efforts in the digital phenotyping of neurobehavioral conditions (26). To our knowledge, only a couple of reports have described the experience of people with ASD with modern VR headsets (27, 28), and to our knowledge, we have published the first report of the use of AR smartglasses in children with ASD (26).
There remains a considerable need to investigate how children and adults with ASD would respond to social and behavioral coaching provided through AR smartglasses (26). Important initial considerations are determining the safety of such an approach and assessing potential negative effects. People with ASD may have sensory, motor, and cognitive challenges that warrant special attention when using such technology. Firstly, ASD is often accompanied by hyper- or hypo-reactivity to sensory inputs such as touch, sound, and sight (2). Altered sensory reactivity is not only prevalent in the ASD population but it is also highly heterogeneous in nature, and may represent a series of sensory subtypes (29). It is therefore important to see if individuals can tolerate wearing smartglasses for an extended period while monitoring how they respond to the visual and auditory cues delivered through the device. Secondly, ASD is often associated with altered motor movements, either seen as repetitive behaviors that are identified as “core” symptoms of ASD, or impairments of motor coordination. It is important to assess how motor challenges may affect how people with ASD utilize AR smartglasses (30). Additional considerations should include the ability of people with ASD to remain attentive and be able to focus on using smartglasses as part of social communication training, especially given the high rate of comorbidity between ASD and attention deficit hyperactivity disorder (31). VR research has reported potential side effects that include eye strain, headache, and disorientation during use of a VR headset (32), although a recent brief report noted that people with ASD experienced relatively few negative effects when using a modern VR headset (28). Instruction manuals for AR smartglasses identify potential side effects as being nausea, motion sickness, dizziness, disorientation, headache, fatigue, eye strain, dry eyes, and seizures (33), although their occurrence among users with ASD has not been studied. It may be challenging to assess the negative effects of such technology in people who may already experience challenges in communicating their experiences, therefore researchers should carefully observe users with ASD, ask for their feedback, and also seek feedback from their caregivers in order to have a more sensitive method of detecting any negative effects. Interviews and questionnaires intended to probe the acceptability of these technologies need to use especially clear and concise language, with the use of visual or other communication aids as appropriate.
Aim of Research
In this paper we report on safety and negative effects of using the Brain Power Autism System (BPAS) in children and adults with ASD. BPAS is a social communication aid that consists of AR smartglasses with apps that allow children and adults with ASD to coach themselves on important socio-emotional and cognitive skills (26). Skills include emotion recognition, socially salient face-directed gaze, self-regulation, and transition management (26). Users learn through gamified interactions and a combination of intrinsic and extrinsic rewards for successfully completing tasks. In certain situations, such as coaching of appropriate face-directed gaze and emotion recognition, BPAS is designed to be used while the user is interacting with another person. The system was designed using an iterative process where continuous feedback from people with ASD, clinicians, neuroscientists, educators, caregivers, design specialists, and engineers, helped to develop the system that was used in this report. Additionally, the facial affective analytics component of BPAS was developed in partnership with Affectiva, an emotion artificial intelligence company. The work was also made possible by Google, Inc., now known as Alphabet, Inc., who provided substantial hardware and well as guidance in engineering. Brain Power, the company that developed the BPAS, has been a long-term Glass partner in the Glass Enterprise Partnership Program. BPAS is designed to be accessible to people with ASD, and to minimize potential negative effects. A number of elements were used to achieve this, including but not limited to the use of calming tones, use of emotional artificial intelligence, minimization of audio and visual sensory load, graduated transitions between learning segments, and modification of the functionality of the tactile input surfaces of the smartglasses. In this study the focus was on understanding the safety and potential negative effects that may be experienced by children and adults with ASD as they used AR smartglasses delivering cognitive and social self-coaching apps.
Methods
The methods and procedures of this study were approved by Asentral, Inc., Institutional Review Board, an affiliate of the Commonwealth of Massachusetts Department of Public Health.
User Recruitment
A sequential sample of 18 children and adults with ASD were recruited from a database of individuals who completed a web-based sign up form expressing interest in our study (mean age 12.2 years, range: 4.4 – 21.5 years; Table 1). Users included males and females, both verbal and non-verbal, and represented a wide range of ASD severity levels. Users had Social Communication Questionnaire scores from 6 – 28, with an average of 18.8. The Social Communication Questionnaire is a validated way to obtain diagnostic and screening information about ASD (34, 35). Written consent was obtained from the legal guardians of children and from cognitively-able adults. Participants between 7 and 17 years-old provided written assent, when possible. In this report, every user was accompanied by a caregiver, and users and caregivers could exit the session at any time and for any reason.
Exclusions
Individuals who had expressed interest via the website signup but who had a known history of epilepsy or seizure disorder were not enrolled in this study. Users who had any uncontrolled or severe medical or mental health condition that would make participation in the study predictably hazardous were also not enrolled.
Data Collection Procedure
Users and caregivers had the opportunity to wear BPAS. In this report, the base smartglasses technology that was utilized was Google Glass Explorer Edition (Figure 1). Users who could physically wear the smartglasses for at least one minute were allowed to proceed to trying the different BPAS social and cognitive coaching apps over a period of 1-1.5 hours. The level of variability required in the session length to use the range of apps was reflective of the considerable range of ASD severity in the user group. Users interacted with their caregivers while they practiced with the apps, and were required to take off and then put the smartglasses back on (Figure 2). Following the experience with the system, structured interviews were conducted with users and their caregivers. The structured interviews focused on identifying any perceived negative effects of using the system, and allowed for comments/concerns to be raised about the design of the smartglasses hardware and the apps.
Results
Sixteen of the 18 users (89%) tolerated wearing BPAS smartglasses for at least one minute. The two users who did not tolerate this initial testing did not use BPAS apps. Both of these users and their caregivers did not report any adverse effects, but they did not express an interest in wearing BPAS and continuing the testing session. It was noted that both users were not verbal, and were relatively young, aged 5.5 and 5.8 years. Of the remaining users, 13 out of 16 users (81%), and 16 out of 16 caregivers (100%) reported no minor negative effects, and 100% of caregivers and users reported no major negative effects (Table 2).
The three instances of user negative effects were all mild in nature, transitory in duration, and did not result in the session being stopped. The reported negative effects were one case of dizziness, one case of eye strain, and one instance of initial discomfort. The caregiver of the user experiencing dizziness reported that the effect may be related to the user not wearing his prescription glasses, and he had previously experienced similar dizziness when he had tried a modern VR headset. This same user also experienced initial discomfort with the nose pads, but resolved the discomfort with adjustment of the placement of the smartglasses. The user who had complained of eye strain resolved the issue with a 20-second break in testing.
The majority of users and caregivers did not have any design concerns about the system (77.8% and 88.9% respectively) (Table 3). The only design concern highlighted by users and caregivers was that the smartglasses became warm to the touch during use, although this did not result in any negative effects.
Discussion
There are a wide range of technologies that are being developed to help individuals with ASD overcome their social, behavioral, and educational challenges. Because ASD is associated with a range of sensory, motor, and cognitive issues, it is important to investigate the safety and negative effects of using these technologies.
This paper explored the use of BPAS, a novel technological system that uses AR smartglasses to deliver social and cognitive coaching to children and adults with ASD. The focus was on exploring safety and negative effects of using this technology across a broad age and severity range of children and adults with ASD.
The system resulted in few reported negative effects; those that were encountered were minor, and did not stop the testing of BPAS. All three the negative effects were reported by users, and there were no caregiver reported negative effects. This highlights how important it is to continue to engage and learn from children and adults with ASD, and not just rely on the reporting of others. Two users who were both non-verbal could not initially wear the smartglasses for a sustained period, and were therefore unable to try testing the system. This suggests that a small group of people with ASD may struggle to utilize current AR smartglasses. Importantly in this population, there were no observed major negative effects, including behavioral episodes (tantrums or meltdowns), seizures, or falls.
Design concerns were raised with the smartglasses hardware that BPAS utilizes, with two caregivers and four users describing a feeling of warmth from the external side of the hardware after extended use. However, this did not result in any reported negative effects.
The relative lack of negative effects is an important finding across such a wide age and severity range of people with ASD, and indirectly supports recent research demonstrating minimal negative effects when modern VR headsets were used by people with ASD (28). It was encouraging to see that despite the ASD-related sensory, motor, and cognitive issues, the majority of users were able to wear the smartglasses and use the system without any notable negative effects. It was also reassuring that there were no behavioral problems reported when users were asked to stop using the smartglasses, especially given earlier outlined concerns regarding the potential for distress around transitions involving technology.
Such findings may not generalize to other AR smartglasses, which may be based on different/unmodified hardware, or may be using other software apps. There remains a critical need to conduct further research to understand the feasibility and safety associated with new emerging technologies, especially those that may be used in vulnerable populations such as ASD. The use of AR smartglasses may have considerable potential as an augmentative technology in helping people with ASD, particularly when they are shown to be usable and safe in the ASD population, and when supported by robust evidence of efficacy. Finally, while this report does not identify any short term adverse events, as with any technology, further research is warranted to explore the positive and negative effects of longer term use of AR smartglasses.