Commentary: Insights into the Development of the Brief Observation of Social Communication Change (BOSCC)

A recent review of early behavioral interventions for autism spectrum disorder6 suggested that many intervention trials are of “low quality”18, in part due to the limited ability of researchers to interpret and evaluate efficacy. Most early interventions focus on teaching social communication skills, core deficits in children with ASD. However, measures currently used to quantify social communication skills, such as the Autism Diagnostic Observation Schedule (ADOS-2)14 were developed to be primarily diagnostic and lack the sensitivity to capture subtle changes over relatively brief periods2,4. The field of ASD intervention research needs measures that can identify subtle changes in core ASD symptoms.

Mullen Scales of Early Learning Receptive Language domain 15 also showed statistically significant decreases on the BOSCC, suggesting that the changes seen in the BOSCC align with changes seen in other commonly used standardized measures.

Benefits of the BOSCC
The BOSCC is a measure of global ASD change and may be particularly useful in intervention trials that focus on improvements across a range of ASD symptoms. The BOSCC was developed to be flexible for use across different sites, intervention types, and naturalistic social contexts. While our initial work applied the BOSCC coding scheme to parent-child free play videos, a standard administration, conducted by a researcher who is unaware of the child's treatment status and timepoint, is recommended. This standard administration was developed in order to eliminate potential bias by parents who are aware of the child's treatment status or timepoint and to provide a context in which changes observed in the child could be distinguished from changes observed in the parent (who may have received training) or the parent-child dyad. We have also had success in applying the BOSCC to other naturalistic social interactions such as segments of ADOS videos 11 . Researchers who are interested in using the BOSCC should evaluate which context would be most feasible and effective for their study, so long as the context is play-based and social.
Video-based coding offers many benefits. Videos capture the totality, richness, and complexity of a behavior, allowing the observer to dissect nuanced behaviors and to explore the dyadic unfolding of social behaviors. Videos also allow for repeated and varied analyses. For example, videos that are used for BOSCC coding may be used for a variety of other purposes, such as coding parent fidelity of intervention or coding other child behaviors unrelated to the BOSCC, such as a child's motor skills, creating a standardized context for a language sample or focusing on more specific features, such as the use of various consonant sounds. Researchers that have gathered videos from previously conducted intervention trials may also be interested in applying the BOSCC coding scheme to applicable videos retrospectively. Use of video-based observations also provides opportunities for advancements in telemedicine. For example, rural study participants could gather video data in their homes without requiring visits from research staff or attendance at clinics.
Unlike other commonly used measures, such as the ADOS-2, which can require up to 60 minutes of both the participant's time as well as the time of an experienced clinician, the BOSCC coding scheme is applied to 10-to 12-minute videos, significantly decreasing the burden on participants and highly skilled clinician researchers. Unlike the ADOS-2 and other standardized measures which require a high level of expertise and training to be able to administer and score reliably, the BOSCC was developed to be used reliably by less experienced individuals. The BOSCC can be administered and coded by post-baccalaureate research assistants. On average, with guidance, training, and participation in BOSCC consensus meetings, our trainees typically became reliable within 12 BOSCC videos.
The BOSCC also offers an appropriate context in which to employ automated algorithms or body sensors that record physiological activity (e.g., skin conductance, heart rate) to detect change. For example, we are working with researchers with expertise in machine learning techniques to generate automated coding of change over time using audio recordings from BOSCC sessions. This work would provide new approaches to capturing change that would not rely on human observation alone, but rather combine multiple methods (human observation and computer automated processes) to maximally identify changes in response to intervention.

Limitations of the BOSCC
Results of the initial study were promising but limitations highlight the preliminary nature of this work. Our first study was based on a relatively small sample of children, although many other researchers have now taken an interest in using the BOSCC and are exploring its utility in new, larger samples. While the BOSCC was developed to be flexibly applied to a range of playful social contexts, researchers should be mindful of the limitations of certain contexts and interacting partners. For example, when applying the BOSCC coding scheme to parentchild interactions for parent mediated interventions, it is important to keep in mind that changes in BOSCC scores could be related to changes in parental behavior or in the parent-child relationship. Similarly, the BOSCC may not be independent of other changes in the child's development, such as changes in language or cognitive skills. This highlights the importance of ensuring consistency between pre and post observations (e.g., same social context, same environment, same materials) in order to most accurately capture the child's behavioral change separate from other changes that may be occurring.
Though the short length of BOSCC administrations is a benefit, the BOSCC results are, consequently, a very brief snapshot of the child's behavior, a limitation also present with the lengthier ADOS-2. Though short observations are commonly used to assess change in early intervention trials 7,8 , the question of whether we can truly gather a representative sample of the child's behavior within such a short period of time remains unanswered. There are ways to promote a useful snapshot of the child's behavior within a short time frame. For example, researchers can ensure that the interaction is comfortable, fun and natural by giving the child "warm-up" time with the play partner before the BOSCC administration begins, confirming that the child is not tired or ill when the sample is gathered, and asking caregivers whether the sample gathered was representative of the child's behavior. There may be instances when a second video sample should be gathered or when a video sample should not be used, such as if the child is unusually irritable.
Our data suggested that researchers should be cautious when using the four BOSCC items in the RRB domain. First, research assistants with less experience with ASD may require more training to be able to reliably identify RRB behaviors compared to Social Communication behaviors. Second, these items showed highly skewed distributions such that the majority of children did not demonstrate these behaviors during the time-limited BOSCC sample. These results are consistent with RRB items on the ADOS-2 over a longer period and with other studies that have highlighted the difficulty with measuring RRBs by observation 1,10,20 . Some researchers have suggested that accurate observation of RRBs may require multiple observations in different contexts or may need to incorporate parent report 12,20 . Though our direct observation method may have been influenced by possible novelty of materials at the first time point 10 , our initial results found stability in RRBs over short periods of time (test-retest reliability was adequate), countering this possibility. Nevertheless, additional research should continue to explore ways to accurately observe and quantify RRBs in children with ASD as well as to maximize the benefits and minimize the limitations of direct observation of RRBs.

Next Steps
While we hope the BOSCC will help researchers identify efficacious interventions, the BOSCC will not answer all questions for all interventions. The BOSCC coding scheme may not adequately quantify changes during interventions that are of low intensity, occur over a very short period of time, or whose primary teaching focus is on a highly specific skill (e.g., pointing) or non-ASD-specific skill (e.g., gait) 16 . Though it is likely that some participants will not demonstrate change or will not "respond" to treatment, it is also possible that participants will not generalize improvements made in treatment to the video context in which the BOSCC was applied. For example, if the intervention is conducted in school by teachers, the BOSCC may not show significant changes when applied to parentchild interactions at home.
Our initial work focused on toddlers and preschoolers who were minimally verbal, or equivalent to ADOS-2 Toddler or Module 1. We anticipate that the BOSCC will follow a similar modular format based on age and language level as the ADOS-2. The BOSCC coding scheme that is applicable to verbally-fluent individuals with ASD is currently under development. While the BOSCC was developed to be applicable to children with ASD, it is possible that the coding scheme may be effective at capturing change in children with other neurodevelopmental disorders, particularly those with overlapping social and/or communication difficulties. For example, researchers outside our lab are currently exploring the utility of the BOSCC in a sample with Fragile X. Studies like this will contribute to our understanding of the efficacy of the BOSCC in non-ASD populations.
Our work and the work of other researchers will continue to contribute to our understanding of the benefits and limitations of the BOSCC. Researchers who are interested in using the BOSCC may attend trainings to learn about the administration and coding of the BOSCC. We are also in the process of developing an online coding system, which would allow researchers to upload videos to be coded by BOSCC-reliable research assistants. Researchers who choose to use the online coding system would not need to attend a BOSCC coding training, would not need to train coders in their own lab, and could receive truly "blinded" feedback without going to extensive lengths to keep examiners and coders separate. An online coding system would also improve clinical applicability of the BOSCC by allowing clinicians to submit videos without needing to receive training or score videos themselves in order to have a reliable metric of change.
Questions remain about the utility of the BOSCC in quantifying changes in ASD symptoms. We will continue to explore, as will other researchers, what is a "clinically meaningful" amount of change on the BOSCC and when, over the course of an intervention, we might expect to capture this amount of change. Future research will assess what it means for interventions if the individual does not show a clinically meaningful decrease on the BOSCC as well as when changes in the BOSCC do not align with clinical judgment or changes in other standard measures. The BOSCC is one of many steps in the development of innovative ways to identify beneficial treatments for individuals with ASD.