Department of Human Services

Appendix G: Challenges and learnings

Timeframe and sample size

Due to time constraints incumbent upon the YJAIS pilot project, four weeks were allocated to complete the screening assessments. Subsequently, the participant group is relatively small. Therefore, while this data provides a valuable snapshot of needs across a brief period of time, the reliance that can be placed on the generalisability of the data from this cohort across the overall population is limited. Prior to the commencement of assessments, there was a hope that the collated data could be used to illustrate the complexity of needs among the population (defined by falling below average on three or more areas assessed); however, the data was unclear on this due to only six participants not meeting this criterion.

Ideally, a longer period of time would have provided the opportunity to include a greater number of participants and allow for all assessments to be completed across a higher number of participants. This would add greater richness to the available data, and the potential for more statistically stronger data analyses.

Staff Resources

The nature of the screening project posed challenges for staffing resources within the YJAIS team and the AYTC team more widely. The movement of young people to and from assessment sessions placed considerable strain on AYTC operational staff, who were required to accommodate the increased workload among the usual resident movements between residential units, school and other appointments.

Additional staff-related challenges that were noted throughout the screening period were:

  • Several YJAIS team members were employed on a part-time basis, limiting availability and available time for assessment completion
  • YJAIS owns single copies of all assessments, therefore assessment resources were required to be shared among team members across two campuses
  • Competing demands impacting the availability of rooms and access to young people during the screening period.

Assessment instrument considerations and limitations

Throughout the screening period, several intricacies and oddities were detected through the repeated administration of particular assessments. A particularly significant limitation of all of the assessment tools was the lack of norms for Aboriginal and Torres Strait Islander young people. A brief outline of other issues is included below:

Sensory Profile

Challenges arose in administering the Adolescent and Adult Sensory Profile (AASP). The assessment is intended to be used as a self-assessment, however the language used in the questions is occasionally complex, with the use of ‘double-negative’ language. The literacy, cognitive and intellectual challenges prevalent was expected to pose a challenge and the AASP was therefore conducted in a structured interview format. Regardless of these challenges, it is the opinion of the YJAIS/YEC Occupational Therapists, that in some situations the AASP was insufficient in identifying areas of sensory-related need for individuals towards the younger end of the assessment’s normed spectrum. This was evidenced by a number of participants self-reporting sensory-related needs that were not reflected in their assessment results and multiple instances where results did not reflect therapist or staff observations. On two instances, the Occupational Therapists completed both the Sensory Profile 2 (completed by a staff member familiar with the young person) and the AASP with particular individuals when such discrepancies were noted (and participant ages allowed this). Results showed considerable disparity between assessment scores and in these cases, the SP2 was believed to reflect staff and therapist observations more accurately. The reason for these disparities is unclear, however the implication of this is the belief that the AASP results may have under-represented the needs of some participants, possibly reflecting a limited capacity for insight or self-reflection. This is a known limitation of self-report measures.

CELF-5 v TNL-2 Results

There were inconsistencies between the results of the two key language assessment tasks; the Clinical Evaluation of Language Fundamentals – 5 (CELF-5) Screening Test and the Test of Narrative Language 2nd Edition (TNL-2).

Several factors may have contributed to this finding. More than half (69.6%) of those who completed the TNL-2 were outside of the age range for the normative data, as they were older than 15 years 11 months. Hence their standard scores and percentile ranks were being compared to a younger age range.

The order of administration for the assessments was the CELF-5 Screening Test followed by the TNL-2. A number of struggling participants only completed the first assessment and then chose to stop due to significant challenges. This means that several young people who may have performed poorly in the TNL-2 did not go on to complete this assessment.

Unlike the CELF-5 Screening Tool, the TNL-2 is not normed against the Australian population. Due to it being normed against the American population, this may lead to discrepancies and inaccuracies in the scores.

It is hypothesised that young people may perform better with the TNL-2 because this has more visual information (pictures) to support young people to respond. In addition, this assessment examines functional narrative skills needed for everyday conversation which may be a strength for many young people in youth justice, and may in particular be a cultural strength for Aboriginal young people. Conversely, the CELF-5 examines foundational language skills, and those required more in an academic environment. These skills may be less necessary for many young people involved with youth justice, such as those who engage with school infrequently.

Weschler Abbreviated Scale of Intelligence - Second Edition (WASI – II)

The WASI-II presented some limitations for several reasons. It is not normed on an Australian population, potentially creating a disadvantage to this population with the verbal subtests. Other concerns regarding language were raised for the young people whose first language differed from English.

Youth Level of Service Inventory/Case Management Inventory 2.0 (YLS/CMI 2.0)

Ideally, the YLS/CMI 2.0 would have been completed for every young person who was involved in the screening project. However, the YLS/CMI 2.0 presented some challenges in administration due to the length of time taken to investigate collateral information. More preparation time prior to the commencement of the project may have enabled sufficient investigation of collateral information so that during interview with the young person, less information needed to be sought, thereby reducing the length of interview time.

The Adolescent Psychopathology Scale – Short Form (APS – SF)

One major limitation of the APS-SF was identified and that was the lack of cultural considerations. The APS-SF is very strongly influenced by a diagnostic (medical model) conceptualisation of mental health.

Data-informed v therapeutic application of assessments

Although consideration was made regarding ideal assessments for project purposes, the assessments used in the screening project were selected from the pool of available resources which were initially purchased for informing therapeutic interventions rather than the gathering of statistically-significant data for research. At the time of purchasing YJAIS’s multi-disciplinary assessment resources, details about and mechanisms that would later be employed in the screening project’s data analysis were unknown. As a result, while some assessments used within this project have featured in published research studies, other assessments utilised may not be considered as a gold standard for research or data-collection purposes. Should a similar screening project process be completed in the future, consideration could be given to the assessment tools utilised, to ensure that the data collected is as representative of the population as possible, and an optimal balance between qualitative and quantitative, objective and self-report information is achieved.

Page last updated : 02 Sep 2020

State Government of South Australia © Copyright DHS .[sm v5.4.7.1]