Measuring SMSD Skills in Literacy Learners Using United Way's Outcomes Measurement Model

Quinte Adult Day School
Belleville & Trenton, Ontario

Our research question

We wanted to tackle far too much initially and have struggled to define what was possible (and necessary) within the confines of the project. In fact, narrowing and refining our initial question has been one of the biggest challenges we have faced with our action research project thus far.

In the final draft of our project plan submitted in August our research questions had evolved to:

  1. How do we recognize, measure and evaluate changes in self-management and self-direction (SMSD) skills in literacy learners?
  2. How do we show that SMSD skills are equally important to achieving “success” within the formal education/training and employment streams as the more traditional literacy skills?
  3. How do we identify the parallels and similarities between our SMSD outcomes and indicators with MTCU’s Learner Skills Attainment framework to collect and analyze data of value to both funders?

The objectives of our project are:

  1. Develop tools to measure and track changes in learners’ self-management/self direction skills
  2. Pilot the tools at Quinte Adult Day School (Belleville and Trenton) and with two additional community literacy programs in the province.
  3. Liaise with MTCU to inform them of our process, incorporate new developments emerging from their Learner Skills Attainment Framework into our project and suggest ways they could measure outcome measures of SMSD skills within their LSA framework.

Progress

Major accomplishments

  • We were trained in United Way’s outcome measurement model and developed outcomes and indicators that we wanted to track. We created a logic model. We adapted our information and referral tracking to collect additional data that we were not already collecting.
  • We did an extensive literature review on a number of related areas that came up during our planning and development phases.
  • We developed an observation tool to assist us in identifying, measuring and tracking desired changes in learners’ SMSD skills. We created an activity sheet to make it easier for staff and volunteers to track progress and indicators.
  • We adapted a Learner Scan from What Works: Recruitment and Retention of Ontario Works Clients – Phase Two to ensure that staff and learner shared the same perspective. We created an Activity Log for staff and volunteers to track daily activity and observations.
  • We are currently in the process of piloting the tools at Quinte Adult Day School with seventeen learners across our two sites. We have already identified several changes we will make before asking two other agencies to pilot the tools.

Challenges

  • Finding a common language in which to articulate the SMSD skills has proven difficult. We wanted to link as much as possible with MTCU’s Learner Skills Attainment initiative so had initially planned to articulate the SMSD skills using four of the essential skills (oral communication, working with others, continuous learning and thinking skills).

    We found this difficult for two reasons. First, we struggled to fit concepts like self-confidence and self reflection under the essential skills categories. Second, we realized that the Ontario Literacy Coalition's SMSD outcomes measured the skill level of the learner while the essential skills are a measure of complexity of a specific task.
  • As expected, using clear language on the learner scan that could be understood by lower level learners was challenging. We simplified several items after our initial pilot phase.
  • Demonstrating the importance of SMSD skills as a foundation to success across all goal paths is much more challenging than strictly showing acquisition of and/or improvement of the skills themselves. Perhaps over time we will be able to show that learners with stronger SMSD skills are more “successful” than those with lower SMSD skills across all LBS levels. The very definition of what “success” constitutes has been a challenge. Here we have defined “success” primarily by the definition used by MTCU.
  • Coordinating our efforts with those of MTCU’s Learners Skill’s Attainment initiative has also proved more difficult than originally expected as they are not as far long with development of their framework as we thought they would be at the time we submitted our plan.

What we've learned so far

It is not surprising that it has been our challenges (i.e. refining our question or articulating our tool) more than our accomplishments that provided much of our learning so far. We found the discussions in our literature review of SMSD skills especially interesting and relevant to our daily realities in the classroom. Some of the research that explores the development of SMSD skills question whether they are in fact “skills that can be learned and transferred” or are they rather “traits that are acquired through relationships with others”. Many suggest that strength in these areas is more a matter of “self-awareness” than of “self-management”, some even objecting to the negative connotation of the term itself. (Grieves)

We have often observed how important an empathic relationship between facilitator and learner is in the development of SMSD skills. We have also noted significant changes in behavior as individuals begin to feel they are in a safe, supportive and trusting environment. We have observed learners with lower core literacy skills make tremendous improvements in their SMSD skills, achieving more significant change and “success” in their lives than some of their counterparts who possess higher core literacy skills but have not achieved the same growth in their SMSD skills. This is, in fact, what inspired our interest in this research in the first place.

Return to Top