Lessons learned from the YSS Program Grant Stream Evaluation
The Youth Success Strategy (YSS) is an evidence-informed initiative from United Way Greater Toronto (UWGT) to support young people in the development of the skills necessary for meaningful employment and successful transitions to adulthood.
In March 2017 Blueprint was contracted by UWGT to design a monitoring and evaluation strategy for three streams of YSS—Career Navigator, netWORKS, and YSS Program Grant Stream—to help UWGT, its partners and its donors better understand the characteristics, progress and outcomes of the youth participating in the programming. The resulting framework is the first of its kind and will generate evidence about the effectiveness of each stream, as well as the initiative as a whole.
To ensure the framework was sensitive to the needs of vulnerable youth, we engaged UWGT and its partner agencies in a co-design process that tailored data collection tools to reflect the experiences of the population they serve—including youth who are Indigenous, transgender, racialized, living with a disability, in conflict with the law, and at risk of or experiencing homelessness. Several options were investigated to ensure the proposed solution was not only operationally feasible for all three YSS streams, but also took into consideration the 16 unique agencies with programs funded through the YSS Program Grant Stream.
Our solution was a collection of surveys tools that could capture the same meaningful data from each of the three program streams; a solution that will allow UWGT to evaluate all three streams using similar indices and a common language. By incorporating modularity into the tools, only relevant information at the appropriate time is collected from participants. This allows us to simultaneously evaluate YSS funding streams regardless of their diverse timelines, demographics and goals. Where necessary, we further customized the tools to ensure they were sensitive to the needs of participants served by each agency. We also created a customized YSS database that would facilitate survey deployment and centralize data collection and reporting.
Starting in the spring of 2017, we piloted our evaluation strategy and database with 15 of the agencies funded through YSS Program Grant Stream. We also adapted our database and data collection tools to be culturally appropriate for the 16th agency. To date, almost 500 youth participating in programming through these agencies have used our survey tools.
These tools have allowed us to provide UWGT with insights into the demographics, needs, strengths and goals of YSS participants and have informed two preliminary evaluation reports.
The challenges that we, UWGT and the YSS agencies experienced during implementation have allowed us the opportunity to further develop our evaluation strategy to address some shortcomings of the initial evaluation approach.
Blueprint is in the process of updating our strategy and data collection tools to incorporate the learning from the YSS Program Grant Stream pilot. This process is further informing an update to our database to customize it to netWORKS and Career Navigator, as we prepare to deploy our evaluation strategy with these remaining two funding streams.