Designing fit-for-purpose evaluations

FSC works closely with project partners to design evaluations that align with each innovation project’s purpose, context, and goals. We avoid a “one-size-fits-all” approach, recognizing that the relevance of different evaluation designs is dependent on a project’s level of development, existing strength of evidence, and scale. Our approach generates actionable insights for service providers, evidence of what works for policy makers, and ideas on how to take each project to the next level in terms of implementation, scale, and evaluative rigour.

Evaluation designs

Broadly speaking, our evaluations use one of three designs:

1. Effectiveness evaluations

Effectiveness evaluations focus on understanding to what extent projects produce target outcomes and the factors that support success. These evaluations assess the links between program activities and intermediate outcomes that align with the project’s logic model, and provide important information about how well a project is working in a particular context.

Effectiveness evaluations also seek to understand what implementation factors are associated with success. This helps partners strengthen the execution of projects and hone their day-to-day operations. This type of evaluation is most appropriate to projects in the early stages of development for which both establishing baseline levels of outcomes and a better understanding of implementation success factors are critical.

Effectiveness evaluations are most appropriate for interventions that have not been formally evaluated and will be the design most commonly used for our innovation projects.

2. Impact evaluations

Impact evaluations focus on estimating the causal impact of a project on target outcomes. We estimate impact using rigorous experimental or quasi-experimental designs. By generating meaningful evidence about causal impact, we can identify which solutions within the FSC portfolio are most effective and cost-effective, and inform decisions to scale up or replicate solutions to address Canada’s most pressing skills development challenges.

Impact evaluations are most appropriate for innovation projects that already have evidence of effectiveness and potential to be a scalable solution to our most pressing skills development challenges.

FSC’s next call for proposals will have a stream focused on interventions that are ready for impact evaluation.

3. Systems change evaluations

Systems change evaluations are used to assess the feasibility, value, and sustainability of innovation projects that are testing out new approaches to skills development policy, program, or practice, or new forms of collaboration to achieve shared goals.

These evaluations focus on understanding the potential of projects to act as systems change catalysts by influencing the broader skills development ecosystem to adopt more effective and efficient approaches. They focus on collecting information from stakeholders about the progress of the initiatives, their perceived effectiveness and value, and the support that the initiatives need to succeed. These evaluations can employ early stage effectiveness assessments and/or on later-stage impact evaluation methods depending on the strength of the project’s existing evidence base and where they are situated in the evidence pipeline.

Systems change evaluations benefit from collaboration with partners including employers, funders, and other stakeholders. Future Skills Centre is engaging with Provinces and Territories to explore opportunities for partnership and enable systems change projects that benefit Canadians, employers, and the wider ecosystem.