How do we know which program elements contributed to outcomes?
In leading programs that incorporate a variety of offerings, resources, and tools, we often seek to understand which of these elements most contributed to outcomes. One of the standard ways we evaluate these programs is to conduct an impact analysis that isolates the variables to identify those that most contributed to the intended outcomes.
However, in instances where it may not be feasible to conduct a rigorous study, there are still ways we can gather input from program participants about the offerings that, for them, most contributed to the outcomes or changes they report. To do this, we can embed the old contribution question into our program surveys. The contribution question enables us, at a very minimum, to invite the program participants to evaluate for themselves which elements of the program most contributed to their learning, particularly once they have had time to transfer the learning to their own contexts.
Framing the “Contribution Question” in Program Surveys
Step 1. First ask participants to describe their changes.
Select from a variety of item types to first ask participants to rate or articulate the changes they experienced as a result of participating in the program. We may use a retrospective pretest to guide participants to reflect upon their changes in performance from before to after participating in the program. Or we may ask open-ended questions about what changes participants have observed in their practice as a result of participating in the program.
Step 2. Ask participants which elements have contributed to their changes.
The second step is to directly ask participants which elements have contributed to their stated changes, like so: “There are many elements of the program that may have contributed to your changes. Which of the following elements of the program do you believe most contributed to the changes you identified? Please rank them in order from 1 = most contributed to your change to 7 = least contributed to your change.” Then list a few of the key (not all!) elements of the program that you wish to assess.
Reporting Results of the Contribution Question
Use a contribution map to rank and report the most frequently reported selections, making sure to focus on the top 3-4 mentions in addition to total mentions. In the example below, I selected the top three mentions to review because there were seven total options the participants could select. Once you rank the responses by their top mentions, you can use many analytic tools to apply data bars or a heat map to help illuminate the range of the results (see two examples of the same report below).
You may notice from the results that the “site visit days” was the most frequently mentioned, and was one of the top three mentions of a greater percent of respondents than any other element. The “back to school PL kickoff” and the “coaching from an instructional specialist” also received a high number of mentions, which may indicate they could be considered for continuation in the program, depending on the context.
While a self-reported contribution analysis may not apply a statistically rigorous design to isolate the program elements that contribute to outcomes, such a tool can prove useful for program developers seeking quick input about what is working in their own contexts.