Part 3: Better Items for Better Meaning
The ubiquity of self-service survey tools in the data landscape today has democratized the survey administration process. Affordable, easy-to-use survey tools have bolstered the prevalence of evidence-based management in education, where data scarcity had previously inhibited our understanding of what is working in our particular contexts. Fifteen years ago, we were clamoring for more data. Today, we pass around reports like free lollipops.
As exciting as it is to have a thriving data culture, our survey management practices in education have had significant unintended consequences, some of which were described in Part 1 of this series. In short, we hear murmurs that our teachers experience survey fatigue, our free lollipop reports aren’t that useful, and we are making some bizarre inferences about Likert-scale data. Some have even begun to devalue self-report data entirely because their experience shows that fancy reports fail to translate to actionable improvements.
Garbage In, Garbage Out
In short, we are experiencing a “garbage in, garbage out” phenomenon. Contrary to the common messaging around survey design by vendors who sell survey tools today, the selection and design of survey items is a highly nuanced artform. Like any other specialization, it takes a significant amount of expertise to clearly identify the goals and outcomes of a particular program, analyze the context, and propose instruments and tools to measure what is important in each context. The quality of the data that will be produced in the final report is exceedingly dependent on the designer’s level of expertise in measurement, the depth of their knowledge of your initiative and aims, and the breadth of their understanding of organizational theory.
Better Measurement
Outside of context, it is a challenge to propose item types that may lead to better measurement for better evidence for all. The reader would be advised to consult with a seasoned evaluator who can serve as a thought partner and support an approach for instrument development and refinement. However, here are a few item types I’ve grown to love because of their power to tell stories of impact, growth, and change. And even better, they are item types that can be combined and embedded into both surveys and collaborative assessments, which empower participants to take ownership of their own learning and transformation.
Item #1. Evaluative Rating Scales
| Description |
| Embed evaluative language directly into the scale instead of using an agreement scale. Report frequency distribution of scores for each progression level. |
| What is Evaluated |
| Perceived quality, relevance, ease of use, etc. |
| Sample Usage |
| Replace agreement levels with terms that thoroughly describe each level (e.g., “I am aware of this practice; I have experimented with this practice; I regularly use this practice; I regularly share with others how to use this practice”). |
Item #2. Analytic Rubrics
| Description |
| Use analytic rubrics to capture self-report data along a progression of practice for multiple key indicators. Report frequency distribution of scores for each progression level. Interpret by identifying which behaviors are needed to achieve the next level on the progression. |
| What is Evaluated |
| Content quality; Individual or group behaviors or competencies |
| Sample Usage |
| Embed short-form rubrics or analytic rubrics into survey-based solutions as self-assessments. Use analytic rubrics with individuals as observation tools or with groups as collaborative assessments. |
Item #3. Gap Models with Analog Rating Scales
| Description |
| Use an analog rating scale to gather self ratings on key indicators that are theoretically related. Report the gap between the related indicators. Interpret which gaps are the largest for most participants. |
| What is Evaluated |
| Content or service quality; Individual or group behaviors or competencies |
| Sample Usage |
| Embed gap models into survey-based solutions as self-assessments or program evaluations. |
Item #4. Retrospective Pretests
| Description |
| Use an evaluative rubric or analog rating scale to capture self-report data. Ask participants to rate both before and after participating in a program or experiencing a service. Report the frequency distribution of scores for before and after. Interpret which gaps are the largest for most participants. |
| What is Evaluated |
| Content or service quality; Individual or group behaviors or competencies |
| Sample Usage |
| Embed retrospective pre-tests into survey-based solutions as post program evaluations. For learning and development offerings, use live retrospective consenograms at the end of program sessions to quickly get input instead of a survey, and use a survey-based solution 3-5 months afterwards, once participants have had time for learning transfer. |
Item #5. Maturity Models
| Description |
| Use maturity models to assess self-report data along a stage of maturity for key indicators. Report frequency distribution of scores for each progression level. Interpret by identifying which behaviors are needed to achieve the next level on the progression. |
| What is Evaluated |
| Systems or organizational stage of maturity |
| Sample Usage |
| Embed maturity models into survey-based solutions as self-assessments. Use maturity models with groups as collaborative assessments. |
Wrap-Up
The items described above demand clarity and intentionality about design, particularly in instances when we wish to guide participants to self-reflect in a meaningful way. When crafted carefully, such item types have the power to tell a story of impact and growth. In addition, these tools can be quite educative for program participants. Empowering stakeholders to use the tools in a collaborative setting communicates that the purpose of the data collection is not about surveillance and compliance, but about our shared desire to work together to self-assess, transform, and sustain desirable performance.
References
Danks, S. (2020, January). Measuring impact after the fact: Using the retrospective pretest design to evaluate professional learning impact in the face of complex change. Flower Mound, TX: ARKEN RESEARCH. Retrieved from: https://www.arkenresearch.com/resources
Danks, S. (2019, November). The Ultimate Measurement Mash-up: Retrospective Rubrics for Measuring Complex Change. TD Magazine.
Danks, S., & Allen, J. (2014). Performance-based rubrics for measuring organizational strategy and program implementation. Performance Improvement Quarterly, 27(1), 33‒49.