by Lori Palen, Youth Thrive Evaluator
There’s a story about my youngest sister that has become family lore. When Katie was a little girl, maybe 5 or 6, my mom brought her into an ice cream shop. “Tell the man what flavor you want, Katie.” But Katie was too shy to tell the employee her choice. After a few moments, my pragmatic, no-nonsense mother said, “Well, I guess we’re not getting ice cream, then” and promptly left the shop.
(“And that,” we now tell my nieces, “was the last time your mother didn’t speak her mind.”)
Organizations sometimes ask me for “evaluation advice.” This type of general request is kind of like walking into an ice cream shop and staring blankly at the employee behind the counter: the whole process will go better if you speak up and get specific. In particular, being prepared to share details about your program, evaluation goals, and evaluation budget will help potential evaluators determine whether they are the right person to meet your needs, and it will help hired evaluators design an evaluation that your organization finds absolutely delicious.
Are you a chocolate kind of person, or are you more vanilla?
Program details feed into all sorts of important evaluation decisions, including what we measure, how we measure it, with whom, when, and how often. The questions below are some of the big ones, but a good evaluator will likely have others.
What is the structure of your program? What are the dosage and frequency? (Is it a classroom curriculum with 7 weekly one-hour sessions? A mentoring program that requires monthly mentor-mentee contact?)
What target population is your program designed to serve? (Do you serve a particular geographic location? Age group?) How many participants are served by your program during a specified time period?
What outcomes does your program target? (Is it designed to improve parent-child relationships? Prevent youth tobacco use?) Do you expect changes in participants to happen immediately, or does change take a while to unfold? A program logic model can be extremely helpful in focusing these kinds of conversations.
Do you just need a little something sweet, or is this a substitute for dinner?
So, you want to evaluate your program. Why do you want to evaluate your program?
Evaluation goals can range from modest to lofty. On the modest end of the spectrum, you may just want to learn what participants enjoy about your program, and what they’d like to see changed. On the ambitious side, you may want to generate scientific evidence of effectiveness that would earn your program a place on a national registry of effective programs, such as Blueprints or NREPP. The former requires a much simpler evaluation design (and fewer resources) than the latter.
Other evaluation goals I’ve heard include complying with funder evaluation requirements and generating data that will position your program for future funding opportunities. If these are your goals, be prepared to share more about what your current funder requires or what potential funders require in their application process.
Will that be one scoop or three? Haagen-Dazs or the store brand?
Like just about everything else that keeps your program running, evaluation is an expense. (More about that here.) A good chunk of evaluation funds go into wages for the people who design the evaluation, collect data, process and analyze data, and report evaluation results. The rest goes into supplies and equipment, like computers and software, copies, and participant incentives.
Expenses vary widely depending on the type of evaluation you conduct. For example, a one-hour telephone interview with 1,000 people will cost more than a 10-question Web survey with 25 people.
There are also factors that impact the cost of hourly wages, including the experience and qualifications of the individual and the type of organization for which they work (e.g., college or university, research institute, independent consultancy).
By knowing your budget up front, an evaluator will be able to propose evaluation designs and a staffing plan that fit within your available resources.
It’s Your Turn!
Have questions? E-mail me at firstname.lastname@example.org.