Evaluation
Evaluation is the best way to determine both the effects of your programming and ways to improve it going forward. Evaluation has become a critical and permanent feature of the Foundation’s grant review process. Below you will find resources to help you plan and carry out evaluation of your grant project. We have included definitions of evaluation terms, sample evaluation plans, a downloadable blank evaluation plan for your use and suggestions of books, websites, etc. If you have further questions relating to the evaluation of your project, please contact Evaluation and Research Officer, Elizabeth Tharp at 812-437-8260 or at etharp@welbornfdn.org.
Evaluation Terms
Sample Evaluation Plan
Blank Evaluation Plan Matrix
Resources/Links
Evaluation Terms
Funders, evaluators and practitioners do not always use the same terminology when referring to different components of evaluation plans. Below you will find the terms used by Welborn Baptist Foundation.
Evaluation Matrix: (Evaluation plan, evaluation model, logic model, dashboard) Clear representation of the elements in a complete evaluation plan including program outcomes, indicators, targets, data sources and data collection plans. This chart is a tool provided for you to plan and organize your evaluation as well as to help you keep your plan on track throughout the grant cycle.
Outcomes: (outcome goals, expected outcomes, results) Skills, behaviors, life conditions, or knowledge that is changed or improved as a result of an intervention or program. These changes can occur to individuals, groups, communities, policies and systems. Outcomes must be capable of being measured and must be realistic within the given time frame. For Welborn Baptist Foundation, grantees will be focused on goals that can produce some results within or just beyond the grant year. For example, a ‘stop-smoking’ program may desire to achieve a lowered rate of chronic disease as a result of a reduction in smokers. While we need to remain focused on those ultimate outcomes, these health benefits take longer to develop and would not be evident within the Welborn grant year. It is realistic however to expect that this effort may get smokers to sign up for the program and that a portion would be smoke free by the end of the grant cycle.
Key question: What will be different as a result of this program?
Indicators: Measurable conditions that show an outcome is achieved; signs or observable evidence of changes taking place. Using our previous ‘stop-smoking’ example, it is realistic to expect that the number of smokers who sign up for the program and the percentage who are smoke free by the end of the grant cycle are indicators that would tell you that the program is beginning to achieve some results on the way towards a healthier community.
Key question: If this program is working, how will we know? What are the signs that will tell us something is different?
Target Audience: (Sample) Those to whom the indicator applies; those that you are including in your evaluation sample. Your target does not need to be every person who walks through the door of your program; however your target audience should be a representative part of the larger group. Again using our ‘stop-smoking’ program, you might decide to sample every participant who has attended at least 90% of the program classes over the year.
Key questions: Who exactly will show the desired changes? Who are we measuring? Targets: (Goals) Since indicators are measurable, they are often categorized as numbers or percents. Targets are the specific amount of outcome change desired. Using our previous ‘stop-smoking’ example, a realistic target might be to expect that this effort may get 100 smokers to sign up for the program and that 80% of those who attended 90% or more of the classes over the year would be smoke free by the end of the grant cycle.
Key question: Exactly how much of a change do we expect to see as a result of this program?
Data Sources: Information sources used to show changes in the target audience as a result of your program. These can be existing records or surveys, assessments, observations, etc. that you construct or you purchase for your use. As a first step, it is always wise to determine what data sources are already in existence and are easily accessible.
Key question: What will we use to show what happened to our audience?
Person(s) collecting the data: Person(s) actually conducting the surveys, collecting data, etc. for the evaluation. This could be an evaluator, project director, program officer, or other indentified person. Ideally, this person will be knowledgeable about the evaluation plan and will often (but not always) be the contact person with Welborn Baptist Foundation for the evaluation component of the project.
Key question: Who is responsible for collecting our data?
Time Intervals of Data Collection: (Data Interval) The time points when data are collected. Data can be collected at specific time frames such as every three months, at the start and end of a program, at the end of a program then at some future time point to measure follow-up, etc.
Key question: At what points of time will we collect our data?
Back to the Top
Sample Evaluation Plan Matrix
The examples below are provided to give you a better understanding of how these evaluation plan elements fit together. You can think of a matrix as an overall summary; the ‘short hand’ of an evaluation plan. Keep in mind these are not comprehensive examples and each project is different.
|
|
Outcomes
|
Indicators of
Success
|
Target Audience
|
Targets
|
Data
Sources
|
Person(s)
collecting data
|
Time Intervals of Data Collection
|
| |
What will be different as a result of this program? Changed or improved skills, attitudes, knowledge, behaviors, etc.
|
Indicators are observable evidence of change. If this program is working, how will you know? What are the signs that will tell you something is different? |
Who are you evaluating? Specific group that is being evaluated. May be the entire audience or a more specific part of that audience |
Exactly how much of a change do I expect to see as a result of this program? (Specific numbers, percentages) |
What will I use to show what happened to my audience? Tools used to measure outcomes |
Who is responsible for collecting data? |
At what points of time will you collect your data? |
| Christian School – Twelve Month Mentoring Program for Middle School Boys |
Boys in program will improve their academic skills
Boys in program will reduce negative behavior at school
|
Grade point average
Number of behavior referrals
Self report and reports from significant relationships
|
All boys 6th – 8th grade in school who received 4 or more behavior referrals in semester prior to program and who attend program for at least 9 of 12 months (40) |
80% of 40 (32) will show GPA improvement and will not receive more than one behavior referral during time period (Jan.-Dec.) |
Report Cards, school data regarding behavior referrals
Surveys to student, guardians, mentors and teachers to capture data about changes in behavior designed by Smith Evaluation Consultants
|
Contracted with Smith Evaluation Consultants to compile, analyze and report all findings |
Beginning of the program (baseline-January) and at the end (December) |
| Community Emergency Medical Technicians – Life Saving Training Program |
Responders will be better prepared for life threatening emergencies |
Performance post-tests |
First responders in county (20) |
100% of 20 responders completing the course will score 90% or better on both post tests |
Post test that comes with the Training Program |
Supervisor will collect and grade post tests |
Post test at end of training program Spring 2011, follow-up test Fall 2011 |
| Children’s Preschool – Get Healthy Children and Families |
Children will become more physically active
Children will eat more nutritious foods
|
Number of days per week of at least 30 minutes of moderate physical activity/play
Number of days where fruits/vegetables are eaten per week
Number of parents taking pledge to continue efforts at home
|
Every child who attends daycare at least 4 days per week (100) |
80% (or 80 children) will engage in moderate physical activity 5 or more days per week and will consume more than 3 fruits and vegetables daily |
Physical activity/nutrition checklist from observation, parent phone surveys, number of parents signing pledge |
Workers will collect data & surveys, report data to supervisor monthly |
Daily checklist, surveys & pledge efforts - Jan 2011 (baseline) then Dec 2011 |
Back to the Top
Evaluation Plan Matrix Template
This blank evaluation matrix is provided for download to help you organize and illustrate how you intend to evaluate the effectiveness of your project. You are not required to submit this chart to the foundation but we strongly recommend you use this form internally to help guide and drive your evaluation plans throughout your project. The number of goals will vary by project. Your project may have several outcome goals or just one or two. Carefully select your outcomes to be as close to your expected results as possible.
The blank evaluation matrix (in MS Word format) can be downloaded from here.
Back to the Top
Resources/Links
These are resources that others have found helpful in their work with program evaluation. This list is not exhaustive. We would like to continue to develop valuable resources for our grantees. If you know of books, websites, journals, etc. that you have found useful and that might be beneficial to others, please let us know at etharp@welbornfdn.org. Please also let us know if any of these links are invalid.
http://www.managementhelp.org/evaluatn/fnl_eval.htm - Free Management Library. Basic Guide to Program Evaluation (Including Outcomes Evaluation). Provides guidance toward planning and implementing an evaluation process for for-profit or nonprofit programs. Includes many additional resources.
http://www.managementhelp.org/evaluatn/outcomes.htm -Free Management Library. Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.
http://www.socialresearchmethods.net/kb/index.php - Social Research Methods knowledge base. Includes resources and links to other locations on the Web that deal in applied social research methods.
http://www.rwjf.org/pr/product.jsp?id=18657 Robert Wood Johnson Foundation, A Guide To Evaluation Primers. Produced by the Association for the Study and Development of Community (ASDC) for the Robert Wood Johnson Foundation, this document offers an orientation to handbooks and basic primers on evaluation. These resources are designed to meet the needs of the non-expert.
http://www.eval.org The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation.
http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html . University of Wisconsin Extension. Program Development & Evaluation. Guides designed to help Extension faculty better plan and implement credible and useful evaluations. They also may be useful to agencies or funders who are seeking assistance with realistic evaluation strategies. Publications can be viewed or printed from the links.
http://www.fastennetwork.org/ Faith& Service Technical Education Network. Networking opportunities and informational resources. Within this website you will find a link to Outcome-Based Evaluation: A Training Toolkit for Programs of Faith: http://www.urbanministry.org/files/outcome-based%20evaluation.pdf
http://www.heritage.org/Events/2007/03/Making-Good-Works-Great-Works-How-Outcome-Based-Evaluation-Helps-Faith-Based-Organizations-Change-Lives Outcome based evaluation begins a cycle of continuing improvement and greater success. Hear three experts tell their personal experiences with organizations, donors, and the public policy community in helping to make good works great works. Claudia Horn President, Performance Results, Inc., Karen Woods Executive Director, Empowerment Resource Network, Calvin W. Edwards, Founder and CEO, Calvin Edwards & Company. March 15, 2007.
Back to the Top