Outreach: Program Evaluation

Assess the success and impact of a program or project

Background

Program planning and evaluation are a significant component of any type of outreach effort, especially if one’s role is the director of the outreach initiative. Individuals with the role of Science Outreach Director can work in a variety of settings e.g. nonprofit, academe or government.

Focusing on evaluation or assessment part of the planning process is important, because it is critical to know if the goals and objectives of the program are being achieved. Thus, each activity or event must be planned with a mission statement lens. For example, if one goal is to train scientists in the method and practice of teaching, several program activities such as a workshop should be geared towards this mission. The knowledge gained through evaluation also aids in acquiring resources through the development of a pitch and summary of impact. Such resources are volunteers and/or collaborators, equipment and/or monies in the form of donations or grants. Therefore, the tasks explored here involve skills that are useful to develop, and have the potential to profoundly effect on the success of a project/program.

Note: The in-depth program evaluation explored here is usually done by a professional evaluator. The evaluator is not typically program staff, but may represent a stakeholder.

The Process

The process below includes categories of information and activities that are necessary to plan and evaluate the effectiveness of a program. Steps 1-5 may be represented in a logic model. A logic model or framework is a tool used for the planning, implementation and/or evaluation of a project/program.

  1. Establish the goals and objectives of the project/program
  2. Evaluate the availability of resources
  3. Determine your activities
  4. Record your anticipated outcomes and impact
  5. Choose a method to evaluate the goals/objectives of the project/program
  6. Run the evaluation and record data
  7. Determine if you have evidence to support step 1
  8. Create a report that summarizes step 6
  9. Update the logical model and program with the appropriate changes

The Exercise

Program X partners local scientists and teachers to teach natural or physical science modules in kindergarten through 6th grade classrooms. Modules must includes hands-on lessons that are active, investigative and meet the Next Generation Science Standards (NGSS). The main goals and/or objectives of Program X are as follows:

  1. Introduce students to a diverse set of role models in science.
  2. Create sustainable partnerships in support of science education.
  3. Expose students and teachers to inquiry based learning.
  4. Give scientists experience and confidence in teaching.
  5. To increase the use of hands-on teaching strategies in the classroom.

Program X is funded for an additional two years and, you must make sure that the goals of the program are being met. The school district is especially interested in this evaluation; since, there is an opportunity to include additional schools in the upcoming academic year.

For this exercise, you are the Executive Director of the Science Education Consortium that serves several elementary school districts and your task is to evaluate Program X.

Task 1: Critique of a logic model
Critique of a logic model
A logic model, program matrix or logical framework is a graphical depiction of the relationships between program elements. These elements include inputs, outputs and outcomes. Logic models are used for planning, executing and properly assessing a project/program. Before starting this task, familiarize yourself with examples of logic models from informalscience.org, such as “A Start to Developing a Logic Model” and “Logic Model for Agricultural Literacy Programming“. Below is a simplified logic model for Program X containing program inputs, outputs, outcomes and evidence. The model does not includes the program goals/objectives (see “The exercise”), a summary of external factors or any assumptions.Your job is to correct the logical model and provide a rationale for your edits. Critique the model by answering the following questions:

  • Are the inputs adequate to generate the appropriate outputs?
  • Are the outputs adequate to meet the short and mid term outcome?
  • Are the outcomes aligned with the program goals/objectives?

Inputs and Resources  Outputs  Outcomes  Evidence

  • Funding sources
  • Instructional materials
  • Staff with expertise in education
  • School district administrators
Activities:
Teacher and scientist partnership modules

Participants:

  • Teachers and their students
  • Scientists (postdocs and grad students)
  • Increase in student learning
  • Increase in confidence to teach science
  • Alignment with NGSS
  • Ability to teach hands on lessons
  1. Post Program survey
  2. Classroom observations
  3. Follow-up survey
  4. Student assessments

After completing this task, check your corrected logic model with a sample corrected logical model provided. How do they differ? Logical models will vary depending on the author and place of employment there’s no right or wrong answer.

Task 2: Evaluate data for Program X
Evaluate Data for Program X
During the previous school year, Program X partnered 37 scientists with 18 teachers in 15 teams that served approximately 600 students. Evaluation data for Program X is available below:

  • 78% of the students surveyed agreed with the following statement: “I want to be a scientist when I grow up.”
  • The percentage of students that viewed science as “fun” changed from 23% to 62% after Program X.
  • 60% of the partner teachers feel as if they have access to resources that will aid them in teaching science.
  • 34% of the partner teachers feel as if they have learned strategies for hands on teaching.
  • 82% of the partner teachers are more likely to use inquiry based science lessons.
  • 67% of the graduate students gained confidence in teaching.

Keeping the program goals in mind (from “The Exercise”), examine the data and answer the following questions:

  • Is the data useful?
  • What are the general trends of the data?
  • Did you have sufficient survey participation?
  • What is the biggest impact?
  • Are there any problems with these data?
  • Does the data support the goals of the program described in “The Exercise”?
  • If you require more data, what type of data do you need?

Write a summary that contains the information above. This summary would be presented in a meeting to your staff that implemented Program X. This task represents steps 7 and 8 of the process outlined above. Step 6 has been hypothetically completed and provided for the purpose of this simulation.

Task 3: Prepare a report
Using your results from Tasks 1 & 2, create a report that includes:

  1. The description of the program and goals. These are listed in the Exercise section.
  2. A description of the population surveyed.
  3. The original and your updated logical model. Include a few statements about why you changed the logical model
  4. Answers to questions from Task 2.
  5. Your overall assessment of Program X and any new or refined goals for the program. If you propose new goals, be sure to include your rationale for those changes.

This report is what you would present to the school district or the sponsors. This task represents step 8 and 9 in the process outlined above.

The Deliverable

For Task 1, edit the logical model and write a few sentences explaining your changes. This logical model would be discussed with your staff and, once finalized, become part of a grant application.

Task 2 requires a written summary but, you may choose to present the data in a drawing, graph, etc. as appropriate. This summary should contain all of the data and, any accordant commentary.

Write a brief report for Task 3. Remember, your audience is the school district,potential collaborators, and sponsors. The report should be broad, clear and concise.

Full reports can be lengthy, for the purposes of this simulation, the report should be no more than 2-3 pages and include the following:

  • Title
  • Description of the program and goals
  • Description of the population surveyed
  • The corrected logical model (deliverable from Task 1)
  • Evaluation data from Task 2
  • Your assessment of the data (deliverable from Task 2)
  • Final assessment and any recommendations for the program

All of these tasks can be completed in a word processor.

Resources

General resources to help you get going:

Also, view this example of program evaluation from the viewpoint of a funding agency.

Skills

For this task:

  • Knowledge of science, education, and outreach
  • Communication skills, written
  • Time and program management
  • Analytical skills
  • Attention to detail
  • Creativity

Needed in this career:

  • The ability to simplify and/or translate complex concepts
  • Analytical and computer skills
  • Marketing
  • Knowledge of science, education and outreach
  • Communication skills, verbal and written
  • Computer skills
  • Time and program management
  • Interpersonal skills: teamwork/collaboration
  • Attention to detail
  • Creativity

Additional Responsibilities

A professional in the field of regulatory affairs may also perform these activities:

  • Develop educational resources
  • Evaluate project or program progress and/or outcomes
  • Develop program evaluation methods
  • Train fellow educators
  • Search and apply for relevant funding sources
  • Marketing
  • Establish and manage relationships with volunteers and vendors
  • Budget management
  • Write and speak to diverse audiences

You are viewing a job simulation. To get started, set up SMART Goals to perform this simulation in a reasonable timeline. If you have completed the task, fill out the Self-Reflection Sheet.

Simulation author: Corin White, PhD
Simulation vetted by professionals in the Bay Area