Module 2: Video Segment 5: Program Evaluation

Hello. Welcome to Module 2, Segment 5 of the Childhood-Led Poisoning Prevention Training on Program Evaluation. At the top, there are images featuring a father with a young daughter in his lap, a picture of a mother and child at the doctor's office, and a classroom with diverse students. At the bottom is an image of the Centers for Disease Control and Prevention logo, which also shows that CDC is a part of the Department of Health and Human Services. This segment is designed to assist trainees in developing plans to conduct an evaluation of jurisdictional childhood-led poisoning prevention programs. Best practices are offered for measuring progress toward program goals and outcomes, finding opportunities for program improvement, communicating success stories with stakeholders, and providing justification for program sustainability. These are the requirements for childhood-led poisoning prevention program evaluation strategies. Applicants must provide an evaluation and performance measurement plan that demonstrates how the funded program will fulfill the requirements of program implementation. At a minimum, the plan must detail performance measures, data collection methods, strategies for responding to evaluation questions, and for using evaluation findings for continuous program quality improvement. Going further, the plan should include plans for updating the data management plan as new pertinent information becomes available, if appropriate, throughout the lifecycle of the project. The following are childhood-led poisoning prevention program expected outcomes. Recipients must be able to use their childhood blood-led surveillance system to collect and report data to CDC to support the following four performance outcomes. Number one, improved blood-led testing and reporting rates for children less than six years of age at risk for lead exposure. Number two, improved use of surveillance system data to capture missing data on child demographic and follow-up information. Number three, improved rates of children less than six years of age who meet state case definitions who are linked to recommended services. And four, decreased disparities in blood-led levels by race, ethnicity, and socioeconomic status. In the next few slides, I will share best practices for conducting a program evaluation. What is an evaluation? CDC defines it as the systematic collection of information about the activities, characteristics, and program outcomes to make judgments, improve program effectiveness, and inform decisions about future program development. A picture of a clock. You might be wondering, why should I conduct an evaluation? An evaluation is important because it can be used to monitor performance, demonstrate program impact, improve program design and implementation, and provide an assessment of our work. CDC's approach to evaluation is a systematic way to improve and account for public health actions by involving procedures that have utility. The evaluation helps us understand whether the actions being taken are useful, feasible, or realistic given the time, resources, and expertise. In addition, are the actions a priority, or do the activities protect the welfare of those involved, and what is the accuracy? Or will the evaluation produce findings that are valid and reliable given the needs of those who will use the results? A picture of a puzzle. Health equity is achieved when every person has the opportunity to attain his or her full health potential and no one is disadvantaged from achieving this potential because of social position or other socially determined circumstances. Therefore, consider incorporating health equity in the framework steps for program evaluation, which we will discuss in the next slides. A snapshot of a fact sheet on health equity. This picture summarizes the essential elements of program evaluation presented in the CDC Framework for Program Evaluation and Public Health. This framework is intended as a practical, non-prescriptive tool. The six steps of program evaluation are shown in a circle, representing the circular nature of the evaluation process. As depicted, the steps are interdependent, and they may be conducted in a non-linear order. Often, more than one step is going on at any time. The framework provides the flexibility to go backward or forward in the process, depending upon the need. Although the steps can be conducted in a different order or simultaneously, there's a reason for the way the steps have been sequenced. The earlier steps provide the foundation for the latter steps. It's important to note that each one of these steps plays an important role and is crucial to the process. The circle also represents that the process for program evaluation is iterative. As program evaluation is conducted, the information or the finding from the evaluation is used to feed back into the program to make improvements. The process repeats itself in a cycle as a mean to improve the program and its effectiveness in achieving objectives. At the center or core of the circle are the evaluation standards. These four standards are intended to maximize the quality and the effectiveness of evaluation efforts. They are the criteria and standards for which the quality of your program evaluation efforts can be judged. They're applicable to each of the six steps in the evaluation framework. In the next series of slides, we will discuss each of the six steps. A picture of the evaluation cycle. Step one. The evaluation cycle begins by engaging stakeholders. There are different kinds of stakeholders involved. At the outermost level, stakeholders are those served or affected by the program. Those include children, families, schools, health care providers, and communities. In the middle of the circle are those involved in program operations. Those include sponsors, collaborators, funders, staff, etc. At the innermost level, the primary users of the evaluation are stakeholders such as program decision makers and policy makers. Concentric circles representing kinds of stakeholders. Step two. The program description begins with defining the program by identifying the need, which is a statement describing the type of problem, how big the problem is in the area of service, identifying the expected effects or the short, medium, and long-term outcomes. Listing activities, which are actions the program takes to affect change, which should be logical and sequenced. Identifying resources, including human, material, and collaborative resources. Staging, or describing the proposed planning and implementation. Context, which is where the program is situated in the community and fits in current practices. A logic model displays a sequence of events connected to demonstrate results. Step three. There are a number of considerations that should guide the design of your focused evaluation. Having a clear purpose will assist with gaining insight, changing practice, or assessing effects. Know the users. Consider the program, agency, or the funders who will use the evaluation findings. Understand the uses. Consider how the findings will be used to inform policy or program improvement. Develop appropriate questions. Evaluation questions will establish boundaries for the evaluation design. Select methods appropriate for your design, which can include experimental, quasi-experimental, and observational, and ensure regulatory requirements are met or obtain agreements with collaborative partners, including institutional review boards or memoranda of understanding to solidify responsibilities of others involved. One of the critical components of step three, focusing the evaluation, is the development of evaluation questions. Here are some example questions for a lead poisoning prevention program. What has been the impact of the childhood lead poisoning prevention programs on lead testing rates among children aged zero to three years in my jurisdiction? Has the childhood lead poisoning prevention program influenced change in pediatricians' and other physicians' lead testing behaviors? Has the childhood lead poisoning prevention program effectively reached key influencers of child welfare in my jurisdiction to provide vital information on the risk of lead exposure? What are barriers to lead testing in my jurisdiction? These are just a few sample questions and not prescriptive for your program. Step four. The next step is gathering credible evidence using indicators. Indicators define the program attributes that pertain to the evaluation's focus and questions. Some examples of indicators are lead testing rates, numbers of doctors incorporating lead testing practices, and the number of child welfare organizations or agencies that are partnering with your program. Sources are where you get your information, including persons, documents, or observations. Additionally, information should be quality or reliable, valid and useful for the evaluation, and quantity refers to the amount of evidence gathered in an evaluation. Step five. In order to justify your conclusions, you will need analysis and synthesis. This includes analysis of an evaluation's findings and combining information. Interpretation. This is the meaning of the findings of your program. Recommendations. What can be done to improve the program? What are the lessons learned? Evaluation is important if we're able to implement findings and share the lessons learned in the following three ways. Feedback. Input from primary users and stakeholders needs to be solicited. Follow-up on the evaluation's findings and provide lessons learned to policymakers and stakeholders. Dissemination. Communicating what has been learned to relevant audiences. In this next section, we will go over key definitions necessary for program evaluation. There are primarily three major types of program evaluations. Formative evaluation ensures that a program or program activity is feasible, appropriate, and acceptable before it's fully implemented. It's usually conducted when a new program or activity is being developed or when an existing one is being adapted or modified. Outcome evaluation measures program effects in the target population by assessing the progress and the outcomes. To develop outcome evaluation questions, examine your work plan activities and how they relate to outcomes defined in the project guidance. Process evaluation determines whether program activities have been implemented as intended and resulted in certain outputs. You may conduct process evaluation periodically throughout your program and start by reviewing the activities and output components. Process evaluations provide immediate feedback on program performance. Process-based evaluations include, one, was this implemented and to what extent? Two, was this implemented by the deadline? And three, what problems were encountered during implementation? Icon of a clipboard. Measuring performance is the ongoing monitoring and reporting of program accomplishments and progress toward pre-established goals. A performance measure assesses a program's progress toward its stated goals. Icon of a ruler. A performance measure is direct and accurate, such as room temperature in degrees Celsius. A performance indicator is approximate and requires interpretation and explanation, even if it's measured as accurately such as percent of at-risk children getting a lead test. Icon of a collection basket. It's important that your evaluation goals are SMART. SMART stands for Specific, Measurable, Achievable, Realistic, and Time-bound. Icon of a person thinking. Let's review types of data. Qualitative data provide information on the context of an intervention to understand and answer questions such as why and how an intervention did or did not occur. For example, why are children not receiving a blood lead test? Qualitative data analysis may include identifying themes from focus groups or key informant interviews. Quantitative data focuses on information that can be counted and can measure the uptake of an intervention, such as how many children have received the blood lead test. This data can be collected by structured surveys, reviewing medical documents, databases, or other numeric data. Icon of a data graph. A mixed methods approach combines the use of qualitative and quantitative data for performance indicators. It provides a more complete answer to public health questions. Mixed method evaluations blend various designs and data collection strategies. In these evaluations, the investigator collects, analyzes, and integrates the findings and draws inferences using both qualitative and quantitative approaches in a single study or program of inquiry. Some examples include asking how many children have received a lead test, a quantitative question, and why are children not receiving a lead test, a qualitative question. Icon of arrows crossing. Primary data is data collected by the program directly to measure specific program variables. Secondary data is collected in the past by another entity. If there will be a need for secondary data within the evaluation, a plan should be in place to obtain it. Icon of an open file folder. In order to collect data, the following steps should be taken. Identify stakeholders' data needs. Define the performance measurement context. Identify useful measures for monitoring progress and informing decision making. Set targets. Collect measurement data. Analyze measurement data and compare results to targets. Then act and repeat. In this section, we will get into performance measures and explain how to use a logic model to measure program performance. A basic logic model is a graphic depiction that illustrates how programmatic resources and activities will lead to desired outcomes. It is a representation that says we plan to do X, which will give Y results. The steps in a logic model include inputs, activities, outputs, and outcomes. Boxes in a sequence showing the steps for development of a logic model. Let's review the parts of a logic model for the program goal of improving lead testing rates of children for lead exposure. This table depicts the four components of a logic model and provides an example. Inputs are the resources, contributions, and investments that go into a program. For example, funding, staff, materials, time, and supplies. Activities are projects or interventions carried out by the program. An example is developing and sustaining a statewide lead advisory committee. Outputs are a quantitative estimate of products, services, or interventions produced from the program activities. For example, education sessions for healthcare providers, community parent education events, and laboratories reached for quality lead testing. Outcomes are the changes expected from the result of the program and can range from short-term to long-term. For example, increase in the proportion of children tested and linked to care, changes in knowledge, skills, or beliefs. Now let's review the steps in logic model development. The first step in logic model development is identifying the goal. The goal being addressed by the program needs to be well-defined so all staff and stakeholders have the same definition of the problem. The logic model should include a description of the goal and who it affects the most. For example, the goal for a lead program could be to improve lead testing rates of children at ages 0 to 3 years at risk for lead exposure, but not tested. The second step is determining the key program inputs or activities. To determine inputs, decide the resources needed for a successful lead testing program, carefully considering human and other resources. To determine activities, decide how inputs can be used to develop or support the activities or interventions. Keep in mind the identified target audiences and that the program should take actions based upon their characteristics and seek to engage these audiences. The chart presents some examples of inputs and activities. Inputs include funding, staff, materials, and partners. Local data. Identify and use local data to inform development of or update the statewide lead screening plan or to target development of outreach programs. The third step is determining the key program outputs. Outputs are products generated from program activities or interventions. Outputs can be viewed as quantifying activities and providing numeric values or attributing percentages. The fourth step is identifying program outcomes. Determine the ultimate program goals or outcomes by asking what the program is going to change or achieve. To track the progress of outcomes, there should be a starting point or baseline for the areas that need to be improved. Examples include increase in children tested for lead by X percent and increase in physicians testing children for lead by X percent. Another vital step is identifying program indicators for measuring outcomes. Process indicators track how the implementation of the program is progressing. Are the activities being implemented as planned? Realistic and achievable numbers for each activity will become the process indicators. Outcome indicators track how successful program activities have been at achieving program goals. Set a baseline and a target outcome. Have program activities made a difference? Program impact is sometimes influenced by a number of external factors. These factors include political environment. Is your program supported by local officials? Economic. Are resources sufficient to carry out program goals? Social-cultural context. Is the community supportive of your program? Ecological constraints, a change in systems and supports. Technological constraints could include equipment issues. Other constraints are transportation availability and weather. An example of an external factor that impacted program implementation is the blood lead testing interruption experienced during public health emergencies. This graph depicts the number of children aged less than six years who received a blood lead level test by month in 34 jurisdictions from 2019 to 2020. As you can see, the number of children receiving a blood lead level test in 2020 significantly declined from 2019 in the beginning of the year. A picture of a graph showing the decline in testing for lead during the pandemic. Another example of an external factor is the blood lead test recall. In 2021, lead care blood lead tests were recalled due to risk of falsely low results. These factors were external factors that could halt or interrupt program efforts. A snapshot of a HealthAlert network post about the blood lead test kit recall. Here we present a sample logic model diagram and the program goal, improving lead testing rates. As mentioned, we would start with the goal for our program and then add inputs, activities, outputs, and the end result is our outcomes. Having a graphical or visual display of the logic model will help you lay out everything and plan. Inputs for this example would include funding, staff, materials, partners, and local data. The activities include develop and sustain a statewide lead advisory committee and develop and update statewide lead screening plans based on the analysis of local data. Outputs are the number of targeted education sessions for pediatricians or other doctors on risk assessment and lead testing, the number of targeted communities in parent lead testing education events held, and the number of meetings with laboratories for quality assurance of lead testing. Set outcome targets at specific but realistic times. For example, expected outcomes at the one-year mark include increase in children tested for lead by X percent, increase in physicians testing children for lead by X percent. Expected outcomes at the five-year mark include increase in children tested for lead by X percent, and increase in physicians testing children for lead by X percent. These are a few suggestions and should not be considered an exhaustive logic model for the goal. Some key things to remember about logic models are that logic models are not static and they are expected to change over time due to new or revised activities or completed benchmarks. Logic models lend credibility to what we do and why by providing transparency to programs, graphically displaying planned activities, and assisting policymakers to understand the relationships of program activities to anticipated outcomes. In summary, this presentation discussed key aspects to consider when developing an evaluation plan. Evaluation is critical for planning, measuring, and identifying lead poisoning prevention program successes and areas for improvement. CDC's evaluation framework can be used to guide your work as you develop different types of evaluation, whether it's outcome, process, or formative-based. A health equity and cultural humility lens should be applied to all aspects of your evaluation framework steps. Performance measures can be used to define a program's progress toward its stated goals, while your logic model depicts the relationship between your program's activities and its intended effects. This presentation was developed in collaboration with the American Academy of Pediatrics and Commander Candace Hunter, Ph.D., formerly a senior public health advisor with Centers for Disease Control and Prevention, National Center for Environmental Health in the Lead Poisoning Prevention and Surveillance Branch, proposed. The contents of this presentation are those of the authors and do not necessarily represent the official views of, nor an endorsement by, the Centers for Disease Control and Prevention, the American Academy of Pediatrics, the U.S. Department of Health and Human Services, or the U.S. government. Thank you for viewing Module 2, Segment 5. For questions or comments regarding this segment, please send an email to LeadInfo at CDC.gov and indicate Training Question Module 2, Segment 5 in the subject line.