1. Home
2. Tools & Resources
3. Toolkit
4. Evaluation

Technical Assistance ToolkitEvaluation

This section presents key considerations for evaluating technical assistance (TA) programs. It includes guidance to determine evaluation goals and processes as well as budgeting and contracting considerations for hiring an outside evaluator. It does not provide a detailed guide to evaluating a TA program, but is instead intended to offer a high-level overview of best practices for evaluating program staff. The Resources section lists additional resources that provide more detailed guidance on some of the topics covered here.

Why is Evaluation Important?

Program evaluation is the “systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development,” (see definitions) Evaluation is a critical component of designing effective TA, but it is too often forgotten or only considered after TA delivery. Evaluating the quality, suitability, and long-term impact of TA is fundamental to understanding opportunities for improvement and communicating successes that can make the case for future investments in TA. For TA to be as effective as possible, your agency should create an evaluation plan in the early stages of TA design. Engaging as many partners as possible during these early stages helps ensure that goals, metrics, and indicators for success are clear from the beginning, mutually agreed upon, and trackable from the outset of the TA effort.

It is critical to have clarity around goals and closely monitor progress of your TA program. This is especially important in the case of TA for pilot programs or in the first round of TA provision when demonstrating success is especially important to ensure future support for the program. Close monitoring and data collection throughout TA activities can shine a light on issues, inefficiencies, or opportunities for improvement that arise during the project term, making it possible to respond quickly. Tracking modifications throughout the project term can help inform the development of the next TA effort and can help tell the story of how the program was able to respond to evolving needs and lessons learned.

Hiring a third-party evaluator is ideal because it provides a fresh and unbiased perspective. However, when that is infeasible, it is also possible to incorporate evaluation deliverables into the scope of a TA contract. Surveys created by TA providers or program staff can help quickly evaluate TA recipients’ satisfaction. You may also consider reaching out directly to TA recipients to set up a call or email check-in to gather feedback. However, for more a robust and comprehensive evaluation of both process and outcome measures related to TA, it is best to engage a third-party.

Key Considerations for Evaluation

Evaluation Goals

The first step for determining your evaluation goals is to clarify the objectives of the TA effort itself (see the Setting Goals and Intended Outcomes section for more guidance). Clearly stating goals for both the process and outcome of your TA effort will help determine appropriate metrics and methods to measure them. You might develop a logic model to help identify the relationships between resources, activities, and results. Logic models can be organized as a series of if-then statements; as a diagram showing factors, activities, outputs, outcomes and impacts;11 or in various other ways. Engaging outside entities – particularly the communities that stand to benefit from the TA program – in determining the desired outcomes of the program you can help ensure that the TA effectively serves the needs of its intended audience.

Strong evaluation plans serve specific goals and the deliverables should be tailored to your target audiences. For example, an effective evaluation process should:

• Help program staff better understand what is working well and identify areas for improvement or build a case to outside partners for additional funding (internal program staff)
• Show the value of TA to encourage leadership to devote more staff time or resources to TA (internal leadership)
• Inform the public of your agency’s TA and capacity building offerings to build transparency and improve trust in government (general public)

Budgeting

Gaining clarity on the goals, scope, and scale of the evaluation enables you to determine an appropriate budget for your evaluation. In many cases it will be important to prioritize goals with your budget amount in mind to ensure that the methodologies necessary to respond to the scope of the evaluation are feasible. Sometimes budget amounts for evaluation are predetermined, such as by legislation. In those cases, you will need to carefully consider what is achievable given budget constraints and may want to prioritize the most important evaluation goals If you do not have a set evaluation budget, reach out to colleagues and external partners who have carried out similar evaluations to better understand an appropriate budget amount. Reviewing the budget for other similar evaluation studies can also help you set realistic expectations about the evaluation’s scope given financial constraints.

Evaluation Plan

Developing an evaluation plan can help focus activities and ensure that everyone involved is clear on purpose of the evaluation, partners, evaluation design, evaluation questions, metrics, methods, outreach and engagement strategy, and timeline. Include this plan as an early deliverable of your evaluation contract, specifying who will be involved in developing and approving the final version of the plan. Once the plan is complete, it should remain a living document to amend as conditions change. Building updates to the evaluation plan into the scope of work can keep the plan relevant.

Public Participation and Engagement

There are many ways to meaningfully engage TA recipients, TA providers, and other relevant partners in the evaluation process. The most appropriate way depends on the nature of the TA, the goals of the evaluation, and the time and resources available. Involving community members, community-based organizations, and other relevant entities in the process of developing an evaluation plan as well as data collection and analysis can help to ensure that the evaluation results respond to the right questions and reflect the lived experience on the ground. Creating a community steering committee to advise the TA process as a whole or just the evaluation is a valuable way to incorporate community feedback into the TA program throughout its implementation. If your agency chooses this option, be clear and intentional about the purpose of the group to ensure that committee members are meaningfully engaged and feel that their time is valued. Developing a participatory model for your evaluation is the best choice if equity is truly integrated throughout the TA program from design to evaluation. There is a significant body of research and guidance available on best practices for participatory evaluation, some examples of which can be found in the Resources section.

Metrics and Data Collection

While the goal of capacity building programs may be to increase a community’s ability to successfully apply for funds or implement a project, the lasting impact of capacity building TA is difficult to quantify through traditional data metrics. You should expect to invest additional time and effort into creating strategies to measure the impact of capacity building programs in communities and making the case for ongoing support. In selecting metrics to assess program impact, consider both process and outcome measures to create a holistic picture of the TA’s effectiveness. Process indicators measure the extent to which activities occurred. These measures help answer questions like the number of TA recipients from different types of communities served, the quality of the services provided, and the extent to which services addressed the most pressing community-identified needs on the ground.

Outcome measures help you understand the impact of the TA program and are especially important in evaluating whether dollars were well spent and if activities should continue in the future. Outcome indicators focus on the immediate impact of the TA - How many TA recipients received grant funding? – as well as on longer-term outcomes of TA and capacity building efforts, such as relationships or skills built. Engaging TA recipients, community members, and other partners in the process of defining metrics is critical to ensure that the evaluation captures a holistic understanding of the success of TA, not just the benchmarks that are important to your agency.

In determining the methodology for tracking metrics, incorporate both quantitative and qualitative methods. Quantitative metrics can help provide information on the extent to which the TA met program goals, such as the number of TA recipients served or how much funding was awarded for TA-supported projects. It can also answer more subjective questions (through survey responses for example), like whether the TA improved trust in government, or built stronger partnerships.

Qualitative metrics help to provide a greater understanding of why or how the TA has been effective or ineffective through narratives and stories. This qualitative data can also help illustrate to non-technical audiences what constitutes effective TA. Qualitative methods can support analysis and understanding of quantitative data, providing an opportunity to “ground-truth” or verify the data by cross-checking it with experiences on the ground.

If you intend to evaluate the program through a formal research process, you will need to consider data ownership, privacy, and Institutional Review Board (IRB) oversight. If you are working with Tribal communities, there may be additional considerations to data collection.

Communicating Results

Keep communications goals in mind when designing the scope of work and deliverables for evaluators. Sharing evaluation results presents a great opportunity to communicate the impact of your program with the public. In most cases it is worthwhile to ensure that the contract includes some deliverables that you can share publicly, for example:

• Compelling data points that demonstrate the success of the project
• Visuals that help tell the story of your TA efforts
• Profiles of TA recipients – organizations or individuals
• Case studies

Such materials can help make the case for future funding for your program, improve the image of your agency or program, build trust in State government, or simply help inform the public that TA services are available and effective.

Evaluation is not only about measuring success, it is also a way to illuminate opportunities for improvement. No one expects for TA to be perfect, especially in the case of a pilot or a new program, but being open about how your agency intends to incorporate feedback helps community members see that the agency is invested in providing the most effective TA possible.