Technical Assistance Guidelines Evaluation

This section presents key considerations and options available to State agencies in evaluating TA programs. It includes guidance on determining evaluation goals and process as well as budgeting and contracting considerations to consider when hiring an outside evaluator. It does not provide a detailed guide to evaluating a TA program but is instead intended to offer a high-level overview of evaluation best practices for State agency staff. The Resources page lists additional references that provide more detailed guidance on some of the topics covered here.

Why is Evaluation Important?

Program evaluation is the “systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development.” [ 1 ] Evaluation is a critical component of designing effective TA, but it is too often forgotten or only considered after TA delivery. Evaluating the quality, suitability, and long-term impact of TA is fundamental to understanding opportunities for improvement and communicating successes that can make the case for future investments in TA. For TA to be as effective as possible, you should create an evaluation plan in the early stages of TA design. Engaging as many partners as possible during these early stages helps ensure that goals, metrics, and indicators for success are clear from the beginning, mutually agreed upon, and trackable from the outset of the TA effort.

It is critical to have clarity around goals and closely monitor progress in the case of TA for pilot programs or in the first round of TA provision, when demonstrating success is especially important to ensure future support for the program. Close monitoring and data collection throughout TA activities can shine a light on inefficiencies or issues that arise during the project term, making it possible to respond quickly. Be sure to track modifications throughout the project term in order to identify which project partners were successful in implementing with fidelity to the original plan, as well as what had to be adapted and how those changes helped meet the original program goals.

Key Considerations for Evaluation

Evaluation Goals

The first step for determining your evaluation goals is to clarify the objectives of the TA effort itself (see Step 1: Analyzing Needs and Gaps for more guidance). Clearly stating goals for both the process and outcome of your TA effort will help determine appropriate metrics and methods to measure them. You might develop a logic model to help identify the relationships between resources, activities, and results. Logic models can be organized as a series of if-then statements; as a diagram showing factors, activities, outputs, outcomes, and impacts; [ 2 ] or in various other ways. Engaging outside stakeholders in determining the desired outcomes of the program – particularly the communities that stand to benefit from the TA program – helps ensure that the TA effectively serves the needs of its intended audience.

Strong evaluation plans serve specific goals and audiences. For example, an evaluation might:

  • Help program staff better understand what is working well and identify areas for improvement or build a case to outside partners for additional funding (internal program staff)
  • Gather data to fulfill reporting requirements (funder)
  • Show the value of TA to encourage leadership to devote more staff time or resources to TA (internal leadership)
  • Inform the public of your agency’s TA and capacity building offerings to build transparency and improve trust in government (general public)

Tailoring evaluation deliverables to your target audiences to ensure they have the desired impact.

Budgeting

Gaining clarity on the goals, scope, and scale of the evaluation enables you to determine an appropriate budget for your evaluation. In many cases it will be important to prioritize goals with your budget amount in mind to ensure that the methodologies necessary to respond to the scope of the evaluation are feasible. Sometimes budget amounts for evaluation are predetermined, such as by legislation. In those cases, you will need to carefully consider what is achievable given budget constraints and may want to further prioritize any evaluation goals written in statute. If not, ensure that funding is available for evaluation and specify evaluation goals.

The budget for evaluation will depend on the method you plan to use to procure evaluation services. Partnering with a UC or CSU through an I/A will likely be more cost effective than releasing an RFP for a private evaluation consultant, but the latter may be preferable in some cases. Reach out to colleagues who have carried out similar evaluations to better understand an appropriate budget amount. Reviewing the budget for other similar evaluation studies can also help you set realistic expectations about the evaluation’s scope given financial constraints.

Hiring a third-party evaluator is ideal because it provides a fresh and unbiased perspective. However, it is also possible to incorporate evaluation deliverables into the scope of a TA contract or to carry out some informal in-house monitoring and evaluation. Surveys created by TA providers or State agency staff can help quickly evaluate TA recipients’ satisfaction. You may also consider reaching out directly to TA recipients to set up a quick call or email check-in to gather feedback. However, for more a robust and comprehensive evaluation of both process and outcome measures related to TA, it is best to engage a third-party.

Evaluation Plan

Developing an evaluation plan can help focus activities and ensure that everyone involved is clear on purpose of the evaluation, partners, evaluation design, evaluation questions, metrics, methods, outreach and engagement strategy, and timeline. Include this plan as an early deliverable of your evaluation contract, specifying who will be involved in developing and approving the final version of the plan. Once the plan is complete, it should remain a living document to amend as conditions change. Building regular updates to the evaluation plan into the scope of work can keep the plan relevant.

Stakeholder Participation and Engagement

There are many ways to meaningfully engage TA recipients, TA providers, and other relevant stakeholders in the evaluation process. The most appropriate way depends on a variety of factors including the nature of the TA, the goals of the evaluation, and the time and resources available. Involving stakeholders in the process of developing an evaluation plan as well as data collection and analysis can help to ensure that the evaluation results respond to the right questions and reflect the lived experience on the ground. While it may not always be feasible to engage stakeholders in all steps of the process, maximizing community engagement in the evaluation process can produce more accurate results.

Consider creating a community steering committee to advise the TA process as a whole or just the evaluation. If your agency chooses this option, be clear and intentional about the purpose of the group to ensure that committee members are meaningfully engaged and feel that their time is valued. Developing a participatory model for your TA evaluation by engaging the beneficiaries of the TA in carrying out evaluation activities themselves is another option to consider. There is a significant body of research and guidance available on best practices for participatory evaluation, some examples of which can be found on the Resources page.

Metrics and Data Collection

Engaging stakeholders, and especially TA recipients and community members, in the process of defining metrics is critical to ensure that the evaluation captures a holistic understanding of the success of TA, not just the benchmarks that are important to your agency. In selecting metrics to assess program impact, be sure to consider both process and outcome measures to create a holistic picture of the TA’s effectiveness. Process indicators measure the program’s activities and outputs, clarifying the extent to which the program is being implemented as planned. Outcome measures help you measure whether the program is meeting its intended goals in the short, intermediate, and long term. [ 3 ] Both are important in evaluating whether dollars were well spent and how to improve the program in future iterations. See the table below for examples of these two types of indicators.

Process and Outcome Indicators

Indicator Type Examples

Process Indicators – measure the program’s activities and outputs.

  • How many TA recipients from different types of communities were served?
  • Did TA reach desired geographic areas?
  • How did TA recipients rate the quality of the services provided?
  • How responsive was TA to community-identified needs?

Outcome Indicators – measure whether the program is achieving the expected effects/changes in the short, intermediate, and long term.

  • Did the community receive funding after receiving TA?
  • Was the project/policy measurably improved through the provision of TA?
  • Did the TA build community capacity to continue similar efforts in the future?
  • Did the TA build long-lasting partnerships?
  • Did the TA increase community understanding of grant program requirements?

In determining the methodology for tracking indicators, it is generally best to incorporate both quantitative and qualitative methods. Quantitative metrics provide information that can be counted and can be useful to show objective data, such as the number of TA recipients served or how much funding was awarded for TA-supported projects. It can also answer more subjective questions (through survey responses for example), like whether the TA improved trust in government, or built stronger partnerships.

Qualitative metrics help to provide a greater understanding of why or how the TA has been effective or ineffective through narratives and stories. This qualitative data can also help illustrate to non-technical audiences what constitutes effective TA. Qualitative methods can also support analysis and understanding of quantitative data, providing an opportunity to “ground-truth” or verify the data by cross-checking it with experiences on the ground. [ 4 ] See the table below for examples of the types of questions that can be answered by each metric type.

Quantitative and Qualitative Metrics

Metric Type Methods Sample Evaluation Questions
Quantitative Metrics
  • Surveys
  • Baseline assessments
  • Observation
  • Analysis of existing documents and databases
  • How many entities received TA?
  • How did TA recipients rate the quality of TA?
  • How many TA recipients were successful in receiving grants?
Qualitative Metrics
  • Interviews
  • Focus groups
  • Case studies
  • Observation
  • Analysis of written documents
  • What “value add” did the TA provide?
  • What best practices emerged from the TA effort?
  • How can TA be improved in the future?

If you intend to evaluate the program through a formal research process, you will need to consider data ownership, privacy, and Institutional Review Board (IRB) oversight. If you are working with Tribal communities, there may be additional considerations to data collection. See Appendix A for relevant guidance.

Communicating Results

Keep communications goals in mind when setting reporting requirements for your TA providers and designing the scope of work and deliverables for evaluators. If the primary audience for the evaluation is internal to your agency, you may not need to budget much time for creating visually pleasing and digestible deliverables. However, sharing evaluation results presents a great opportunity to communicate the impact of your program with the public; and in most cases it is worthwhile to ensure that the contract includes some deliverables that you can share publicly, for example:

  • Compelling data points that demonstrate the success of the project
  • Visuals that help tell the story of your TA efforts
  • Profiles of TA recipients – organizations or individuals
  • Case studies

Such materials can help make the case for future funding for your program, improve the image of your agency or program, build trust in State government, or simply help inform the public that TA services are available and effective.

Keep in mind that evaluation is not only about measuring success, it is also a way to illuminate opportunities for improvement. No one expects for TA to be perfect, especially in the case of a pilot or a new program, but being open about the ways your agency intends to be responsive to feedback helps community members see that your agency is invested in providing the most effective TA possible.


  1. Patton MQ. Utilization-focused evaluation: The new century text. 3rd ed. Thousand Oaks, CA: Sage, 1997.  
  2. W.K. Kellogg Foundation, Logic Model Development Guide, 2004.  
  3. Centers for Disease Control and Prevention. Approach to Evaluation.  
  4. Agency for Toxic Substances and Disease Registry. Principles of Community Engagement, Second Edition.