Incorporating SMARTIE Outcomes and KPIs Into Your Performance Measurement Plans

In the 1980s and 1990s, many public schools across the country implemented The Drug Abuse Resistance Education (DARE) program to warn fifth graders about the dangers of drugs and alcohol. The DARE program was deemed successful because of the number of students who completed the program (output); the outcome, which measures the quality or effectiveness of a program or service, was not considered.

Unfortunately, DARE backfired. The “cool” lessons made students more curious about drugs than they might have been, and in turn, studies showed students involved with DARE were more likely to use drugs and alcohol than their non-participating peers.

DARE demonstrates how outcomes are more meaningful than simply relying on outputs in your performance measurement projects. Outcomes can be more effective if written using the SMARTIE (Specific, Measurable, Achievable, Relevant, Time-Bound, Inclusive, and Equitable) format, discussed further below.

This blog covers the basics of performance measurement so you can make sure you’re considering quality, not just quantity, in your programs and services.

What Is Performance Measurement?

Performance measurement is the process of collecting, analyzing, and reporting data about the effectiveness of a program or service. Local governments use performance measurement to evaluate the efficiency, equity, and quality of their operations.

Performance measurement may help local governments with:

• Discovering unidentified patterns and trends.
• Incorporating real-time tracking to improve responsiveness.
• Sharing information with the public and officials.
• Identifying priorities.
• Justifying budget requests.
• Motivating departments with appreciation of their performance.

Stages of Performance Measurement

© King’s Printer for Ontario, 2021. Updated November 2023. Reproduced with permission.

The Government of Ontario, Canada suggests performance measurement can be broken up into four stages, which are covered below.

Stage 1: Preparing for Performance Measurement Plans

This stage involves building a team for the project, identifying stakeholders, and considering baseline factors about the program or service to be evaluated. 

Most performance measurement teams start by collecting information about inputs, defined as resources used for departments, agencies, programs, or services. These inputs might include:

• Program costs,
• Number of staff members, or
• Labor hours.

Outputs, in turn, are the results of a department, program, or service. Local governments typically want to make sure they understand how many inputs are required for each output. Examples of outputs may include:

• Clients served,
• Calls answered, or
• Fires responded to.

Next, the performance measurement team should identify the goal or goals of whatever they are measuring. This is the purpose of the measurement project: to determine if the program or service is meeting its objective.

Stage 2: Identifying Outcomes

Identifying meaningful outcomes is the most important component of any performance measurement effort. 

Examples of outcomes include:

• A decrease in unemployment rates,
• Percent of pavement in good condition, or
• Percent of late bills collected.

Measuring outputs alone, without considering outcomes, is less effective in evaluating program quality. For instance, if a city department only measured the number of people hired to mow public lawns (output), they would not be aware of the percentage of lawns mowed well (outcome). 

Stage 3: Creating Key Performance Measures

The next step in the performance measurement process is identifying Key Performance Indicators (KPIs) that allow for assessment of whether an outcome has been met. Whereas an outcome is the goal of a project or service, a KPI is the metric that tracks the achievement or underachievement of that goal. Tracking outcome achievement may require one or more KPIs.

For instance, if an outcome was to increase employment by a specific percentage, the team would need to identify KPIs contributing to that increase (see table below). In this case, KPIs like continuing education completion and number of clients trained for employment served as this outcome's measures.

Specific KPIs include efficiency and service quality indicators.

Efficiency indicators — The ratio of input to output; for instance:

• Cost per counter transaction
• Hours to remedy per complaint
• Cost per mile of road paved

Service quality indicators — Service quality measures can include customer satisfaction, accuracy, or timeliness; for example:

• Errors per data entry operator
• Percentage of satisfied customers
• Average wait time

Benchmarking

How do local governments know which increases they should aim for? Through benchmarking.

A benchmark is used as a point of reference to determine how effective a project or service is compared to the same project or service in another department or government. This is called external benchmarking.

As performance measurements are repeatedly collected, agencies can then compare current benchmarks to benchmarks from previous points in time. These are called internal benchmarks. Internal benchmarks help departments determine if they are maintaining or improving the achievement of their outcomes.

After reviewing internal and external benchmarks, teams should then identify the outcomes they want to achieve.

© King’s Printer for Ontario, 2021. Updated November 2023. Reproduced with permission.

SMART and SMARTIE Goals

Vague goals hinder the success of performance measurement projects. The SMARTIE model helps performance measurement teams develop more precise outcomes. You may be familiar with SMART goals, like these:

• SMART: Reduce invoices not paid within 30 days to less than 150 per month, by the end of the fiscal year.
• Not SMART: Reduce the number of unpaid invoices.
• SMART: Build relationship with public library to increase shared resources, with a goal of three shared resources by the end of the fiscal year.
• Not SMART: Build relationship with public library.

However, adding the “I” (inclusive) and “E” (equitable) dimensions ensure that a SMART goal does not create an “unintentional disparate impact.”

Here is an example of a SMARTIE goal:

Increase percentage of staff with marginalized identities by end of the fiscal year while incorporating regular checks so these staff members are not burdened with a greater share of DEI work.

Stage 4: Collecting, Analyzing, and Communicating Results

Collecting Data

The only way local governments can know if they are meeting outcomes is if they collect KPI data. Some collection methods include:

• Professional or practitioner expertise or rankings (like road crews offering cleanliness scores to streets and roads);
• Pre- and post-assessment checklists;
• Customer surveys;
• Exit interviews;
• Social media tracking; and
• Mobile applications and sensors.

Data can be tracked using different tools, from the complex to the simple, including:

• Formal data collection systems,
• Case management systems, and
• Spreadsheets.

Analyzed data can be shared annually or semi-annually through static performance reports. Often, however, analyzed data is continuously collected and shared on dashboards or performance scorecards.

Take the City of San Francisco’s Performance Scorecards dashboard as an example. Viewers can scan to see red (not meeting target), yellow (needs improvement), and green (on target) metrics for many outcomes and KPIs.

Sharing Results and Making Improvements

A performance measurement project is not complete until the results are shared with stakeholders, even if outcomes are not met. If this happens, teams or departments should identify what could be modified to improve. For instance, if a public safety department set a goal of achieving 1,000 spot checks at businesses and apartments each year over three years, but only achieved 900, they may decide to require a certain number of spot checks per employee per day.

Alternatively, the team may need to reconvene to decide if the outcomes or KPIs they set are unrealistic.

A rule of thumb is to conceive of performance measurement initiatives as continuous improvement projects, where departments are always collecting data and using it to maintain or improve performance, rather than one-off tasks. This way, quality is always being assessed.