Many corporate executives are rightly frustrated about paying immense and growing costs for compliance programs without seeing clear benefits. At many companies, strengthening compliance has become synonymous with hiring more compliance managers, buying more-sophisticated software, and creating more policies, even when those moves are redundant and wasteful or just don’t deliver results. Employees also often resent compliance programs, seeing them as a series of box-checking routines and mindless training exercises. All this expense and frustration is tragic—and avoidable.
The answer lies in better measurement. At its core, the idea is as simple as it is crucial: Firms cannot design effective compliance programs without effective measurement tools. One of the main reasons that companies keep investing more and more in compliance is that they do not have the right measures and thus cannot tell what works and what doesn’t. Put simply, better compliance measurement leads to better compliance management.
How We Got To This Point
The development of corporate compliance programs followed a stream of corporate scandals in the 1970s and 1980s, when industry groups adopted policies and procedures to attempt to prevent and detect misconduct. In 1991, the United States Sentencing Commission amended its “sentencing for organizations” guidelines to offer firms reduced fines if they could show that they had an “effective compliance program.” The United States Department of Justice (“DOJ”) adopted policies urging prosecutors to consider the effectiveness of a firm’s compliance program when deciding on criminal charges. Other civil regulators also adopted similar policies. An industry quickly sprouted to provide compliance-related products and services such as training, whistle-blower hotlines, and risk assessment.
The DOJ recognized that it was often challenging to distinguish substantive programs from those that were merely window dressing, since evaluating a program required considerable time and expertise. In 2015, the DOJ retained Hui Chen to address the challenges of evaluating the actual effectiveness of firms’ compliance efforts. Chen drafted an extensive list of questions for prosecutors to consider when assessing compliance programs. The questions covered a wide range of compliance areas, including training, individual accountability, and leadership. The DOJ publicly released the questions in February 2017 in a document titled “Evaluation of Corporate Compliance Programs.”
The document was not intended to be used as a checklist; indeed, all evaluations would continue to be individualized. Nonetheless, firms quickly began to appropriate the document as a manual on constructing an effective program. Even more worrisome, firms were selectively picking data to support the notion that their practices were effective, rather than recognizing that some were clearly falling short.
How Compliance Metrics Go Astray
In seeking to assess program effectiveness quantitatively, firms tend to make the same mistakes. Here are the common pitfalls:
For example, the DOJ evaluation document asks: “Has the company ever terminated or otherwise disciplined anyone for the type of misconduct at issue?” To demonstrate individual accountability, firms often list the employees who have been terminated or denied promotions and bonuses as a result of compliance-related transgressions. Yet such statistics aren’t enough to substantiate that a firm rigorously holds employees accountable since they don’t indicate the number of employees who were not disciplined. So the simple statistic on the number of sanctioned employees can be incomplete and misleading.
Although a wide range of data may be collected on the various facets of a compliance program, only a subset of that data actually correlates with the impact of a program. For example, in response to the DOJ question asking how the company has measured the effectiveness of its training, firms often focus on the percentage of employees who’ve completed the training or the number of hours they’ve spent doing so. Those are entirely the wrong metrics to use. A meaningful measure of effectiveness must be directly tied to a clearly articulated outcome—for example, employees’ demonstrated an understanding of policies and procedures, their acquisition of useful skills for confronting anticipated scenarios, or a change in their behavior.
Mistaking legal accountability for compliance effectiveness.
Take this question: “How has the company assessed whether these policies and procedures have been effectively implemented?” Firms often respond by showing that employees signed a statement that they had read and understood the company’s policies and codes of conduct. While such a signature may provide legal grounds to fire someone who violates a rule, it does not demonstrate that an employee has converted knowledge about policies into everyday work practices. Employees may sign an acknowledgment of corporate policies without actually having read or understood the terms. Thus, counting employees’ legally binding assents to policies is not an appropriate way to quantify the effectiveness of a compliance initiative.
Self-reporting and self-selection bias.
Compliance managers often rely on surveys to assess the performance of their programs. For instance, to gauge employee comfort with reporting mechanisms, a firm might ask: “Do you know when to seek compliance advice? Are you willing to do so?” The challenge with surveys is that self-reporting and self-selection by the respondents may bias the results and lead managers to draw incorrect conclusions. Thus, bias in the data collected needs to be accounted for when interpreting the metrics.
Linking Compliance Initiatives to Objectives
So how do you create models that can credibly evaluate the impact of a compliance program? The first step is recognizing that such programs actually have multiple purposes. As discussed in memoranda by senior DOJ officials, the three main goals are to prevent misconduct, to detect misconduct, and to align corporate policies with laws, rules, and regulations. Each component of a compliance program should be linked to one of these objectives to create more meaningful metrics.
Consider a confidential hotline for whistleblowers. Its objective is to improve the timely detection of wrongdoing. To understand whether it’s achieving this goal, certain information is needed, including whether the hotline works (“mystery tester” reports), whether people actually use it (usage data), how they use it (data on types of calls received), the firm’s responsiveness to allegations (response time, investigation completion time, investigation results, communication of results), and whether employees feel comfortable contacting the hotline (periodic surveys of employees’ sentiments). Each of those metrics captures a different dimension of the initiative’s efficacy.
However, tracking those variables independently is insufficient, because it doesn’t allow managers to identify which ones are responsible for particular outcomes. For instance, a “hot” hotline might reflect a rising number of problems or just a high level of employee comfort with calling. To get clarification, managers can apply multivariate regression analysis. Regression models allow an investigator to examine the impact of one variable while holding the others constant. In this case, to ascertain whether an increase in calls indicates an increase in compliance breaches, we would seek to hold the following other factors constant: the availability of the hotline, people’s comfort in using it, its operational performance, and the number of potential callers (people who have access to it).
Designing appropriate regression models takes time and experience, but it is the most reliable way to know whether to be reassured by or concerned about shifts in call volume. As this example demonstrates, firms should use empirical data generated from their compliance programs to gauge how well a program is meeting its objectives. Again, we stress that firms need to do more than simply track metrics independently. They must focus on creating models that measure the desired output while controlling or excluding other factors.
As compliance programs continue to be more closely scrutinized, those that cannot show meaningful results will fail to meet the stronger regulatory standards being applied today. It would be convenient if there were a one-size-fits-all yardstick that could show whether or not a compliance program is on track. But simple univariate metrics will not adequately capture a program’s effectiveness. For compliance programs to have a real impact, managers need to test what works and what doesn’t. This will require firms to engage in some experimentation and innovation. By developing better measures of effectiveness, firms can adopt more ambitious and innovative programs that really do curb the improper behavior.
Companies worldwide are already spending a fortune on compliance. Let’s make sure that all those resources are being spent well. Better measurement can help managers identify redundant or ineffective initiatives that can be replaced or eliminated—and ultimately reveal opportunities to make programs more effective.
This post is a modified and abbreviated version of an original article by the same title published in the March-April 2018 issue of Harvard Business Review. The complete article is available for download here.