Hui Chen, Ethics and Compliance Advocate and former Compliance Counsel Expert for the U.S. Department of Justice, contributes today’s guest post with the assistance of Alaina Bird at Irell & Manella:
Many corporate executives are rightly frustrated about paying immense and growing costs for compliance programs without seeing clear benefits. At many companies, strengthening compliance has become synonymous with hiring more compliance managers, buying more-sophisticated software, and creating more policies, even when those moves are redundant and wasteful or just don’t deliver results. Employees also often resent compliance programs, seeing them as a series of box-checking routines and mindless training exercises. All this expense and frustration is tragic—and avoidable.
The answer lies in better measurement. At its core, the idea is as simple as it is crucial: Firms cannot design effective compliance programs without effective measurement tools. One of the main reasons that companies keep investing more and more in compliance is that they do not have the right measures and thus cannot tell what works and what doesn’t. Put simply, better compliance measurement leads to better compliance management.
How We Got To This Point
The development of corporate compliance programs followed a stream of corporate scandals in the 1970s and 1980s, when industry groups adopted policies and procedures to attempt to prevent and detect misconduct. In 1991, the United States Sentencing Commission amended its “sentencing for organizations” guidelines to offer firms reduced fines if they could show that they had an “effective compliance program.” The United States Department of Justice (“DOJ”) adopted policies urging prosecutors to consider the effectiveness of a firm’s compliance program when deciding on criminal charges. Other civil regulators also adopted similar policies. An industry quickly sprouted to provide compliance-related products and services such as training, whistle-blower hotlines, and risk assessment.
The DOJ recognized that it was often challenging to distinguish substantive programs from those that were merely window dressing, since evaluating a program required considerable time and expertise. In 2015, the DOJ retained Hui Chen to address the challenges of evaluating the actual effectiveness of firms’ compliance efforts. Chen drafted an extensive list of questions for prosecutors to consider when assessing compliance programs. The questions covered a wide range of compliance areas, including training, individual accountability, and leadership. The DOJ publicly released the questions in February 2017 in a document titled “Evaluation of Corporate Compliance Programs.”
The document was not intended to be used as a checklist; indeed, all evaluations would continue to be individualized. Nonetheless, firms quickly began to appropriate the document as a manual on constructing an effective program. Even more worrisome, firms were selectively picking data to support the notion that their practices were effective, rather than recognizing that some were clearly falling short.
How Compliance Metrics Go Astray
In seeking to assess program effectiveness quantitatively, firms tend to make the same mistakes. Here are the common pitfalls:
Incomplete metrics.
For example, the DOJ evaluation document asks: “Has the company ever terminated or otherwise disciplined anyone for the type of misconduct at issue?” To demonstrate individual accountability, firms often list the employees who have been terminated or denied promotions and bonuses as a result of compliance-related transgressions. Yet such statistics aren’t enough to substantiate that a firm rigorously holds employees accountable since they don’t indicate the number of employees who were not disciplined. So the simple statistic on the number of sanctioned employees can be incomplete and misleading.
Invalid metrics.
Although a wide range of data may be collected on the various facets of a compliance program, only a subset of that data actually correlates with the impact of a program. For example, in response to the DOJ question asking how the company has measured the effectiveness of its training, firms often focus on the percentage of employees who’ve completed the training or the number of hours they’ve spent doing so. Those are entirely the wrong metrics to use. A meaningful measure of effectiveness must be directly tied to a clearly articulated outcome—for example, employees’ demonstrated an understanding of policies and procedures, their acquisition of useful skills for confronting anticipated scenarios, or a change in their behavior.
Mistaking legal accountability for compliance effectiveness.
Take this question: “How has the company assessed whether these policies and procedures have been effectively implemented?” Firms often respond by showing that employees signed a statement that they had read and understood the company’s policies and codes of conduct. While such a signature may provide legal grounds to fire someone who violates a rule, it does not demonstrate that an employee has converted knowledge about policies into everyday work practices. Employees may sign an acknowledgment of corporate policies without actually having read or understood the terms. Thus, counting employees’ legally binding assents to policies is not an appropriate way to quantify the effectiveness of a compliance initiative.
Self-reporting and self-selection bias.
Compliance managers often rely on surveys to assess the performance of their programs. For instance, to gauge employee comfort with reporting mechanisms, a firm might ask: “Do you know when to seek compliance advice? Are you willing to do so?” The challenge with surveys is that self-reporting and self-selection by the respondents may bias the results and lead managers to draw incorrect conclusions. Thus, bias in the data collected needs to be accounted for when interpreting the metrics.
Linking Compliance Initiatives to Objectives
So how do you create models that can credibly evaluate the impact of a compliance program? The first step is recognizing that such programs actually have multiple purposes. As discussed in memoranda by senior DOJ officials, the three main goals are to prevent misconduct, to detect misconduct, and to align corporate policies with laws, rules, and regulations. Each component of a compliance program should be linked to one of these objectives to create more meaningful metrics.
Consider a confidential hotline for whistleblowers. Its objective is to improve the timely detection of wrongdoing. To understand whether it’s achieving this goal, certain information is needed, including whether the hotline works (“mystery tester” reports), whether people actually use it (usage data), how they use it (data on types of calls received), the firm’s responsiveness to allegations (response time, investigation completion time, investigation results, communication of results), and whether employees feel comfortable contacting the hotline (periodic surveys of employees’ sentiments). Each of those metrics captures a different dimension of the initiative’s efficacy.
However, tracking those variables independently is insufficient, because it doesn’t allow managers to identify which ones are responsible for particular outcomes. For instance, a “hot” hotline might reflect a rising number of problems or just a high level of employee comfort with calling. To get clarification, managers can apply multivariate regression analysis. Regression models allow an investigator to examine the impact of one variable while holding the others constant. In this case, to ascertain whether an increase in calls indicates an increase in compliance breaches, we would seek to hold the following other factors constant: the availability of the hotline, people’s comfort in using it, its operational performance, and the number of potential callers (people who have access to it).
Designing appropriate regression models takes time and experience, but it is the most reliable way to know whether to be reassured by or concerned about shifts in call volume. As this example demonstrates, firms should use empirical data generated from their compliance programs to gauge how well a program is meeting its objectives. Again, we stress that firms need to do more than simply track metrics independently. They must focus on creating models that measure the desired output while controlling or excluding other factors.
Compliance Engineering
As compliance programs continue to be more closely scrutinized, those that cannot show meaningful results will fail to meet the stronger regulatory standards being applied today. It would be convenient if there were a one-size-fits-all yardstick that could show whether or not a compliance program is on track. But simple univariate metrics will not adequately capture a program’s effectiveness. For compliance programs to have a real impact, managers need to test what works and what doesn’t. This will require firms to engage in some experimentation and innovation. By developing better measures of effectiveness, firms can adopt more ambitious and innovative programs that really do curb the improper behavior.
Companies worldwide are already spending a fortune on compliance. Let’s make sure that all those resources are being spent well. Better measurement can help managers identify redundant or ineffective initiatives that can be replaced or eliminated—and ultimately reveal opportunities to make programs more effective.
This post is a modified and abbreviated version of an original article by the same title published in the March-April 2018 issue of Harvard Business Review. The complete article is available for download here.
Very interesting post and distinctive way to address the issue with our compliance programs.
The constant surveying of employees speaks little of the actual compliance culture as they do tend to be biased and many employees are usually eager to just get it out of the way so may breeze through the questions without actual thought and reflection on their true feelings or experiences. The metrics mentioned are a great proposal to resolve these problems and should without a doubt be utilised.
However, even with stricter regulations and measurements, I believe there will still be the ongoing issue of firms just wanting to meet the bare minimum requirements, checking each box as you put it, for the sake of complying with regulation or fulfilling its CSR. For this to change, there needs to be a full restructuring of corporate culture.
I find this topic to be very interesting, and learned immensely after reading both articles on compliance and doing some research. I found the issue with compliance is not only about effective measurements but also a companies reason for undertaking an effective compliance programs. Compliance started progressing through scandals and unfortunately throughout the years compliance departments only get a breath of fresh air after another round of scandals has occurred. Compliance programs are now compared to that of an insurance policy or simply a paper program despite all the millions spent to create the program.
While I agree that better measurements would make compliance departments more productive, I am starting to believe compliance should be outsourced. An independent firm should be used to actually measure compliance within corporations. Corporations are investing in compliance programs as preventive measures and to adhere to stricter regulations, but how much can we really measure ourselves internally without bias?
Thank you for this interesting article.
Last year, I made a competition law compliance program for a Belgian company. The training focused on how to be compliant with the EU competition rules and how to prevent staff from crossing the line and incurring fines on behalf of the company or on individuals.
First of all, I assessed the management problem within the company. Competition law policies and procedures were in place, but were rather technical and not sufficiently known by the employees. The last competition law refresh was in June 2014. The target audience of that training was only for managers, but not employees. The previous training or relevant existing documents on competition law compliance were no longer used by the target audience. Only the basic and general rules were remembered by the managers. Accordingly, there was a need for a new competition law program tailored to the business needs.
During this consultancy project I have learned that there are three important points that have to be taken into account when making a good compliance programme.
Firstly, it is crucial to identify the target audience for a compliance program. In my case it was important to get to know the business units before I could decide who had to attend the competition law trainings. Only business units encountered competition law issues or questions had a need for compliance training.
Secondly, the content of the training is another important point in a compliance program. The training must be tailored to the business needs and must provide an answer on the legal issues and questions of the target audience. Through online surveys I could measure the existing knowledge of managers and employees and accordingly ascertain which topics had to be trained.
Lastly, the form of the compliance program must be adapted to the target audience. In my case the competition law training had to be given to certain business units without any legal background. Therefore, the training must be presented in clear and understandable language, without complicated legal terms. Further, it was necessary to make the training interactive by asking questions, giving recognizable cases, flowcharts and short movie fragments. HR management decided to give the training in the form of classroom training as this was the most suitable form. As the law changes often and infringements of competition law have very severe consequences for the business, there was need for a periodical revision of the policies and procedures and a refresher training once a year.
In my opinion, a perfect compliance program does not exist. I gave this company all the material to have a successful compliance program: I made an interactive training, a quick guide for each attendant and I established a training plan for the future. But in the end, the success of a compliance program depends on the willingness of the management in a company. The main goal of a corporation is still doing business and making profits. Besides that, there is an overload of compliance programs which corporations have to deal with. From my perspective this makes it rather impossible to be compliant at any time.