Creating a Successful Central Metrics Group

Thanks to a focus on productivity and cost reduction within the pharmaceutical industry, along with a prevalence of Six Sigma-inspired improvement efforts, metrics are on a lot of people's minds. In fact, one might argue that metrics are on too many people's minds. Consider the following situation, where you are collecting data to track cycle times for processes within your area of responsibility.

At some point you're talking to a colleague about the idea of creating a handy dashboard to display all of your key metrics in one location. Your colleague, having some experience with dashboards, sends you a prototype of one she's working on in her department—and it contains much of the same data you have been painstakingly collecting over the past several months. Worse, there are discrepancies between your numbers and hers.

This story is undoubtedly all too familiar within many departments in clinical R&D, an area that has only recently begun to develop a true operations mindset. Most, if not all, managers of R&D departments recognize the criticality of collecting data and tracking the performance of their many processes. However, many of these efforts are disjointed, inefficient, and misguided. Conflicting reports, duplication of efforts, and misapplied metrics plague many companies' attempts at making sound data-driven business decisions. The creation of a central metrics group can go a long way toward eliminating wasted effort and drastically increasing the value of an organization's data.

Needs assessment

Before thinking about how to create a central metrics group, one must first ask why they should create one. Is it even necessary? The answer, of course, varies from situation to situation.

Certainly all companies, regardless of industry, should adopt a continuous improvement mentality, meaning that metrics should play an important role in just about every business. But a number of factors will help to decide whether or not a central group is appropriate. Company size is one, as larger companies will typically benefit more from such an arrangement by consolidating voluminous amounts of information and reducing duplication of efforts. A company's technological maturity is another key factor, as central metrics groups tend to rely heavily on an advanced technological infrastructure for the collection of data and dissemination of information. But perhaps the most important factor is the overall corporate strategy.

The degree of commitment among top levels of the organization with regard to continuous improvement and their views on how it should be accomplished will ultimately be the deciding factor. Companies with a high level of commitment toward improvement will take the time to seriously consider whether or not a central group will work best for them. For many, the benefits of increased efficiency and consistent reporting of data will outweigh the bureaucratic headaches that come with setting up such a group.

Adding value with metrics

Too often, metrics are collected largely for the sake of collection. Alternatively, metrics are utilized by individual departments to make their lives easier (e.g., to keep tabs on things out of mere curiosity) or are measured against self-imposed yet meaningless targets. This often leads to localized optimization within the R&D value stream and a misalignment with other departments and perhaps the whole organization.

For example, many clinical operations groups focus on the cycle time from protocol approval to first-patient screened. In the long run, does it really matter how quickly one site begins enrolling patients? The critical measure—the one that speaks to the flowing of data to the next step in the clinical development value stream—is when all sites finish enrolling.

This example demonstrates the "management by intuition" approach that seems to run rampant throughout the clinical R&D space. Years of experience, although inarguably valuable, seem to be commonly used as a substitute for hard data as opposed to a supplement. The sooner that clinical departments recognize the fact that metrics can be used to validate intuition and unearth previously hidden trends and root causes, the better off their decision making will be.

Regardless of how a company structures its metrics function, corporate vision should set the context for performance measurement.¹ (Figure 1 shows one possible strategic management framework.) Such a framework assures that performance measures align from top to bottom, which in turn puts focus on metrics that show how well value flows through the value stream. A central metrics group can do a much better job of maintaining this type of framework than disparate entities can due to the coordination inherent in the structure of a central group. It should be part of the central group's mission to keep metrics in alignment.

Figure 1

In order for a central metrics group to be effective, it must have a thorough understanding of both the business and basic analytical methods (in order to properly determine root causes of problems, etc.). To this end, it is often worthwhile to create a group utilizing staff from a variety of areas, such as clinical, operations, and data management. Ultimately, the skill requirements will be based on the central group's scope, which needs to be defined early on. For example, will the group only handle late-stage development processes or everything from discovery through product launch?

The next critical responsibility for the central metrics group is defining key performance indicators. For any given process, there should only be a handful (no more than five or six) and, more importantly, they should speak to the true nature of performance in light of the higher level objectives. Therefore, an organization ultimately concerned with productivity should lean toward metrics on outputs (e.g., patient visits processed) and resource allocation (e.g., FTEs utilized). An organization whose primary goal is to shorten time to market should focus on selecting key milestones that will help it manage business from a speed standpoint. Ideally, the central group will work with members of the respective functional areas to define the key performance indicators (see Figure 2), thus keeping metrics in alignment while still satisfying the business need for useful management reports at the operational level.

Figure 2

Defining key performance indicators (see Table 1) requires a detailed understanding of not only the process at hand (having process maps to work with is obviously helpful), but of the suppliers of the inputs (data) as well as the customers' requirements for the outputs (dictated by what they plan on doing with the information they receive).

Table 1. Examples of key performance indicators.

Ensuring lean reporting

A central metrics group can potentially have many different customers from all areas of the organization. Identifying critical-to-quality requirements for these customers is the first step in defining a data collection, analysis, and reporting process that satisfies customer needs in the most efficient way. Critical-to-quality requirements might center on the timeliness of the reports (how real time the data needs to be), the level of detail required (e.g., the patient visit level), drill down capabilities (going from protocol level to visit level), ease of use, and accessibility.

Keeping the critical-to-quality requirements in mind, the group must then identify sources of data and evaluate the underlying system's ability to provide that data on a just in time basis. Where required key performance indicators data is not available (or where the current system will not adequately support the new metrics process), a decision must be made to either implement a new system to collect the data or to alter or eliminate the key performance indicators—keeping in mind that doing so may compromise the effectiveness of the metrics program.

Next, the methods for refreshing and distributing standard metrics reports must be determined. Typically, this will involve automation of data collection, the use of company intranets, and other tools that allow the ultimate customer to pull the reports whenever they need them. These real time reports and the expectations surrounding them must be carefully planned early on to avoid customer frustration later.

Metrics reports are of little use, of course, if they are not accurate. It is therefore important to implement controls (mistake-proofing) and checks (quality assurance) within the process to ensure accuracy of information. For example, using a clinical trial management system that makes users select inputs from drop down lists reduces the chance of data entry errors. Having the system prohibit the user from moving to the next screen before all required fields are filled in can greatly help eliminate gaps in data. Automating queries of metrics data can also provide systematic stability to the process. Meanwhile, a gate review system on data quality and whether or not the final report meets the critical-to-quality requirements will further reduce the chances of the customer being dissatisfied.

Clearly, ensuring accuracy is one of the most important jobs of the central metrics group. As such, a significant amount of time should go into the planning, implementation, and continuous improvement of the metrics QA/QC process.

One final distinction in the development of a metrics reporting process must be noted. The standard reports mentioned earlier refer to those reports that contain the agreed to key performance indicators, have a standard design, and are simply refreshed on some recurring schedule as dictated by business needs. Users are also likely to request ad hoc reports from the central metrics group. Although the general process for gathering and reporting data will be largely similar, creating ad hoc reports is a far more resource intensive task due to the volume of requests, difficulty in finding data, complexity of the requests, and the need for additional quality checks. It is recommended that these types of requests be scrutinized by the central group to determine if there is legitimate business value in addressing them and, if so, how they should be prioritized.

Central metrics organization

Once the processes have been defined, work can begin on creating the organizational structure of the central metrics group. By identifying all tasks that make up the new processes, an estimate of required resources can be made. One uncertainty here is the volume of ad hoc requests that the group will need to support, which needs to be addressed with the group's executive sponsor ahead of time.

Next, determine if new job descriptions need to be created or if tasks should be assigned as parts of existing job descriptions. Again, this will largely depend on the level of commitment and support from senior levels in the organization. It is critical that the proper amount of resources be dedicated to setting up the central group or it may be doomed to failure.

Once the job roles are made clear and line and staff responsibilities identified, relationships between process owners and the central metrics function should be defined. For example, will the site management process owner be assigned one particular resource in the metrics group (the expert in that area) or will the group lead assign a resource based on availability? Clearly, the more flexible the group is with its resources, the better it will be able to handle the flux of incoming work.

Another consideration in creating the new group is how process improvement projects will be identified and executed. Within a management framework such as Kelvin and Cross's Performance Pyramid mentioned earlier, the central metrics group will produce items of integrated financial and nonfinancial information that operating managers can use as a catalyst for process improvement.¹ How much involvement should the central group have in these process improvement efforts? Although obviously a valuable resource for such activities, care must be taken to not overwhelm the central group and distract it from its core mission.

Lastly, a final organization chart needs to be created for the central metrics group so that the structure and relationships are clear to both those internal and external to the group.

Change, documentation, and training

It is quite likely that the establishment of a central metrics group will be a radical shift for most R&D groups. Regardless of how much effort is put into creating the processes and structure of the group, if people are not aware of the new group and how it will function—and, most importantly, how it will benefit them—it will wither and die.

As usual the best place to start is at the top, by gauging the level of executive support for the group. Is the effort being trumpeted by the highest levels of management as a showpiece of their effectiveness and innovation? Or has the green light been tentatively given with a wait-and-see attitude? The amount of drum beating required at the ground level (i.e., departmental communications, informational presentations, informal meetings with key personnel, etc.) will be determined by the answer to these questions.

If the higher-ups are trumpeting the effort, it's likely that most people will get on board—whether they want to or not—without having to be pushed too hard. If, however, they are tentatively giving approval, some serious consideration needs to be given to how the new group will be marketed to the larger organization. As mentioned, showing people what's in it for them will make all the difference. If others within R&D can be convinced that they can get what they want when they want it, and with little or no effort on their part, the central metrics group will be on its way to organizational acceptance.

Having an idea of high-level support, the next step is to go about identifying all stakeholders and engaging the key ones in developing implementation plans and identifying potential obstacles. This not only provides additional buy-in from key stakeholders but can eliminate potentially serious problems before they have a chance to derail the new group. Developing and executing a communication plan is critical (i.e., who gets what information, how frequently, and in what form). People in the organization, especially key stakeholders, need to know what's happening at all times so they feel connected to the change and also so they can communicate back issues or concerns they may have. Again, the voice of the customer needs to be kept in the forefront of everyone's mind; the central group lead should be mainly concerned about adding value, and value is always defined by the customer.

Finally, training programs need to be developed, not only on the new processes but also on how to use the new reports. It cannot be assumed that all end users will automatically understand what they are supposed to do with the information that will be available to them. The end goal is to enable better business decisions, and all personnel need to be shown how the new reports (and the central metrics group as a whole) will help them reach this goal, and what limitations exist.

Implementation

Many promising initiatives fail in the implementation stage. Too often, everyone is so relieved that the solution is in place that they fail to support it when it counts: at rollout. The key issues at this stage involve managing the new process and workload, fielding questions and concerns, and tweaking the new process as needed. While this is largely the responsibility of the central group lead, having the support of key stakeholders (as discussed previously) can be extremely helpful in addressing any issues that arise while at the same time minimizing damage to the credibility of the new group.

Conclusion

This article has shown the high-level steps that need to be taken in the establishment of a central metrics group. The more detailed steps will of course depend on the answers to the high-level questions raised throughout this article. But perhaps the most important part of the whole process occurs in the beginning, at the point when the decision is made to undergo such an endeavor in the first place (see Table 2).

Table 2. Benefits and risks of establishing a central metrics group.

As stated earlier, it is critical to maintain a top-to-bottom alignment of performance measures as strategic goals are linked to operations by translating aggregate market and financial goals into operational terms.¹ The drive to create a central metrics group should be based strictly on the need to make better, faster decisions within this context.

The justification should be based on a business case that clearly states that the value in establishing a central metrics group far outweighs the costs. Even though pharmaceutical companies exist in an industry founded on science, they are businesses at heart and need to be run as such.

There are many secondary benefits in setting up a central metrics group. Metrics always need to be actionable, and meaningful key performance indicators can be used to create high-level scorecards and tactically manage studies or smaller processes. Creating baselines on which to base future improvement efforts, establishing a repository for organizational knowledge, and identifying and communicating best practices are all crucial in adding value to the organization.

There is also the possibility of measuring the company against external benchmarks, but this author tends to agree with the comment by Womack and Jones: "To hell with your competitors, compete against perfection by identifying all activities that are muda [waste] and eliminating them."² Regardless of one's views toward benchmarking, it is clear that what data is available for comparison is selectively shared, due to either a desire to avoid giving away too much or a reluctance to participate in the sharing of metrics via one of the existing benchmarking companies.

In an ideal world, metrics customers would be able to get the exact information they need exactly when they need it, and without errors. These attributes closely reflect the lean concepts of value, flow, pull, and perfection that are sought after by all companies regardless of industry. Using these concepts will enhance the value of any central group, and bring order to the sometimes chaotic world of clinical R&D metrics.

Eric Lake is a partner with Pharmica Consulting, PO Box 2754, Oak Ridge, NJ 07438, (973) 945-4482, fax (973) 726-3204, email: eric.lake@pharmicaconsulting.com

References

1. R.L. Lynch and K.F. Cross, Measure Up! How to Measure Corporate Performance, 2nd Ed. (Blackwell Publishing, Oxford, 1995).

2. J.P. Womack and D.T. Jones, Lean Thinking, 2nd Ed. (Free Press Publishing, New York, 2003).