The First Quantitative Industry Assessment of Single-Use System (SUS) Reliability: Raising the Bar for BPSA’s Value to Industry

Scientist writing test results while standing in the lab. Protective suit and gloves on.

Over the past 24 months, the leadership of the Bio-Process Systems Alliance (BPSA) has initiated a pilot program to demonstrate the joint feasibility of collecting and sharing pertinent industry business data. We began with “quality-complaint data” centered on the premise that product complaints are tightly associated with product-quality defects in single-use systems (SUSs). This approach can be viewed as “risk assessment 101.”

BPSA’s Representation of the Single-Use Industry
BPSA is an industry organization primarily comprising drug manufacturers, single-use system suppliers, SUS component suppliers, specialty test laboratories, and educational institutions. The organization’s goals are to “advance the adoption of single-use worldwide” through technical guides, national and international conferences, and advocacy efforts around and within industry standards groups and regulatory bodies; and through providing a safe-harbor environment allowing customers, colleagues, and competitors to interact and share knowledge. To that end, this BPSA-data effort has been funded and contracted by its members. We are pleased to share the results of our initial efforts with an eye toward expanding the BPSA Market Data Program in 2019 and beyond.

Why Market Data? Why BPSA? It is said that “nowhere [else] can you replicate the all-encompassing common bond, mutual respect, hunger for information, and pride for being a part of an industry that is experienced by members of a professional or trade association. Not even through LinkedIn, Facebook, or Twitter” (1). No mere online platform can collect and populate snippets of data to compile an analytical report that addresses niche SUS issues such as market size, system reliability, and data-driven growth projections. That best can be accomplished with subgroups of knowledge leaders and subject-matter experts (SMEs) within a chosen industry and through a dedicated industry organization.

Through its status as a nonprofit entity for and about the business of single-use manufacturing technologies, BPSA is uniquely positioned to facilitate accurate, legal, and confidential data sharing. The structure of the program described below is such that additional BPSA member supplier companies can join in the program if it is extended beyond this pilot stage. BPSA’s first foray into data capture is a proof-of-principle exercise by its members, and it offers great potential to expand into other data spaces. No longer are anecdotal data enough. Capturing real data is the goal. To begin, we focused on the main risk-assessment challenge: single-use system reliability.

SUS Reliability: Project Background
SUSs for biopharmaceutical manufacturing have gained popularity over the past several years and are now viewed as a critical manufacturing modality and a complement to traditional stainless-steel manufacturing. SUS technology and drug manufacturers’ strategic use of it have evolved to the point that the following advantages of SUSs are commonly accepted:

Assessing Risk in SUS — Real Data Are Essential: As the importance of SUSs grows, so too does a focus on the risks associated with a manufacturer’s reliance on them. One risk is the impact to both clinical and commercial manufacturing associated with using a defective SU assembly. As with biomanufacturing overall, drug manufacturers’ risk management of SUSs would be greatly facilitated by the availability of accurate and reliable data about defect rates. At present, companies are forced to rely on internal data and invariably are influenced by anecdotal reports of high failure rates.

From a complementary perspective, suppliers of critical equipment to the biopharmaceutical industry should practice continual quality improvement. Defect rates represent a fundamental measure of quality. Such a measure industry-wide would enable suppliers to benchmark not only against their internal quality and trending data, but also against that of the industry as a whole. Thus, the challenge to the SUS industry is to provide its customers with what they need for evidence-based decision making, quantitative risk assessment of SUS, and better-informed approaches to continuous improvement.

Data Collection and Management — A Crooked Path: When initiating a data-collection effort with no precedent to lean on, a company encounters strict, intersecting boundaries. A first question — “is it legal?” — segues into definitions of product categories, complaint categories, and ground rules around confidentiality, along with the deceptively simple question about who gets to see the data. Those procedures and legal checkpoints matter, and BPSA addresses them in depth.

Legal Requirements: Regardless of industry, programs intended to aggregate confidential data must meet the “3/70” rule. This criterion holds that a program and any single element of that program must include at least three companies, none of which can have a market share ≥70%. Therefore, BPSA’s data from participating companies were submitted solely to a neutral and independent third party whose responsibility was to aggregate the data accurately and ensure compliance with the 3/70 rule.

Product Categories: The initial list of product categories included seven entries. The 3/70 rule applied to each product category. Therefore, we combined similar product categories as needed to reach its criteria. To illustrate, the initial list included “2D bioreactors and 3D bioreactors,” so they were consolidated into the final category of “2D and 3D bioreactors.”

The final list of product categories is as follows:

Table 1: List of complaint categories

Complaint Categories: For our purposes, a complaint is customer feedback that has been entered into a participating company’s complaint handling system and that has been classified by the participating company into one of the nine complaint categories listed in Table 1. That list was derived by applying the 3/70 rule to the complaint categories of each participating company and aggregating complaints into more generic categories to meet the 3/70 rule, all the while providing the highest level of granularity possible. For example, the more detailed categories of cosmetic — film wrinkles/creases and cosmetic discoloration — were consolidated into the final category of cosmetic. Each supplier’s complaint categories have been mapped to the final list, ensuring that no recorded complaint is discarded in the exercise (Table 1).

Figure 1: Number of complaints per thousand units sold

Data Collection and Management: To date, the five participating companies are GE Healthcare, Millipore Sigma, Pall Biotech, Sartorius Stedim Biotech, and Thermo Fisher Scientific (in alphabetical order). Going forward annually, they plan to provide data on the number of units shipped and number of complaints received. Those data will be blinded and provided to a contracted third party: No participating company will have access to data provided by the other participating companies. The total number of complaints and units shipped will be aggregated per complaint and product category combination (Table 1).

The ratio of total complaints to total units shipped will be calculated for each complaint–product-category combination to yield the complaint rates that will populate Figure 1, which is provided here as an illustration. Those ratios preclude visibility of the total number of units sold in the industry because that was not the objective of this pilot study.

Discussion and Expectations
The logistics involved in assigning each complaint to the year in which its associated unit was sold were prohibitive. Therefore, the team decided to pull complaint data and shipping volume data from the same year, thereby accepting a certain level of error in the complaint-rate calculations. The expectation is that this “time-lag error” will not affect the utility of the data. Because the magnitude of the error will be less than the rate of industry growth, the absolute values of complaint rates will not be affected meaningfully.

To illustrate, if the magnitude of error is 10%, then a reported complaint rate of 0.01% may have a more accurate value of 0.011%. Moreover, because the value of the error is not expected to change year on year — industry growth rates are relatively steady — the accepted error should remain constant. For that reason, the utility of year-on-year data trending will not be compromised because of the time-lag error. Complaint rate is a surrogate for the more meaningful measure of defect rate. The time-lag error also will delay the impact of continuous improvement initiatives (aimed at reducing defect rates) on complaint rates. An improvement step taken in the first year might manifest as a complaint-rate reduction only in the second year. Through experience, we expect the magnitude of that effect to be likewise small — 10% or less.

The complaint rates include only those issues within the participating company’s complaint-handling system. Largely for that reason, complaint rates and defect rates do not perfectly align. A significant reason for nonreporting is that some end-user companies report defects only on items that are used in GMP settings, but not, for example, on those used in process development.

Aligning Improvement Initiatives
Through use of complaint rates derived from the BPSA data-gathering initiative, end users can assess the risks of SUSs quantitatively to help make comprehensive manufacturing-network business decisions. This initiative will allow each supplier and end user to benchmark its own performance against industry averages. Each group then can target improvement initiatives toward complaint-rate categories that align with its specific business or quality goals. Through initiatives of this sort, we aim to enable industry-wide improvements in SUS reliability.

Reference
1 Silverman K. Recruitment and Retention: The Value of Market Research for Professional and Trade Associations. Market Research Blog, 12 March 2015; https://blog.marketresearch.com/value-of-market-research-for-professional-trade-associations.

Janmeet Anant is a regulatory advocate, MilliporeSigma. Jeffrey Carter, PhD, is consumables product strategy leader, GE Healthcare Life Sciences. Hélène Pora, PhD, is vice president of technical communication and regulatory strategy, Pall Biotech. Paul Priebe is head of application specialists, key accounts, Sartorius Stedim North America. And Clive Wingar is director, integrated solutions, bioprocess equipment and automation, single-use technologies, bioproduction, Thermo Fisher Scientific. Other contributors are Steven Teslik, CEO, K545 Analytics, LLC, and Kevin Ott, executive director, BPSA. For further information on upcoming technical and data initiatives within BPSA, please contact Kevin Ott, [email protected].

To share this in PDF or professionally printed format, contact Jill Kaletha: jkaletha@mossbergco. com, 1-574-347-4211.