Lessons Learned in Global CQV
Global commissioning, qualification, and validation (CQV) project delivery has in recent years been required to push the boundaries on delivery methodologies and techniques to ensure sufficient production capacity is available to meet ever-expanding patient needs. This article focuses on lessons captured in the execution and resource management of large-scale global CQV projects in an environment of change and compressed project timelines.
Recent years have witnessed increased challenges from global pharmaceutical manufacturers to reduce the standard timelines for the completion of major capital development programs.
In response to these challenges, well-established and long-used CQV methodologies and management practices were reevaluated to determine how project scope could be adjusted to not only meet pharmaceutical regulations and industry standards, but to do so in highly compressed timelines. Established ideologies were challenged and new ways of working were championed to ensure client facilities delivered on schedule.
Additionally, as the number and scale of major CQV projects being executed in parallel increase globally, resource recruitment, onboarding, and retention are becoming ever more important for the successful completion of projects. Companies must be able to not only source and rapidly mobilize experienced teams, but also ensure that key personnel are retained for the full duration of projects.
This article focuses on selected lessons captured during the execution of large-scale global CQV projects. Areas of focus include CQV delivery methodologies, including the one-test approach and implementation of an electronic evaluation platform (EVal); use of virtual factory acceptance tests (FATs); and enhanced/improved CQV project resource management, including recruitment, retention, and onboarding using a competency assessment and tracking tool (CATT). The projects on which these lessons were captured were delivered while maintaining the highest possible pharmaceutical standards to achieve and exceed regulatory requirements.
CQV Delivery Methodologies
Project delivery schedules for a “qualified” plant have received significant focus in recent years in an effort to reduce overall project timelines. These efforts have escalated even further in response to the pandemic, with the need to produce hundreds of millions of vaccine doses in an expedited timeline.
This reality has pushed the pharmaceutical industry to reassess standard practices and project delivery methodologies that have been used for years.
Identifying and eliminating non-value-adding, or wasteful, activities in the preparation and execution of CQV tasks has been a primary focus. Recognizing potentially wasteful activities can be a challenge, as the pharmaceutical industry is rooted in predefined and agreed-upon methods of working that are informed by compliance and regulatory guidance. The implementation of a one-test approach and the introduction of EVal platforms were identified in conjunction with client project execution resources in discussion at the outset of projects as two potential enablers for a reduction in non-value-adding activities.
One-Test Approach
The one-test approach aims to ensure that each test is executed only once and is not repeated in subsequent CQV phases. It attempts to maximize the value of all tests conducted across each project phase. The challenge is embedding the one-test approach at the start of the project, which would ensure that all requirements are identified in the user requirements and are then cascaded through the design, procurement FAT, and on-site testing phases. To successfully implement the one-test approach, all project personnel who support the design and FAT phases must be suitably trained to ensure that the output from the FAT can be used to support subsequent phases.
For equipment packages, the one-test approach seeks to complete as much testing as possible offsite in the FAT phase. This leads to a reduction in the level of on-site testing and thus the potential for repeat testing. Repeat on-site testing is a consistent source of adverse cost and schedule impacts on CQV projects of all sizes. The one-test approach also ensures that the equipment is shipped only after robust testing and that confidence is established that the equipment is functioning as per design and user requirements before it leaves the vendor facility.
Lessons learned from implementing the one-test approach:
- Embed the approach at project kickoff, before detailed design or vendor discussions.
- Establish vendor capabilities prior to any purchase order placement either through a preinquiry assessment process or by performing a quality audit during the package inquiry stage.
- Detail, track, and document the required training for FAT execution personnel to avoid the risk that the FAT testing may not be considered acceptable to support subsequent phases.
- Do not modify design or equipment after completion of the FAT phase. There should be no further design development or equipment modification other than the close out of the agreed-upon FAT punch list.
EVal Platform Implementation
EVal is a digital platform that enables the electronic generation and execution of CQV activities. It is generally used for life-cycle or good manufacturing practice (GMP)-critical documents such as strategy, traceability, and protocols. Templates are drafted, reviewed, and approved within the platform. It also enables full electronic execution. EVal can also be used for non-life-cycle documents, but this will require the team to extend platform access outside the direct project/CQV team to equipment vendors. Some pharmaceutical companies are already extending this access to support a full electronic delivery target to address commonly repeated wastes across the CQV project life cycle.
The implementation and use of EVal can significantly help improve CQV timelines and eliminate waste throughout the documentation preparation and execution phases, but they require a cultural change within organizations to support right first-time delivery. Value is extracted by implementing the appropriate digital technologies throughout the project life cycle to maintain the flow of data, documents, and models to support CQV requirements. The use of digital technologies for CQV is still in relative infancy, in part due to lack of platforms suitably compliant with 21 CFR 11. This is starting to change. Other strategies in use include the partial adoption of some elements of EV such as use of site/company documentation management systems for pre-and postexecution approval of protocols, use of databases to gather equipment, instrumentation and other information during design and construction stages, and subsequently transfer to a site master database. Content of the databases would be verified during CQ activities either at the FAT in the vendor facility or on site during the initial start-up and commissioning stages.
CQV documentation preparation
Document preparation is labor intensive, with documentation preparation waste repeatedly contributing to higher-than-anticipated CQV labor costs. Examples of this type of waste include inconsistent approaches to document formats, inconsistent documentation quality, and poor management of information repeated across multiple documents.
EVal platforms work to address these quality issues in a number of ways, providing for a centralized control of templates throughout the project life cycle and automatically generating protocols and complete systems test packs. Digital implementation also provides for the seamless transfer of design data from the digital design platform to the EVal platform. This reduces the time wasted in the population of data in CQV protocols. It also assists in data accuracy and can eliminate time spent in quality control checking.
CQV protocol execution
CQV execution schedules frequently have multiple critical paths, and a delay in one path potentially impacts multiple other paths. Electronic execution can help reduce project schedule risk by decreasing the potential for execution completion errors, providing real-time progress completion tracking, and offering a platform for rapid deviation assessment and resolution. Using preapproved protocols, electronic execution is enabled via the use of digital handsets, which can be used in the field to support paperless execution and recording.
Implementing digital delivery structures such as EVal requires careful planning to ensure that all parties are aligned on the deliverables, are suitably trained, and have the correct access to the digital platform. EVal can be scaled according to project size and the appetite for its usage, which will be determined by the project team as part of the overall project delivery strategy. Considerations such as the size of the project, schedule demands, cost of implementation, and training requirements must be reviewed to determine the scope of EVal implementation. The overall EVal strategy will be documented as part of the project and CQV strategy.
Important lessons gathered around the use of digital platforms in support of CQV delivery to date:
- Select the platform as early as possible.
- Get buy-in on the project templates up front and then build on the electronic platform.
- Determine the full extent of digital implementation (for example, will all CQV documents be generated or just the project life-cycle document?).
- Determine the level of access to the digital platform (will the platform be solely for CQV team use to generate and execute protocols, or will other project partners such as equipment vendors be provided with access to upload and execute their documentation, such as FATs and site acceptance tests?).
Virtual FATS
The onset of the COVID-19 pandemic in early 2020 impacted major pharmaceutical investment projects worldwide. One of the most significant impediments to project completion was the prevention of process equipment deliveries due to the inability of project teams to attend FATs at vendor facilities.
Traditionally, significant numbers of core engineering design team representatives and client end user personnel traveled to vendor facilities worldwide to inspect equipment and witness test protocol execution as part of FATs. In the pandemic, innovative solutions were required to facilitate FAT testing and ensure the release of equipment for delivery to site. Virtual FATs were conceived as a format where the vendor would execute the FAT at their facility while being observed virtually by core engineering design team representatives and client end user personnel.
When virtual FATs were introduced, there were a number of initial obstacles. Foremost among those obstacles was the need for an industry mindset change, which required engineering firms, clients, and vendors to agree that new ways of working for FATs were possible, that the new ways of working could be implemented quickly, and most important, that these ways met the requirements of all parties.
Another significant challenge was identifying suitable technology, equipment, and applications to facilitate the execution of virtual testing to a level that would meet the requirements of all FAT stakeholders. The required technology needed to facilitate both desk activities (such as reviewing vendor documentation) and field activities (such as piping and instrumentation diagram [P&ID] walkdowns and witnessing live testing). Additionally, vendor facilities needed to have sufficient wifi connectivity, particularly on the factory floor, to ensure good quality, uninterrupted streaming of test execution.
Once these initial connection challenges were overcome, practical structures had to be developed to ensure that the FATs could be run efficiently. Project-specific procedures for FAT testing had to be updated to detail the additional requirements introduced by virtual testing. Training was completed on the updated procedures by all parties involved in the FAT, including vendor resources. Microschedules were developed by the FAT lead for all testing, structured around half-day blocks, which ensured each team member knew when they needed to be available to attend test execution.
Where necessary, vendors’ capabilities had to be upgraded to support the virtual FAT concept by investing in technology upgrades (hardware and software) in workplaces and by ensuring staff were trained to meet the requirements for virtual FAT execution. Additionally, the approval of the use of electronic signatures (e-signatures) was a key enabler for recording virtual testing.
The initial FATs took place at a steady, controlled pace as all parties came up to speed on the processes. Several important lessons were identified during these initial FATs. First, it was key to designate the FAT lead as the single point of contact for all requests—design changes, additional testing—as this ensured that the vendor’s focus remained on executing the FAT and not on addressing questions from multiple different team members. Meeting ground rules ensured etiquette was maintained, particularly where multiple attendees were observing the same sequence of testing. The need to allow adequate time in the daily schedule for the vendor to get set up and address issues was also identified.
When the lessons captured on the initial FATs were implemented, subsequent FATs ran much smoother. The technology was proven to work, quality was not compromised, and the benefits of virtual FATs became evident to all involved. Implementing virtual FATs was also found to be an enabler for the execution of concurrent FATs in multiple locations. Before COVID-19, the availability of specialist resources (e.g., quality assurance, electrical, and instrumentation) would have prevented simultaneous FATs in multiple locations due to the travel time required between locations.
Virtual FAT benefits
The virtual FAT had a number of benefits, one of the most significant of which was safety: Personnel were not required to travel to a vendor facility during the COVID-19 pandemic. Companies also experience a reduction in costs associated with FAT travel and accommodation. The reduced travel also led to carbon dioxide reduction. On one major capital project in 2021, an estimated 250 metric tons of carbon dioxide was offset by reduced FAT travel.
Virtual FATs also showed an increase in quality and productivity. Several FATs were attended on the same day in different locations. Virtual FATs pro-vide for the right people being available at the right time. FATs were recorded with all-party agreement. Teams were focused on what was on the monitor.
Although virtual FATs have facilitated a new way of working on capital projects, they have not fully removed the need for the core engineering de-sign team to attend vendor facilities. Engineering design team representatives are still advised to regularly visit vendor sites pre-FAT to ensure the vendor is on target for FAT. Additionally, there should be a provision for core engineering design team representation at the FAT itself. This core engineering design team will support coordination at the vendor facility, witness specific testing (such as vessel drainability and pump range), and assist with the setup and operation of the audio-visual equipment required for the FAT to run successfully.
CQV Project Resourcing
Resourcing has taken on a new significance in an environment of increasingly compressed project schedules and newly emerging execution approaches, such as those discussed previously. The recruitment, retention, and onboarding of resources are key activities in support of successful CQV project delivery.
Personnel Recruitment and Retention
There are four elements that have been found to significantly contribute to a successful recruitment and retention program for CQV projects: a dedicated global CQV resourcing manager, an in-house talent acquisition team, referral rewards, and reward and recognition programs.
Dedicated global CQV resourcing manager
A coordinated, centralized approach to personnel management is crucial where companies are executing multiple major CQV projects in parallel for different clients. Having a designated global CQV resourcing manager has been found to provide a coordinated CQV resourcing effort.
One of the resourcing manager’s primary goals is to ensure that the needs of all major projects are met equally. Weekly meetings should be held with individual CQV project managers to ensure all resourcing requirements (e.g., number of roles, experience level, need date) are tracked. The information should be collated on a centralized database and used to prioritise project resourcing needs.
The resourcing manager should coordinate daily with the Talent Acquisition (TA) team to review progress on candidate searches, plan candidate interviews, and align on commercial negotiations. Where relevant, the resourcing manager can also communicate across internal company departments to identify resources from other sectors or departments (e.g., design, construction) who are qualified for and interested in a CQV execution role.
Talent retention is a key activity for the CQV resourcing manager. The establishment of a long-term relationship between an employer and a capable CQV resource is beneficial to both parties. The CQV resource is provided with long-term continuity of employment, and the employer can plan across multiple projects and mitigate against the risk of personnel leaving a project before completion. Building the long-term relationship requires regular contact between the CQV resourcing manager and individual CQV resources, particularly in the last four to six months of a project, when resources are naturally beginning to consider the next steps in their career paths.
Talent acquisition team
An experienced in-house CQV talent acquisition (TA) team will play a key role in identifying and bringing suitable candidates through the recruitment process. The dedicated TA team will support candidate recruitment by working with the CQV resourcing manager to (a) ensure the resourcing effort re-mains continually focused on the prioritized open roles as identified in the resourcing database, and (b) identify opportunities for talent sourcing such as global projects nearing completion and recruitment platforms search profiles.
Beyond actively supporting resourcing, the TA team will positively represent the company and client projects to candidates throughout the recruitment progress, develop multimedia content for use on company media platforms and external recruitment, and maintain roles on the company career portal and recruitment websites.
The recruitment, retention, and onboarding of resources are key activities in support of successful CQV project delivery.
Referral rewards
Referral rewards allow employers the opportunity to use the contact networks of their employees to source new talent to join their organizations. Experienced CQV team members are often the best recruiters, as they have built up a network of similarly knowledgeable contacts and will be driven to refer suitable candidates by the desire to be part of the strongest possible CQV team. Companies need to ensure that the terms of the referral rewards are sufficiently generous to make it worthwhile for existing team members to recommend candidates, with minimal administration requirements for employees to obtain their reward when a recommended candidate is hired.
Reward and recognition programs
Reward and recognition programs provide a platform to support project safety, quality, and schedule objectives while also supporting personnel retention. The majority of major capital projects in the pharmaceutical industry now have safety recognition initiatives in which the performance of individuals or companies is audited on an ongoing basis, with weekly and monthly rewards for excellence.
Companies are also implementing programs where individual team members are rewarded for the overall project safety performance, project quality (right first-time execution), and long-term commitment to the project. Retention bonuses pay team members a lump sum at project end if they remain until completion. The lump sum must be large enough to entice an individual to stay on until the end of their contract. Such programs also assist with the long-term retention of key personnel.
Resource Onboarding
Once hired, CQV resources need to be appropriately mobilized and onboarded on site in a timely fashion, without impacting existing project progress. Effective onboarding of new resources is challenging in any work environment, and especially so in live CQV environments.
New project resources need to be brought up to speed on all applicable aspects of the project (plant layout, P&IDs, automation interfaces, safety procedures, ways of working) as quickly as possible while ensuring ongoing CQV activities are not adversely impacted. Upon completing the onboarding process, CQV execution resources should be sufficiently competent to execute live commissioning activities in the field, either on their own or in teams with minimal direction and supervision.
Standard onboarding structures are normally designed to handle a small number of new hires (one to three resources) per month on a project. When onboarding numbers increase in excess of five resources per month, onboarding programs stop functioning efficiently, resulting in delays in new hires becoming effective additions to the existing team.
New project resources need to be brought up to speed on all applicable aspects of the project as quickly as possible while ensuring ongoing CQV activities are not adversely impacted.
Onboarding case study
The challenge of onboarding a larger number of resources presented itself on a major capital CQV project in 2020, which required over 20 CQV execution resources to be onboarded onto a schedule-critical project within a three-month period while ensuring (a) the safety of both new and existing personnel was not impacted, (b) new hire onboarding was completed as per in-house project and client site procedures, and (c) planned project execution progress in the three-month period was not impacted.
Utilizing experienced team members to mentor new hires on field execution activities during their initial six weeks on site was a key part of the existing onboarding process and needed to be retained. The increased number of new hires to be mentored required the workload of experienced team members to be balanced between achieving planned project execution progress and supporting the mentoring process.
While training on the CQV procedures (documentation, classroom lock out/tag out [LOTO] training) was possible for the new hires within the required six-week timeframe, successful completion of field training in this period was a major coordination issue for existing team members.
A competency assessment and tracking tool (CATT) was developed to plan, coordinate, track, and record all aspects of the onboarding program of all new hires while maintaining project safety, quality and schedule targets.
CATT
The CATT compiled the details of all hiring and training into a single database. Each new hire was assigned their own tab in the database dedicated to recording their training progress. Template pages were developed that captured the specific CQV area training programs (upstream, downstream, CIP) with the new hire assigned to the training program for their CQV work area. The page also contained the assigned mentor, duration of mentor period, and a list of the activities the new hire could and could not carry out during their initial mentoring period. Confirming what tasks new hires were not to complete was regarded as a key safety benefit of the CATT implementation.
The CATT database was managed by a CQV project engineer responsible for all aspects of the database, including coordinating with new hires, managing day-to-day issues with the training programs, and ensuring regular updates were provided to all stakeholders. One of the primary responsibilities of the CQV project engineer was to hold weekly meetings with CQV management representatives (project, technical, and safety) to ensure proactive management of the training progress of each new hire.
In advance of the weekly meetings, the project engineer updated the individual pages of the CATT database with training progress information gathered from the assigned mentors and CQV area leads. The meeting provided for additional training supports to be put in place if deemed necessary based on constructive feedback or allowed for the acceleration of the initial mentoring period based on suitable positive feedback. The weekly meeting also allowed for regular review of mentor workloads to ensure a balance was being maintained between mentoring and execution tasks.
When new hires were deemed to have successfully completed the initial mentoring phase, the CATT was signed off by the CQV manager and the relevant CQV area lead. A “close out” section confirmed the specific tasks the new hire was deemed competent to perform. Where relevant, it recorded any remaining specialist training that the new hire may yet have to complete (such as vessel entry).
The initial implementation of the CATT provided some key early lessons, which were subsequently incorporated into the program.
- Early engagement and alignment on the information in the CATT: Ensure the new hire, their assigned mentor, and the area CQV lead meet to review the CATT and align on the content at the outset of onboarding.
- Mentor selection: Mentor suitability can be influenced by many factors, such as mentor workload, new hire experience, and any previous time spent working together.
- Regular communication on progress during the mentoring phase: Feedback to the new hire from management is key, especially in cases where resources are struggling to meet the initial assessment progress expectations. This feedback should include the development of an agreed-upon support plan.
- Implementation of tools such as the CATT: Use of a CATT can assist in rapidly deploying competent resources to meet peak loading demand in fast-track project timelines while maintaining safe working environments.
Conclusion
The timelines required to deliver projects to meet patient needs will continue to challenge the CQV sector. The execution of CQV practices will continue to need reassessment to identify more efficient ways of working. The lessons outlined in this article represent a small cross-section of steps taken on recent large-scale capital projects in support of accelerated timelines. The new ways of working identified here must be embraced and implemented in a structured, regulatory-compliant manner to fully realize the benefits for project execution and, ultimately the benefits for the patient.
