Remote Acceptance Testing of Automation Projects
Before the COVID-19 pandemic, it was unthinkable for a system integrator to suggest remotely conducting an acceptance test for an automation project. This article shows how automation engineers and client validation personnel were successful in navigating COVID-19 restrictions and overcoming previously held preconceptions about remote testing to meet end-user and regulatory requirements. Although there were some advantages, both testers and reviewers found the experience inefficient and unsatisfactory in terms of rapport and visibility.
Before the COVID-19 pandemic, acceptance testing of automation projects was conducted in person, with multiple subject matter experts attending. Factory acceptance tests (FATs) were conducted on the system integrator’s (SI) premises, executed by a representative of the SI and witnessed by a representative of the client. FATs could be a simulated operation of the automation system on a bench or the full operation of a machine, skid (prefabricated modular process), or system. Site acceptance tests (SATs) were conducted on client sites in situ, after initial commissioning and before operations. During SATs, all field equipment is connected and operational.
Acceptance test protocols are executed during FATs and SATs. They are approved before execution and during execution; the tester and reviewer/witness initial and date each test and sign that the tests were conducted correctly. The aim of the tests is to verify that the system is compliant with the requirements and the design specification. The granularity of the requirements can vary between industries. In the pharmaceutical sector, the ISPE Good Automated Manufacturing Practice (GAMP)
Measures had been put in place to reduce the close contact of engineers during COVID-19 restrictions, including working from home, and this presented a unique set of challenges for those working in the sector. The writing of requirements, design documentation, and test scripts, and the configuration of hardware and software, can be conducted from home. However, conducting remote factory acceptance testing requires a methodology to allow projects to proceed to the site for site acceptance testing for validation before handover. The objective of our research: What techniques and procedures are required to enable reliable remote acceptance testing of automation projects?
This was a qualitative field study conducted on five automation projects across three SIs and four clients. One client is a medical device manufacturer in Ireland. Another two clients are pharmaceutical manufacturers, also in Ireland. The fourth client is a waste recycling company in Australia. Seven interviews with automation testers and client reviewers were conducted. Qualitative data analysis using the thematic analysis methodology was performed. As reported by Pazhayattil and colleagues, remote testing of biopharmaceutical manufacturing facilities has been successfully executed.
Due to the COVID-19 pandemic and associated restrictions, remote acceptance testing was the only way these tests could be conducted. In these scenarios, the tester is present with the equipment and the reviewer is remote, either at work or at home. Computer screens are shared via remote viewing software; equipment is viewed using webcams, smartphones, or wearable cameras; and participants communicate using telephone, voice over internet protocol (VOIP), or text message. Despite the prevalence of remote acceptance testing in the COVID-19 pandemic, a search for literature on remote acceptance testing yielded no appropriate results.
Research Objective
The research objective is to determine the techniques and procedures that are required to enable reliable remote acceptance testing of automation projects while maintaining compliance with regulatory requirements and industry norms. During the COVID-19 pandemic, the US FDA issued guidelines for remote evaluations of drug manufacturing and bioresearch facilities.
Research Questions
The following research questions serve to achieve the research objective:
- What techniques and technologies are employed to facilitate remote testing?
- What are the advantages and disadvantages of remote testing compared to in-person testing?
- How are projects affected from commercial, quality, and schedule points of view?
Research Design
As a sociotechnical system, research in the field of information systems needs to be defined in a number of ways in order to establish the best approach, methodology, data collection, and analysis choices.
Philosophical Overview
The philosophy chosen, and hence the methodology, depends primarily on where the research sits on the objectivism–subjectivism continuum. Without going into the detail of how it was established, the author chose a value-free axiology, a conventional ontology, and an opinion-based epistemology.
Theory Development and Methodology
In this research, themes emerge from the data collected, which in this case were interviews with practitioners. This is the definition of the inductive approach to theory development. The methodology, strategy, and time horizon were chosen with the development of a timely report in mind. Forgoing the luxury of triangulation, due to limited access because of COVID-19, a monomethod was chosen. As the research lent itself to a qualitative study, the quantitative strategies of experimentation and survey were discounted. It was decided that a case study
Data Overview
Access was limited to existing contacts including work colleagues and clients of the researcher. This enabled short lead times to conduct interviews and had the further advantage of a familiarity with the context. The personal credibility of the researcher and the goodwill between the researcher and the participants aided the access to participants and data collection.
The sampling was of a nonprobability nature, purposive and typical case, selecting engineers who had performed remote acceptance testing since the start of the pandemic. Participants were asked to recommend other participants and one extra contact was found using this snowball method.
| Interviewee | Project | Duration of Interview |
Word Count |
|---|---|---|---|
| Tester 1 | Project 1 | 16 m 28 s | 1,823 words |
| Tester 2 | Project 3 | 12 m 19 s | 1,386 words |
| Tester 3 | Project 2 | 9 m 7 s | 1,268 words |
| Tester 4 | Project 4 | 21 m 22 s | 2,044 words |
| Reviewer 1 | Project 2 | 14 m 57 s | 1,370 words |
| Reviewer 2 | Project 2 | 12 m 10 s | 1,581 words |
| Reviewer 3 | Project 5 | 21 m 36 s | 2,678 words |
| Phase | Description |
|---|---|
| 1: Familiarizing yourself with the data |
Transcribing data (if necessary), reading and rereading the data, and noting down initial ideas. |
| 2: Generating initial codes |
Coding interesting features of the data in a systematic fashion across the entire data set, and collating data relevant to each code. |
| 3: Searching for themes |
Collating codes into potential themes, and gathering all data relevant to each potential theme. |
| 4: Reviewing themes | Checking if the themes work in relation to the coded extracts (Level 1) and the entire data set (Level 2), and generating a thematic “map” of the analysis. |
| 5: Defining and naming themes |
Ongoing analysis to refine the specifics of each theme, and the overall story the analysis tells, and generating clear definitions and names for each theme. |
| 6: Producing the report |
The final opportunity for analysis. Selection of vivid, compelling extract examples, final analysis of selected extracts, relating back of the analysis to the research question and literature, producing a scholarly report of the analysis. |
Seven remote video conference interviews were conducted with participants of remote acceptance tests using Microsoft Teams. Four were testers, three were reviewers. All were male. The range of experience varied, with two having over 20 years of industry experience, three having approximately 10 years of experience, and two having less than 5 years in the industry. Seven companies were represented: two automation and control SIs, four clients, and a project management company. The client companies were two pharmaceutical manufacturers, a medical device manufacturer, and a waste recycling company. One reviewer worked for a project management company on behalf of the pharmaceutical client. The remote testing discussions were conducted on five projects. Three were in Ireland, one was in Australia, and one was in Germany. For the German project, the reviewers were based in Ireland and for the Australian project, the tester was based in Ireland.
The interviews ranged from just under 10 minutes to just over 20 minutes, resulting in transcripts with word counts from approximately 1,200 words to over 2,500 words. In total, approximately 2 hours of recorded interviews resulted in 12,150 words, transcribed (Table 1).
Qualitative Data Analysis
There are a number of qualitative data analysis techniques available: grounded theory,
| Codes | |
|---|---|
| Advantages | Paperwork writing and signing |
| Agreement | Personal property |
| Audio-visual communication | Physical human interaction |
| Change of mind | Planning preparation |
| Commercial | Positive sentiment |
| Delays | Previous experience |
| Disadvantages | Professionalism |
| Feeling curious | Quality |
| Feeling strange | Remote test after COVID - No |
| First remote test | Remote test after COVID - Yes |
| Guessing and assumptions | Roles |
| Hiding | Schedule impact - longer |
| Improvement suggestions | Schedule impact - shorter |
| International | Stress |
| Internet connection | Success |
| Language | Surprise |
| Location | Text communication |
| Mistakes | Time saved on travel |
| Mixed sentiment | Trust honesty |
| Multitasking | Using camera or smartphone camera |
| Negative sentiment | Using remote desktop viewing |
| New company | View only |
| New country | Visibility |
| Subthemes | Further Analysis |
Themes |
|---|---|---|
| Technology and communications requirements Procedural requirements Human factors |
→ | Requirements/ enablers |
| Negative sentiment Positive sentiment Ambiguous sentiment |
→ | Sentiment |
| Future Time Miscellaneous |
→ | Miscellaneous |
All six phases were used in this field study. The interviews were transcribed and read to gain familiarity with the data. The transcriptions were imported into the computer-assisted qualitative data analysis software (CAQDAS) NVivo
CAQDAS is a tool for transparent display of codified data but the researcher still uses the data analysis method of choice and sets up the CAQDAS to accommodate that method.
Ethics
The research was submitted to the University College Cork (UCC) Social Research Ethics Committee (SREC) for approval. The participants are either work colleagues or clients of the researcher/interviewer, which could be perceived as a conflict of interest in that the results could be manipulated to benefit the company. A perception of coercion to participate could also arise. These considerations were mitigated by giving full transparency of the nature of the research to all participants and receiving signed informed consent forms.
Confidentiality concerns were addressed by the anonymization of all persons and organizations in the transcripts, the deletion of interview record-ings after transcription, and the plan to destroy the interview transcripts after publication. All emails scheduling the interviews were deleted. Interview recordings and anonymized written transcripts are stored on the university’s secure server before destruction. There should be no General Data Protection Regulation (GDPR) concerns, as anonymized data do not fall under GDPR regulations.
Results
Codes
After the initial search, 46 codes emerged. These are shown in Table 3.
The codes were then distributed across nine subthemes, and the subthemes were further analyzed and organized under three main themes, as shown in Table 4.
Three Pillars of Remote Acceptance Testing
The requirements to enable remote acceptance testing were collected into the subthemes of technology and communication requirements, procedural requirements, and human factors. These were collected into the theme “requirements/enablers.”
Technology and communication requirements
Essential to the operation of remote testing, the technology and communication requirements include the ability of the participants to see and hear each other, to see graphical screens, to see equipment installed and operational, and to view documents being completed and signed. This is achieved by off-the-shelf technology. Zoom, MS Teams, and Webex are some of the video conferencing systems used successfully. Webcams, smartphone cameras, and sports action cameras were also used to allow equipment to be inspected remotely. This was used to confirm that the correct hardware was installed, that it was constructed correctly, and that it used the specified techniques.
The objective of our research: What techniques and procedures are required to enable reliable remote acceptance testing of automation projects?
Procedural requirements
During in-person testing, test protocols are completed (pass/fail, initial and date) by the tester, and the reviewer signed that they are satisfied that all testing is complete and successful. For remote testing, several procedural changes have been made to accommodate the capturing of document approval. For some, the protocols were signed on camera. In other cases, the tester signed the protocol and the document was scanned and emailed to the reviewer daily, weekly, or on completion. In some cases, the tester was granted the authority to sign on the reviewers’ behalf once the reviewer was satisfied.
Human factors
The participants described various ways in which the lack of personal interaction hindered the testing. The rapport established during in-person testing is missing.
- Reviewer 3: “In a real FAT, you go have dinner, you have a few beers, and the next day, you’d be on first name terms. When you’re doing a remote, that doesn’t happen.”
The lack of the firsthand experience meant that more explicit explanation was required.
- Tester 2: “In some ways, you needed longer to explain things. If a client had been there, sitting there, doing it, interacting with it themselves, they would have seen it more clearly and you might not have had to spend more time explaining it then.”
With the lack of rapport, more effort had to be made to establish trust between the participants. This was made all the more difficult by the perceived ease with which mistakes could be overlooked in the remote scenario.
- Tester 2: “You do need understanding from customers that will have that level of trust and give you credit for being honest. When you build relationships with people, it’s easy for them to do that.”
- Tester 2: “It was the customer who mentioned it in this case, that there has to be a degree of trust. And they trusted that I was being honest in what I was saying.”
- Reviewer 1: “I’ve been surprised about the professionality and honesty of the people involved.”
Sentiment
Codes were organized into the subthemes of positive, negative, and ambiguous sentiment. These subthemes were collected into the theme “sentiment.”
Some examples of negative sentiment include the lack of visibility.
- Tester 1: “Maybe from the point of view of seeing what he wanted to see or experiencing what he would have wanted to experience, it probably wasn’t ideal. Certainly not as effective as in-person testing.”
Another disadvantage was the lack of the personal connection.
- Reviewer 2: “It’s a disadvantage that I couldn’t actually meet the clients in person or the people I’ve been working with. The personal communication, the personal touch.”
Positive sentiment included the ability to multitask and the ability to have subject matter experts attend for short productive spells:
- Reviewer 2: “Yes, so maybe with the likes of online, we didn’t have to waste time. While he was doing that, I was able to do other work.”
- Reviewer 2: “Like my supervisor could have dialed in. All I had to do was IM him, ‘could you dial in for a minute?’ and he came in for 10 minutes to figure out the problem and he could go away then again with his day’s work and he’s not tied up all day at a FAT.”
The remaining codes were collected into subthemes of future, time, and miscellaneous. These subthemes were collected under “miscellaneous.”
Time saved traveling to and from the test site was seen as an advantage.
- Reviewer 3: “That would normally be a full-day exercise, a flight over the night before. And we got it done in 2.5 hours.”
- Tester 3: “Not so much for us, but long term if there was another FAT and it had to be at the client side or abroad, you’re looking at time saved traveling, hotels, money, all that.”
Some ambiguity was expressed.
- Reviewer 1: “I would say that pros and cons were really 50/50.”
- Tester 2: “Plusses and minuses in that way, really for me.”
Tensions
Once coding was complete, the data were queried using a matrix coding query to establish agreements and tensions between the testers and the reviewers. A matrix coding query creates a table to find intersections of codes, to discover patterns [15]. Tester sentiment was split evenly between positive and negative sentiment, but reviewers had an overwhelmingly negative sentiment about the experience (Figure 1).
When asked if remote testing took more or less time than in-person testing (not including time saved on travel), both reviewers and testers were divided in their opinions on an almost equal basis (Figure 2).
Testers and reviewers overwhelmingly agree that they would prefer not to continue using remote testing when the COVID-19 restrictions are lifted. Both groups would prefer to return in-person testing (Figure 3).
Conclusion
The ability to perform remote acceptance testing has been implemented as a workaround in the period of travel and meeting restrictions imposed by the COVID-19 pandemic. It has allowed automation projects to progress to site. Methods, procedures, and techniques that would have been rejected as unacceptable before the COVID-19 pandemic are now de rigueur.
However, after analyzing the data collected in the seven interviews, it is clear that the engineers involved would prefer to conduct the testing in person. They felt that the rapport between colleagues working together builds trust which, along with the unrestricted visibility available in situ, allows for a more efficient and effective test. The major advantage identified was saving time and cost associated with travel to test sites. When one or more parties is required to travel internationally, the savings can be considerable. The savings may outweigh other considerations after COVID-19 restrictions are lifted.
Avenues for future research include investigating the savings to be made by reducing international travel and accommodation costs, studying the dichotomy between the commercial savings and the inconvenience of the testers, and researching the impact of reduced communication and relationship building resulting from remote testing.
