Let’s take a glance at the past. In the pre-digital era, protection devices were set up with only a few parameters (Figure 1). These parameters could be tested easily, while the results were documented by hand. Then, incredible developments in the field of microelectronics completely changed the game. Over roughly 30 years, ever-new advancements entered our daily lives. It was only a matter of time until they also became present in the substations and ultimately arrived in protection relays.
Electromechanical and static devices’ parameters were limited, making it possible to test them all. The first generation of digital devices only had a few additional parameters. So, nothing really changed, as checking every parameter during commissioning or routine testing was still possible. But soon, new technologies and the possibilities they offered to protection engineers led to many parameters that could no longer be tested in the field.
Therefore, sending a test engineer into the field to test a substation’s protection devices is no longer sufficient. Today, there are far too many tasks to do before testing and details to recognize during a test. Preparation is vital nowadays and has the most significant potential for efficiency gains.
Before we dive in further, one more thing needs to be defined clearly – the scope of the term protection testing. This article not only considers a single protection testing task but instead tries to cover the long-term efforts and potential connected to testing assets within substations over their entire lifespan to create a realistic and comparable picture of protection testing. For that reason, the term includes testing strategies and the full scope of the protection testing task, from preparation to execution to documentation, when considering effectiveness and efficiency.
Ensuring Effectiveness
As mentioned above, testing all the parameters of a modern protection relay in the field is no longer feasible for testing engineers. Before they even start testing, they need to know which parameters must be tested to be effective. An effective protection test requires procedures that contain the necessary steps – nothing more, nothing less. In simple terms, effectiveness can be described as “Doing the right things.”
As each test has its purpose, not every test needs to be performed regularly. Some tests are only relevant for a specific phase in the life cycle of a protection device. In contrast, others should be performed more frequently to ensure the reliable operation of the relay.
Regardless of the test’s purpose, before a test case can be run, it must be defined and prepared accordingly, as seen in Figure 2. All these tests will be effective if performed correctly, but is this approach also efficient?
Improving Efficiency
Efficiency depends on various factors, from the testing approach, state-of-the-art tools supporting it, and the test engineer’s experience. However, all of them are subject to change. On the one hand, test objects are advancing and use new technologies. On the other hand, protection testing solutions add new features, and test engineers are becoming more experienced. Of course, they could also miss out on catching up on new developments. These factors must be considered when trying to “Do the right things correctly” which is a basic definition of efficiency. Or, to say it another way – executing the right test procedures, without mistakes, in the shortest amount of time.
The two most significant factors for efficiency are usually standardization combined with automation. Once the most effective testing approach is found, further efficiency gains are highly dependent on eliminating manual interventions in the testing process. While this clearly can be achieved with automation, the level of automation is essential. This is where standardization comes into play, as it creates the prerequisites for using automation within or even across testing approaches. Generally, the more standardized an environment is, the more automation is possible. Whereas single-function tests can be automated relatively easily, a complete test procedure for a protection relay can also be automated to a large extent depending on the degree of standardization.
State-of-the-art tools allow you to combine automated test steps in templates flexibly, capture the results automatically, and create required legal documentation from them. This is important to mention, as a clear scope is needed for effectiveness and efficiency to be able to measure it. Preparation and documentation are part of the protection testing task and must be included to cover the entire picture. However, efficiency gains are not the only reason automated testing based on pre-defined templates should be favored because there’s also test quality.
Quality Aspects of Protection Testing
Quality is closely connected to the effectiveness of a test. Similar to efficiency, the quality of a test can have different aspects, which is why we first need to define the properties of a high-quality test. These are some of the vital quality characteristics of a protection test:
- Test depth: This measurement indicates how thoroughly specific tests are executed, e.g., how many test shots are performed to derive the characteristic curve of a time overcurrent protection. In-depth testing positively impacts reliability but can negatively impact efficiency if overdone.
- Test repeatability: It lets you know if a test can easily be repeated with the exact same test values. This is a prerequisite for high-level testing quality and the basis for a structured analysis of protection devices’ misbehaviour. In addition, test results can easily be reproduced comprehensibly. When deriving mean values of trip times, relay times, etc., repeatability is also necessary.
- Test coverage: Test coverage indicates how many existing parameters are verified with the test setup in a settings-based testing approach. For a system-based approach, this characteristic indicates which parts of the protection system were included in the test. In general, the value for test coverage is higher for a system test, as it verifies if the entered parameters match the protection concept and if they are suitable to fulfill the desired behaviour.
- Test performance: If the testing time required for a specific test or sequence of tests can be reduced while covering the same scope, the value of the test performance increases. Short testing times create capacities to increase test depth, reduce costs or integrate a wider range of other work tasks. They may also allow to fulfill the tremendously increasing challenges and workloads brought about by the energy transition.
Other quality characteristics may also be relevant depending on the purpose of the test, and their importance can be assessed differently from company to company.
We’ve now discussed the essential aspects necessary for taking a closer look at key performance indicators and to answer the question: How can protection testing efficiency be measured?
Key Performance Indicators for Protection Testing
Many discussions about protection testing usually focus on the best way to perform tests and argue either for automated testing or manual approaches. However, all these arguments and claims only have limited value if not backed by data. Unfortunately, most of these discussions don’t cover how efficiency can be described or how it should be measured, let alone discuss any key performance indicators. Nonetheless, they are the relevant means for assessing different approaches.
According to Wikipedia article November 2023, https://en.wikipedia.org/wiki/Performance_indicator, “A performance indicator or key performance indicator (KPI) is a type of performance measurement. KPIs evaluate the success of an organization or a particular activity (such as projects, programs, products, and other initiatives) in which it engages.”
These KPIs must be identified and clearly defined before assessing a protection testing process. If suitable measures can’t be found to evaluate the overall process, subprocesses or workflows can be evaluated instead. Here are some dimensions that are usually a source for relevant KPIs:
- Quantity of tests (e.g. number of test shots)
- Quality of tests (e.g. test coverage, test depth)
- Duration (e.g. overall test time)
- Efficiency (e.g. time, costs)
- Costs
The test process must be analyzed to define the exact metrics and KPIs that suit a dedicated company. The following chapters will cover some examples of metrics and KPIs that could be used to evaluate particular aspects of a protection testing process:
- Test coverage
- Test repeatability
- Performance of the test execution
Test coverage
The test coverage for a test procedure can be used as a KPI as follows:
Obviously, the KPIcov = 0 is optimal, as zero is the highest reachable value for this KPI. This easy example shows that having clear references for assessing the measured KPI is essential.
Various steps can be implemented to reach the optimal KPI for the testing process, e.g.:
- Introducing test specifications for each test that must be fulfilled.
- Standardizing protection testing tasks by specifying the entire testing procedure to avoid missing steps.
- Analyzing the test specifications regularly to monitor the test coverage and minimize system errors.
Test repeatability
Another pillar of high quality in protection testing is the repeatability of the test procedure.
The KPI for repeatability can be defined in a similar way:
The optimum state for this KPI is also zero. If the KPI is greater than zero, there are two options for improving the performance of quality aspect repeatability:
- Using test plans to ensure the same test values are used every time the test is performed.
- Switching from manual testing to automatic testing.
KPI for test performance and cost efficiency
Measuring the test duration is one of the most significant KPIs directly related to costs. It can be measured easily and is highly relevant for almost all utilities, e.g., cost reduction targets usually drive process optimizations. In the example below, we assume that the quality of the work in the protection testing field and the utilization of the personnel is at a consistently high level (value >90%). That means the work is being done effectively – tasks are done correctly, and the quality is high. But is this process time efficient?
The following considerations focus on the time needed to fulfill all relevant test tasks. The KPI for testing process efficiency can be defined as:
Where:
- Tpreparation: overall preparation time
- Ttest: overall test time, including documentation
- Tmin: minimum possible time for preparation and test
Although the minimum possible time is unknown in this equation, we know that the optimal state of the KPIeff = 1. Therefore, this KPI cannot be used to measure the performance of a specific testing task, but it can be considered for measuring changes in the testing process. Let’s assume a utility performs manual testing and wants to change to an automated testing approach. The KPIs for these two testing methods are:
for manual testing and 
for automated testing. As Tmin is in both equations equal the quotient of the two KPIs describes the KPI for improving the testing process as a measured value. This KPI can be used as a basis for assessing actual cost reductions resulting from an increase in efficiency due to the testing time reduction.
To make this theoretical discussion more tangible, the last part of this article will focus on a practical example.
Efficiency Gains in Practice
What possibilities do efficiency gains and cost reductions offer? In the following example, we’ll take potential actions for turning efficiency gains into numbers. The time when actions are introduced is relevant to the overall benefit that can be generated. Therefore, in the example, we’ll look at the different phases of a new substation or the extension of an existing substation. In the following example, we will focus on testing protection relays, as shown in Figure 3.
Over the entire lifetime of a substation, the assets installed pass through various phases. The assets must be tested during each phase with a defined set of tests. Preparing and executing the individual tests can be done separately and manually, but that would be the least efficient approach (as seen in Figure 2 above). If test plans and automation are introduced instead of a manual approach, however, the time required is reduced drastically (as seen in Figure 4).
Preparation is usually the most time-intensive part of a protection test. It consumes about 50% to 60% of the entire test process. Whenever tests are performed for the first time, their preparation and design can consider future requirements. Each test step can be repeated without changing individual test modules. In Figure 5, this equals five-time units. Execution accounts for roughly one-time unit and documentation for another two-time units. For each following protection relay the preparation time is reduced to one time unit, which increases the efficiency substantially.
Test procedures stored in repeatable test plans allow tests to be performed one-to-one in later phases of an asset’s life cycle. At the same time, test coverage and depth can be increased substantially, while test repeatability can be easily pushed to an optimum level. This pays off once the assets enter their maintenance phases, and the same tests need to be performed regularly. Such an approach can reduce the needed time by up to 80%.
Let’s focus on another example. Figure 6 shows a section from an industrial power grid. It contains several motors which are similar in size. These motors are protected with the same protection scheme with similar protection settings. In most cases, the protection devices are from the same manufacturer, which means that the differences between the individual protection devices are minor. Differences may be found in specific parts of the protection scheme (e.g., pickup values, trip times).
On average, we have to deal with approximately 30 different parameters from one protection device to the next. Therefore, nearly every test case in the protection testing procedure must be recalculated and overworked for testing the next protection device. A procedure with 30 test cases is required for testing these 30 parameters and ten more essential ones (e.g., CT and VT settings) for a protection device.
Table 1 displays an average of 3 calculations and 12 altered parameters per test case that can be used to discuss the efficiency of protection testing.
Hence, preparing the 30 test cases from our example needs 90 calculations and 360 parameters, which must be changed during the testing of the protection devices used for every motor in Figure 6. If 20 similar protection devices in this substation need to be tested,1800 calculations (3 × 30 × 20) must be done, and 7200 parameters (12 × 30 × 20) must be changed manually.
For automated testing, two steps only need to be carried out once. In preparation, 90 calculations and the input of 360 parameters are required. During the testing of the 20 protective relays, a further 600 parameters (20 times the 30 variants) must be changed. For an easier calculation, let’s assume that every step requires the same amount of time. Using the formula for KPIimprov we get the following:
The more relays are tested, the higher the efficiency. With 30 similar relays the KPI already increases to 10.
In addition to substantial efficiency improvements, by factors 8,6, and 10 in our examples, two additional aspects must be highlighted. Firstly, the effort to prepare a manual test procedure is greater than the preparation time needed for automated test plans. Secondly, the manual testing approach increases the possibility of human error during preparation and execution dramatically. This is why improving efficiency with automation also enhances the quality of the protection testing process.
What does this mean for daily work?
New technologies in one business field lead to changes in other technologies and working processes. Nowadays, the pressure due to time and cost limitations is becoming more and more severe. On top of that, the energy transition is a catalyst for these two topics. In this context, validating technical working processes by analyzing their economic properties is critical.
This article has shown why KPIs are vital to the protection testing field and why they should be implemented if they’re not being used yet. They help to improve and optimize working processes by making them measurable.
When using a KPI to compare a manual testing approach to an automated one, its time-saving efficiency shows that state-of-the-art testing methods can leverage significant improvements. Measurable improvements by, e.g., a factor of 10, is more than enough reason to examine your own processes more closely.
Michael Albert studied general electrical engineering at Saarland University. After his studies, he worked in power engineering, focusing on protection technology. He’s worked for OMICRON since 2005 as a Product Manager and in the engineering service field.
