With a market size of over $35 billion in 2024, milk powder plays an essential role in the food sector, particularly in the manufacturing of infant formulas, because of its combination of high nutritional value and stability over time. The low water content in milk powder and derived products makes contamination by pathogenic microorganisms more difficult, but risks continue to exist from certain resistant pathogens [1]. One such genus is Cronobacter, of the Enterobacteriaceae family, a group of Gram-negative and oxidase-negative bacteria that are rod-shaped and facultatively anaerobic [2]. While Cronobacter rarely cause disease in adults, infants are extremely vulnerable; one eight-year study reported a 42% mortality rate for infants infected with Cronobacter spp. meningitis. Additional studies have documented a clear link between the numerous outbreaks that have been reported globally since 1960 and consumption of powdered infant formula [3], while a 2014 assessment of American milk powder facilities determined that Cronobacter was present in the manufacturing areas of 69% of the 55 facilities tested [4].

The other pathogen most frequently responsible for outbreaks associated with milk powder consumption is Salmonella, also of the Enterobacteriaceae family, and similarly capable of surviving for extended periods of time in foods with low moisture levels. While contamination of milk powder by Salmonella is less prevalent than by Cronobacter, there are significant risks as Salmonella is considered one of the pathogens inducing the highest rates of serious illness and hospitalization. Beginning in late 2017, France experienced a major Salmonella Agona outbreak that led to over 30 cases under six months old from consumption of infant milk products, including 18 hospitalizations, and similar outbreaks have been reported across the world with substantial economic and health consequences [5].

It is therefore essential for laboratories to detect bacterial contamination of milk powder by these two pathogens. The ISO 22964 and ISO 6579-1 standards describe, respectively, reference methods for the detection of Cronobacter spp. and Salmonella spp. in food products, and a number of alternative methods have been validated according to ISO 16140-2 [6–8]. Quality control of all of these methods is critical to ensure consumer safety and confidence, and the ISO 17025 standard [9] indicates that laboratories must demonstrate the validity of their results through external controls, including interlaboratory proficiency tests (PTs). To this end, BIPEA developed and implemented new PTs in February 2019 dedicated, initially, to the detection of Cronobacter in milk powder; Salmonella detection in the same samples was added to the tests in June 2023 to address laboratory demand.

Participation in these proficiency tests allows laboratories to detect and correct analytical problems, to demonstrate their performance for these analyses, and to compare results obtained by different protocols for the detection of these pathogens under operating conditions using real matrices.

Proficiency tests involve the analysis by different laboratories of the same analytical parameters on identical samples. The implementation of a PT can be summarized in three principal steps: preparation of homogeneous and stable samples, analysis by participating laboratories, and statistical treatment of the data, which includes determination of assigned values and evaluation of laboratory performances.

The first trial of the PT for Cronobacter detection in milk powder, in 2019, gathered nine participants, and participation has steadily increased in the intervening years (Fig. 1) to reflect increased interest in the detection of these pathogens in food products, particularly milk powder infant formulas. Regular commission meetings are organized by BIPEA to allow participants to discuss past results and potential technical evolutions for the program.

Figure 1 Evolution of participation since implementation of the proficiency testing program, divided by geographic region

For proficiency tests to be effective, it is crucial that homogeneous and stable samples be produced. For this PT, a batch of milk powder is first analyzed to detect the possible presence of pathogens, before being contaminated with calibrated suspensions of Cronobacter sakazakii and/or Salmonella Enteritidis, with target final concentrations around 10² CFU (colony-forming units) per sample for each microorganism. The batch is then homogenized and divided into a series of samples.

To demonstrate stability, three samples were stored at room temperature and analyzed for the two target microorganisms after zero, four, eleven and fourteen days; detection of both Cronobacter and Salmonella in each of these analyses confirmed the stability of the samples for the course of the test period. Moreover, for positive samples, the homogeneity of the batch is additionally verified by an experimental study on ten samples, taken randomly across the batch and analyzed in random order, according to the requirements of ISO 13528 [10]. Detection analyses of Cronobacter and Salmonella are conducted using the reference methods ISO 22964 and ISO 6579-1 respectively, and the set of samples is considered homogeneous if the microorganisms are detected in all analyzed samples.

Three samples are then shipped to each participating laboratory at room temperature.

Laboratories can perform the analyses using either the reference methods or alternative methods, and are also asked to indicate the date of analysis. They then submit their results to BIPEA via online reply forms.

Considering the unstable nature of the samples, participants are requested to analyze the samples as soon as possible after reception, with a limit of three weeks from the shipment date.

The results from these tests are qualitative (detected/not detected), and are therefore evaluated as follows:

In addition, BIPEA has chosen to present an overall assessment of each laboratory’s ability to correctly identify negative and positive samples by calculating relative specificity (r_SP), relative sensitivity (r_SE), and relative accuracy (r_AC), defined as follows:

For this PT, each laboratory’s overall performance is considered acceptable if their relative specificity and relative sensitivity are 100%, and therefore if their relative accuracy is 100% as well.

Since January 2020, BIPEA has organized two regular proficiency tests per year for these analyses. The results of the four most recent tests, since Salmonella was added to the samples, are summarized in Table 1 (Cronobacter detection) and Table 2 (Salmonella detection).

For both pathogens, results are generally very satisfactory, regardless of the geographical origin of the laboratories. For 16 of the 24 sets of samples studied here, all laboratories concluded correctly, and for the remaining samples only between 7 and 15% of laboratories reported false negatives or false positives. Approximately 75% of laboratories used the reference method ISO 22964 [6] for Cronobacter detection and approximately 50% of laboratories used the reference method ISO 6579-1 [7] for Salmonella detection; no noticeable effect of the method utilized can be observed on the results. In addition, performance in these tests has remained relatively stable over time, demonstrating that most laboratories master these detection analyses. It is also important to note that the rates of false positives are either similar, in the case of Salmonella, or greater, in the case of Cronobacter, than the rates of false negatives. This is reassuring, as the consequences of false positives are primarily economic, such as unnecessary product recalls, while false negatives can lead to outbreaks and have serious impacts on public health, including the death of contaminated persons.

In recent years, several propositions have been published for numerical scoring systems, designed to allow for easy interpretation of participant performances, for qualitative proficiency testing data. The objective of these systems, which include the L-score [11], the a-score [12], and the S-score [13], is to mimic the widely accepted z-score used for quantitative data and give participants a simpler way to evaluate their results.

Each of these systems is a useful contribution to the assessment of qualitative PT data, making it easier to compare between tests and examine laboratory performance over time. However, each also has certain limitations. The L-score requires at least 10 participants, five different parameters where failure has been recorded, and specific statistical modeling software; in addition, it is fundamentally a relative evaluation rather than an absolute one, as the most satisfactory scores are impossible for a laboratory to achieve unless other laboratories perform poorly. For this reason, identical results can be judged differently on different tests, making continued assessment over time complicated. The a-score remedies several of these difficulties but needs a minimum of 20 participants to be implemented. The S-score removes this barrier but uses a more complex system that requires PT providers to define a priori the difficulty of each analysis, which leads to numerical scores with less transparent interpretations when compared with the simplicity of the z-score. Replicates are also necessary in some cases.

BIPEA applies the specificity and sensitivity indicated in ISO 22117 and considers the use of relative specificity, sensitivity, and accuracy to be the system best adapted to evaluating laboratory performance for its qualitative proficiency tests [14]. The relative accuracy is an easy-to-interpret assessment of the overall ability of the laboratory to complete the analyses studied, while relative specificity and sensitivity allow the essential distinction to be made between difficulty detecting positive samples and incorrectly identifying negative samples, which are errors necessitating significantly different corrective actions—just as large positive and large negative z-scores clearly indicate different kinds of analytical problems. In addition, if a laboratory participates in multiple rounds of such a PT, it is simple enough to monitor evolution in performance by graphing relative accuracy against time, as described by Chabirand et al. [15].

By examining in detail the results of the four trials previously presented for the detection of Cronobacter (Table 3) and Salmonella (Table 4) in milk powder, each laboratory’s global performance can be evaluated using these assessment parameters. For each pathogen, all but three laboratories achieved relative accuracy scores of 100%, and therefore 100% relative specificity and sensitivity as well. The overall performance on these tests can thus be considered highly satisfactory. Participants in this program are provided with their relative specificity, sensitivity, and accuracy for each trial for which they submit results, and can easily calculate these three scores over a period of multiple trials if desired, as demonstrated here.

If one of the primary goals of proficiency testing is to enable laboratories to demonstrate their competence for given analyses, there is a final factor to be considered. While it is clear that a laboratory with 100% relative accuracy has demonstrated greater ability than one with 33% relative accuracy, and that a laboratory that consistently achieves scores of 100% masters the analyses to a greater extent than one that oscillates between scores of 100% and 50%, frequency of participation must also be taken into account. For example, by studying multiple trials collectively as in Tables 3 and 4, it is possible to observe for each participant not only the rates of relative specificity, sensitivity, and accuracy, but also a final rate, the rate of participation (r_P), which corresponds to the number of samples analyzed divided by the total number of samples proposed over a given time period. For an analytical laboratory, achieving a relative accuracy of 100% while participating in all four trials can be a way to signal greater expertise than obtaining the same rate while participating in a single test, and such performance can be extremely valuable for earning and maintaining consumer trust.

Outbreaks involving Cronobacter and Salmonella in milk powder or infant formulas have been reported in many countries and can have enormous consequences, including serious illness, hospitalization, and death, and the global nature of food supply chains means that contamination in one facility can lead to cases across the world. The health risks associated with non-detection of contaminated samples of milk powder are particularly high because the principal consumers are infants, who have underdeveloped immune systems. Furthermore, the rates of reported Cronobacter infections in infants have risen significantly in recent years.

The interlaboratory proficiency tests presented here have been developed to provide laboratories with a quality tool to assess their ability to detect these pathogens in milk powder. The PTs are offered regularly, with two rounds per year, and the performances of participating laboratories are highly satisfactory, with relative accuracy equal to 100% for both pathogens for most laboratories. By participating in proficiency testing, laboratories can verify the reliability and stability of their results, as well as obtain recognition of their analytical procedures by customers and accreditation bodies according to ISO 17025. These initial results are encouraging and reassuring for consumers and public organizations, as an indicator that laboratories master these essential detection analyses.