Evaluation of the Inter-laboratory Comparison for Arsenic Analysis

(Prepared by Dr. M. Khaliquzzaman)

 

 

Abstract

The results reported by the laboratories taking part in the recent inter-comparison exercise conducted by IAEA/WHO have been analyzed using a judgmental scoring system. The performances of the laboratories have been graded in a five point (star) scale. None of the laboratories obtained the five-star grade in this grading, two laboratories obtained 4-stars, five obtained 3-stars, two 2-stars and two obtained one star grades. Six laboratories have been classified as not yet ready for service analysis. Considerable input in the form of human resources and skills will be needed to improve the performance of these laboratories. Comments on individual laboratories have also been provided. Considering the fact this is the first such exercise, the results can be considered very hopeful. The laboratories should carry out internal audit to evaluate their analytical procedures and identify actions required to improve their performance. More such inter-comparisons should be carried out to evaluate the effect of the reviews and subsequent actions taken on the performance of the laboratories.

1. Introduction

Arsenic contamination in ground water in Bangladesh is major public health issue now. This issue has given rise to the analytical challenge of quality assured determination of Arsenic in the parts per billion (ppb) level in different kinds of samples. To meet this challenge laboratories have been set up in the Government, public and private sectors. Until, this intercomparison exercise for water samples no systematic attempt to validate the results from these laboratories was attempted. The intercomparison report provides a descriptive presentation of results obtained in the exercise. This evaluation tries to look at the results in a more quantitative fashion using scores. As many of the participating laboratories are new, the score estimations have been explained in plain language for the benefit of both the technical and management personnel of the analytical laboratories. Analysis of the results of individual laboratories along with recommendations are given in the annex to help the laboratories to improve their performance.

2. Estimation of uncertainties

Estimates of uncertainties of analytical data form a single laboratory represent the precision of the measurements in that laboratory. Such estimates of uncertainties do not indicate the measure of accuracy ( i.e., deviation from actual values). In intercomparison measurements data for the analytical parameters are obtained from several laboratories with varying estimates of uncertainties. In such cases standard deviation of data for a given parameter can give better estimates of uncertainty in the data.

Table: 1 shows the data from three reputable reference laboratories for the 5 samples with de-ionized water matrix (DI 1-5). It can be seen that the standard deviations are found to more than the values reported by at least one laboratories in all the cases. So, as the measure of uncertainties for score estimates discussed latter, the values of standard deviations have been accepted as the base values. However, considering the fact that all three laboratories use ICP-MS/ ICP-OES systems which are state of the art best equipment available and the performance of such laboratories are unlikely to be matched by the laboratories participating in the exercise using various assorted kinds of equipment, it is desirable that some judgmental estimates for the values of uncertainties larger than the ones obtained in the reference laboratories should be used. It appears that twice the value of standard deviations of the reference laboratories should be reasonable.

However in the case of samples with simulated ground water matrix and well water samples measurements from a single reference laboratory are available. So, twice the actual reported values were used in these cases. These values are shown in table:2. The well water samples (PW-1&5) contained too much chlorine and actual uncertainty estimates for the sample were estimated to be about 10% rather than smaller instrumental values shown in table:2. This happened due to interference from ArCl ions which have the same mass(75) as the As atoms. These higher values of uncertainty estimates were accepted for score calculation.

 

Table: 1 Concentration of As in five reference samples with de-ionized water matrix as made and measured by three reference laboratories. ( All concentrations are in m g/litre)

 

Made

Monaco

Sieversdorf

Rochester

Average

Sl.

Sample

Conc.

Unc.

Conc.

Unc.

Conc.

Unc.

Conc.

Unc.

Value

Std.

1

DI-1

0

0

2

DI-2

17.7

0.1

18

0.9

20.5

0.1

16.1

0.8

18.2

2.21

3

DI-3

63.3

0.5

64.5

2.9

66.5

0.5

60.2

3

63.7

3.22

4

DI-4

101.3

0.8

102

7

102

1

96.1

5

100.0

3.41

5

DI-5

506.4

4.1

507

33

511

4

495

25

504.3

8.33

Table: 2 Concentration of As in five reference samples with simulated water matrix as made and measured by three reference laboratories and two well water measured at a reference laboratory. ( All concentrations are in m g/litre)

Sl.

NIST

Monaco

Sl.

Rochester

Sample

Conc.

Unc.

Conc.

Unc.

Sample

Conc

Unc.

6

G-1

0

11

PW-1

67.7

1.3

7

G-2

22.5

0.2

23.4

1.9

12

PW-5

60.9

1.5

8

G-3

56.3

0.5

58.4

4.7

9

G-4

90.1

0.7

90

7

10

G-5

450.5

3.6

444

36

 

3. Score analysis

Results from independent measurement of an analytical parameter are expected to be normally distributed. The standard deviation of the distribution is determined by the uncertainties of the measurements. In a case where there are more that one source of uncertainties, the rms combination of all the uncertainties determine the value of standard deviation. A simple estimate of the goodness of a measurement can be obtained by calculating the score (Z) which is defined here as:

Z= ½ ( Known value- Measures value) ½ / Standard Deviation

 

This definition is similar to the one usually used in standard statistical usage. The known value here correspond to the mean of the standard distribution, measured value correspond to the value of the distribution variable and standard deviation here is derived from judgmental consideration as discussed earlier. The absolute value in the numerator ensures that all the values of the scores calculated are positive.

A score value essentially determines how far a measured value is away from the know value in units of the standard deviation. A score value of 1 for a measurement means that it falls within the central 68% area of the probability curve, a value of 2 means that it falls within the 95.5% of central area of the probability curve and a value of 3 correspond to 99.7% of the area. Consequently, a score value larger than 3 correspond to a very unlikely happening, thus measurements with such scores are considered out-liers. As the scores are pure numbers, these can be subjected to normal numerical operations as long as they refer to the same variable. Hence, scores have been calculated separately for three groups of samples. An average score for each laboratory is also given but as three different kinds of measurements have been averaged, it can not be taken as indicator for the performance of a laboratory. The results of the score calculations are given in table:3 along with the ratings. The rating criteria are given at the end of the table.

 

Table:3 Laboratory Scores based on Intercomparison results

Lab.

Score in different categories

Rating

Comments

DI

G

PW

Average

Lab-1

1.61

1.68

2.15

1.82

***

OK

Lab-2

0.73

0.56

1.12

0.80

****

OK

Lab-3

6.89

3.23

3.08

4.40

-

Not yet ready for service measurements

Lab-4

6.68

3.30

3.02

4.33

-

Not yet ready for service measurements

Lab-5

16.64

11.26

24.16

17.35

-

Not yet ready for service measurements

Lab-6

0.43

0.14

1.50

0.69

****

OK

Lab-7

4.78

3.80

0.72

3.10

*

The sample results are OK

Lab-8

2.87

2.65

3.54

3.02

**

Need to improve sample results

Lab-9

4.62

3.48

4.97

4.36

-

Not yet ready for service measurements

Lab-10

1.36

1.99

0.99

1.45

***

OK

Lab-11

0.92

1.56

2.98

1.82

***

OK

Lab-12

1.96

2.83

3.88

2.89

**

Need to improve sample results

Lab-13

5.14

1.41

0.73

3.52

***

The sample results are OK

Lab-14

5.79

4.81

3.07

4.55

-

Not yet ready for service measurements

Lab-15

14.75

6.12

4.98

8.62

-

Not yet ready for service measurements

Lab-16

1.49

-

*

The sample results are OK

Lab-17

1.60

0.93

1.27

1.27

***

OK

***** Score below 1 on all 3 categories

**** Score below 1 on 2 categories

*** Score below 2 on 2 categories

** Score below 3 on 2 categories

* Score below 3 on 1 category

4. Discussion and conclusion

The results reported by the laboratories taking part in the recent inter-comparison exercise conducted by IAEA/WHO have been analyzed using a judgmental scoring system. The performances of the laboratories have been graded in a five point (star) scale. None of the laboratories obtained the five-star grade in this grading, two laboratories obtained 4-stars, five obtained 3-stars, two 2-stars and two obtained one star grades. Six laboratories have been classified as not yet ready for service analysis. Considerable input in the form of human resources and skills will be needed to improve the performance of these laboratories. Comments on individual laboratories have also been provided in the annex. Considering the fact this is the first such exercise, the results can be considered very hopeful. The laboratories should carry out internal audit to evaluate their analytical procedures and identify actions required to improve their performance. More such inter-comparisons should be carried out to evaluate the effect of the reviews and subsequent actions taken on the performance of the laboratories. Continued intercomparison exercises are needed to maintain the analytical capabilities of the laboratories and such exercises are performed on regular basis in the advanced countries and by international agencies. National mechanism should be built in Bangladesh to carry out similar intercomparison exercises locally, regionally and internationally.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Annex

 

Performance of Individual Laboratories

 

Table: A1 Individual Laboratory Performance ( Laboratory – 1 , Method – AAS-HG)

( All concentrations are in m g/litre)

Reference Lab

Lab.- 1

Comments

Sl.

Sample

Conc.

Std.

Conc.

Std.

Score

Avg. Gr.

Score

Rating

1

DI-1

0

-

11

-

-

1.61

***

  1. Good performance

2. Operators should be careful when high standard deviations are obtained (c.f.,G3,G5)

3. Actual sample values are on the high side.

4. Should try to improve the results further and aim score less than 1 on all categories.

2

DI-2

17.7

4.41

29.3

4.7

2.63

3

DI-3

63.3

6.44

69.1

12.6

0.90

4

DI-4

101.3

6.81

-

-

-

5

DI-5

506.4

16.65

528.1

32.3

1.30

6

G-1

0

-

-

-

-

1.68

7

G-2

22.5

3.80

26.4

9.9

1.03

8

G-3

56.3

9.40

84.7

20.4

3.02

9

G-4

90.1

14.00

108.1

6.8

1.29

10

G-5

450.5

72.00

551.6

93.8

1.40

11

PW-1

67.7

13.60

95.1

14.7

2.01

2.15

12

PW-5

60.9

12.20

88.9

6.4

2.30

***** Score below 1 on all 3 categories

**** Score below 1 on 2 categories

*** Score below 2 on 2 categories

** Score below 3 on 2 categories

    1. Score below 3 on 1 category

 

 

 

Table:A2 Individual Laboratory Performance ( Laboratory – 2 , Method – SP )

( All concentrations are in m g/litre)

Reference Lab

Lab.- 2

Comments

Sl.

Sample

Conc.

Std.

Conc.

Std.

Score

Avg. Gr.

Score

Rating

1

DI-1

0

0.00

<10

-

-

0.73

****

  1. Good performance

2. Low concentration samples have not been measured. So, low scores may be artificial.

3. Although results are good, the operator should be careful with known interference for SP.

4. Should try to improve the results further and aim score less than 1 on all categories.

2

DI-2

17.7

4.41

-

-

-

3

DI-3

63.3

6.44

-

-

-

4

DI-4

101.3

6.81

96

6

0.78

5

DI-5

506.4

16.65

495

52

0.68

6

G-1

0

-

<10

-

-

0.56

7

G-2

22.5

3.80

-

-

-

8

G-3

56.3

9.40

55

10

0.14

9

G-4

90.1

14.00

-

-

10

G-5

450.5

72.00

380

56

0.98

11

PW-1

67.7

13.60

79

10

0.83

1.12

12

PW-5

60.9

12.20

78

10

1.40

***** Score below 1 on all 3 categories

**** Score below 1 on 2 categories

*** Score below 2 on 2 categories

** Score below 3 on 2 categories

* Score below 3 on 1 category

Table: A3 Individual Laboratory Performance ( Laboratory – 3 , Method – ???)

( All concentrations are in m g/litre)

Reference Lab

Lab.- 3

Comments

Sl.

Sample

Conc.

Std.

Conc.

Std.

Score

Avg. Gr.

Score

Rating

1

DI-1

0

0.00

4

-

-

6.89

None

1. Operator training needed

2. Problem with concept of

uncertainty estimation

3. Problem with supervision

4. Serious calibration

problem. All values

systematically high.

5. The Laboratory is not yet

ready for service

measurements

2

DI-2

17.7

4.41

47

-

6.64

3

DI-3

63.3

6.44

100

-

5.70

4

DI-4

101.3

6.81

158

-

8.32

5

DI-5

506.4

16.65

-

-

-

6

G-1

0

7

-

-

3.23

7

G-2

22.5

3.80

32

-

2.50

8

G-3

56.3

9.40

93

-

3.90

9

G-4

90.1

14.00

136

-

3.28

10

G-5

450.5

72.00

-

-

-

11

PW-1

67.7

13.60

108

-

2.96

3.08

12

PW-5

60.9

12.20

100

-

3.20

***** Score below 1 on all 3 categories

**** Score below 1 on 2 categories

*** Score below 2 on 2 categories

** Score below 3 on 2 categories

* Score below 3 on 1 category

 

Table:A4 Individual Laboratory Performance ( Laboratory – 4 , Method – AAS-GF)

( All concentrations are in m g/litre)

Reference Lab

Lab.- 4

Comments

Sl.

Sample

Conc.

Std.

Conc.

Std.

Score

Avg.

Score

Rating

1

DI-1

0

0.00

4.91

1.85

-

6.68

None

1. Operator training needed

2. Problem with concept of uncertainty estimation. Values reported too low.

3. Problem with supervision

4. Serious calibration/ instrument problem

5. The Laboratory is not yet ready for service measurements

2

DI-2

17.7

4.41

-

-

-

3

DI-3

63.3

6.44

33.66

2.36

4.60

4

DI-4

101.3

6.81

41.64

6.18

8.76

5

DI-5

506.4

16.65

-

-

-

6

G-1

0

15.48

2.46

-

3.30

7

G-2

22.5

3.80

16.97

1.94

1.46

8

G-3

56.3

9.40

21.19

3.23

3.74

9

G-4

90.1

14.00

37.422

4.8

3.76

10

G-5

450.5

72.00

144.53

18.61

4.25

11

PW-1

67.7

13.60

32.75

2.19

2.57

3.02

12

PW-5

60.9

12.20

18.54

2.71

3.47

***** Score below 1 on all 3 categories

**** Score below 1 on 2 categories

*** Score below 2 on 2 categories

** Score below 3 on 2 categories

* Score below 3 on 1 category

Table:A5 Individual Laboratory Performance ( Laboratory – 5 , Method – AAS-HG)

 

Reference Lab

Lab.- 5

Comments

Sl.

Sample

Conc.

Std.

Conc.

Std.

Score

Avg. Gr.

Score

Rating

1

DI-1

0

0.00

4.18

-

-

16.64

None

1. Operator training needed

2. Problem with concept of uncertainty

3. Problem with supervision

4. Serious calibration/ instrument problem

5. The Laboratory is not yet ready for service measurements

2

DI-2

17.7

4.41

65.9

-

10.92

3

DI-3

63.3

6.44

79.01

-

2.44

4

DI-4

101.3

6.81

350.3

-

36.55

5

DI-5

506.4

16.65

-

-

6

G-1

0

-

3.42

-

11.26

7

G-2

22.5

3.80

79.3

-

14.95

8

G-3

56.3

9.40

-

-

9

G-4

90.1

14.00

175.7

-

6.11

10

G-5

450.5

72.00

556.1

-

1.47

11

PW-1

67.7

13.60

395.6

-

24.11

24.16

12

PW-5

60.9

12.20

356.2

-

24.20

 

 

***** Score below 1 on all 3 categories

**** Score below 1 on 2 categories

*** Score below 2 on 2 categories

** Score below 3 on 2 categories

    1. Score below 3 on 1 category

      Table:A6 Individual Laboratory Performance ( Laboratory – 6 , Method – AAS-HG)

      ( All concentrations are in m g/litre)

    Reference Lab

    Lab.- 6

    Comments

    Sl.

    Sample

    Conc.

    Std.

    Conc.

    Std.

    Score

    Avg. Gr.

    Score

    Rating

    1

    DI-1

    0

    0.00

    bdl

    -

    -

    0.43

    ****

    1. Good performance

    2. Sample values are on the high side.

    3. Should try to improve the results further and aim at score less than 1 on all categories.

    2

    DI-2

    17.7

    4.41

    18.2

    3

    0.11

    3

    DI-3

    63.3

    6.44

    60.9

    2.2

    0.37

    4

    DI-4

    101.3

    6.81

    103.1

    12.3

    0.26

    5

    DI-5

    506.4

    16.65

    490.1

    30.1

    0.98

    6

    G-1

    0

    7

    G-2

    22.5

    3.80

    21.9

    2.1

    0.16

    0.14

    8

    G-3

    56.3

    9.40

    55.4

    7.6

    0.10

    9

    G-4

    90.1

    14.00

    -

    -

    -

    10

    G-5

    450.5

    72.00

    462.2

    35.9

    0.16

    11

    PW-1

    67.7

    13.60

    96

    2.08

    1.50

    12

    PW-5

    60.9

    12.20

    72

    0.91

     

    ***** Score below 1 on all 3 categories

    **** Score below 1 on 2 categories

    *** Score below 2 on 2 categories

    ** Score below 3 on 2 categories

  1. Score below 3 on 1 category

    Table:A7 Individual Laboratory Performance ( Laboratory – 7 , Method – AAS-HG)

    ( All concentrations are in m g/litre)

    Reference Lab

    Lab.- 7

    Comments

    Sl.

    Sample

    Conc.

    Std.

    Conc.

    Std.

    Score

    Avg. Gr.

    Score

    Rating

    1

    DI-1

    0

    0.00

    4.78

    *

    1. Operator training needed

    2. Calibration problem at high concentrations

    3. Problem with the standard deviation. Values reported are too low.

    4. Problem with supervision

    5. Sample values are good (PW-1&5)

    6. Should try to improve the results and aim at score less than 1 on all categories.

    2

    DI-2

    17.7

    4.41

    5.9

    0.7

    2.67

    3

    DI-3

    63.3

    6.44

    88.9

    2.2

    3.98

    4

    DI-4

    101.3

    6.81

    103.4

    4

    0.31

    5

    DI-5

    506.4

    16.65

    303.8

    4.9

    12.17

    6

    G-1

    0

    bdl

    3.80

    7

    G-2

    22.5

    3.80

    49.8

    2.9

    7.18

    8

    G-3

    56.3

    9.40

    9

    G-4

    90.1

    14.00

    123.9

    12.3

    2.41

    10

    G-5

    450.5

    72.00

    321.1

    9.3

    1.80

    11

    PW-1

    67.7

    13.60

    63.1

    7.2

    0.34

    0.72

    12

    PW-5

    60.9

    12.20

    74.3

    2.7

    1.10

     

    ***** Score below 1 on all 3 categories

    **** Score below 1 on 2 categories

    *** Score below 2 on 2 categories

    ** Score below 3 on 2 categories

  2. Score below 3 on 1 category

    Table:A8 Individual Laboratory Performance ( Laboratory – 8 , Method – TXRF)

    ( All concentrations are in m g/litre)

    Reference Lab

    Lab.- 8

    Comments

    Sl.

    Sample

    Conc.

    Std.

    Conc.

    Std.

    Score

    Avg.

    Score

    Rating

    1

    DI-1

    0

    0.00

    0

    0

    -

    2.87

    **

    1. Problem with sample values (PW-1&5)

    2. TXRF method uses only small sample volumes. Sample non-uniformity due to micro-precipitation should be carefully looked into.

    3. Should try to improve the results and aim at score less than 1 on all categories.

    2

    DI-2

    17.7

    4.41

    25

    1.7

    1.65

    3

    DI-3

    63.3

    6.44

    -

    -

    -

    4

    DI-4

    101.3

    6.81

    84.7

    7.4

    2.44

    5

    DI-5

    506.4

    16.65

    431

    23.9

    4.53

    6

    G-1

    0

    -

    -

    -

    -

    2.65

    7

    G-2

    22.5

    3.80

    49.3

    5.7

    7.05

    8

    G-3

    56.3

    9.40

    85.7

    9.8

    3.13

    9

    G-4

    90.1

    14.00

    85

    4

    0.36

    10

    G-5

    450.5

    72.00

    445.7

    7.6

    0.07

    11

    PW-1

    67.7

    13.60

    20.3

    4

    3.49

    3.54

    12

    PW-5

    60.9

    12.20

    17

    2

    3.60

     

    ***** Score below 1 on all 3 categories

    **** Score below 1 on 2 categories

    *** Score below 2 on 2 categories

    ** Score below 3 on 2 categories

  3. Score below 3 on 1 category

    Table:A9 Individual Laboratory Performance ( Laboratory – 9 , Method – SP)

    ( All concentrations are in m g/litre)

    Reference Lab

    Lab.- 9

    Comments

    Sl.

    Sample

    Conc.

    Std.

    Conc.

    Std.

    Score

    Avg.

    Score

    Rating

    1

    DI-1

    0

    -

    -

    -

    4.62

    None

    1. Operator training needed

    2. Problem with concept of uncertainty estimates

    3. Problem with supervision

    4. Intermittent calibration/ instrument problem

    5. The Laboratory is not yet ready for service measurements

    2

    DI-2

    17.7

    4.41

    -

    -

    3

    DI-3

    63.3

    6.44

    37.4

    -

    4.02

    4

    DI-4

    101.3

    6.81

    112

    -

    1.57

    5

    DI-5

    506.4

    16.65

    368.58

    -

    8.28

    6

    G-1

    0

    -

    25.5

    -

    -

    3.48

    7

    G-2

    22.5

    3.80

    50

    -

    7.24

    8

    G-3

    56.3

    9.40

    78.45

    -

    2.36

    9

    G-4

    90.1

    14.00

    144

    -

    3.85

    10

    G-5

    450.5

    72.00

    485.9

    -

    0.49

    11

    PW-1

    67.7

    13.60

    93

    -

    1.86

    4.97

    12

    PW-5

    60.9

    12.20

    159.4

    -

    8.07

     

    ***** Score below 1 on all 3 categories

    **** Score below 1 on 2 categories

    *** Score below 2 on 2 categories

    ** Score below 3 on 2 categories

  4. Score below 3 on 1 category

    Table:A10 Individual Laboratory Performance ( Laboratory – 10 , Method – SP)

    ( All concentrations are in m g/litre)

    Reference Lab

    Lab.- 10

    Comments

    Sl.

    Sample

    Conc.

    Std.

    Conc.

    Std.

    Score

    Avg.

    Score

    Rating

    1

    DI-1

    0

    0.00

    <10

    -

    -

    1.36

    ***

    1. Operator training needed

    2. Problem with concept of uncertainty

    3. Performance is good judged on the basis of results

    4. Although results are good, the operator should be careful with known interference for SP.

    5. Should try to improve the results further and aim score less than 1 on all categories.

    2

    DI-2

    17.7

    4.41

    -

    -

    3

    DI-3

    63.3

    6.44

    64

    -

    0.11

    4

    DI-4

    101.3

    6.81

    90

    -

    1.66

    5

    DI-5

    506.4

    16.65

    545

    -

    2.32

    6

    G-1

    0

    -

    <10

    -

    1.99

    7

    G-2

    22.5

    3.80

    40

    -

    4.61

    8

    G-3

    56.3

    9.40

    68

    -

    1.24

    9

    G-4

    90.1

    14.00

    -

    10

    G-5

    450.5

    72.00

    460

    -

    0.13

    11

    PW-1

    67.7

    13.60

    62

    -

    0.42

    0.99

    12

    PW-5

    60.9

    12.20

    80

    -

    1.57

     

    ***** Score below 1 on all 3 categories

    **** Score below 1 on 2 categories

    *** Score below 2 on 2 categories

    ** Score below 3 on 2 categories

  5. Score below 3 on 1 category

    Table: A11 Individual Laboratory Performance ( Laboratory – 11 , Method – ASV)

    ( All concentrations are in m g/litre)