diff --git a/IRIS_PERFORMANCE_CARD.md b/IRIS_PERFORMANCE_CARD.md new file mode 100644 index 0000000..a38e299 --- /dev/null +++ b/IRIS_PERFORMANCE_CARD.md @@ -0,0 +1,58 @@ +# IRIS Performance Card + +## ICE 2005 dataset + +ICE 2005 dataset is used by the National Institute of Standards and Technology (NIST) in Iris Challenge Evaluation (ICE) 2005 [1] and was made publicly available as part of the ND-Iris-0405 dataset from the University of Notre Dame du Lac [2]. This dataset was collected from 132 subjects using an LG EOU 2200 scanner with image resolution 640x480. + +## Comparison with ICE 2005 participants (commercial and academia) + +A list of public and private entities participated in the ICE 2005 is shown below. The full evaluation report can be found in [1]. + +ICE 2005 results + +Fig. 1(a) and 1(b) show the ROC on right and left eye, respectively, from all participants in Iris Challenge Evaluation (ICE) 2005 and Fig. 2(a) and 2(b) show our ROC in comparison. Although the exact error rates were not published in [1], these plots show IRIS performs as good as if not better than commercial algorithms available at the time. + +|1a Fig. 1(a) participant ROC on ICE 2005 (right eye) [1]| 1b Fig. 1(b) participant ROC on ICE 2005 (left eye) [1]| +|---|---| +| 2a Fig. 2(a) IRIS ROC on ICE 2005 (right eye)| 2b Fig. 2(b) IRIS ROC on ICE 2005 (left eye)| + + +## Comparison with other open source algorithms + +We also compare our performance with the latest available iris open source algorithm OSIRIS, which has three versions OSIRISV2, OSIRISV4, OSIRISV4.1 [3]. As we can see in the table below, IRIS achieves significantly lower False NonMatch Rate (FNMR) at False Match Rate (FMR) of 0.001 and 0.0001 than OSIRIS. + +| | OSIRISV2 | OSIRISV3 | OSIRISV4 | IRIS | +|-------------------|----------|----------|----------|-------| +| FNMR @ FMR=0.001 | 0.174 | 0.031 | 0.019 | 0.003 | +| FNMR @ FMR=0.0001 | 0.268 | 0.058 | 0.034 | 0.006 | + +## ND-LG4000 dataset + +ND-LG4000 dataset is part of the ND-CrossSensor-Iris-2013 public dataset from the University of Notre Dame [4]. This dataset was collected from 676 subjects using LG IrisAccess 4000 scanner with image resolution 640x480. We randomly chose 541 subjects for training, 67 subjects for validating the segmentation model in IRIS. The remaining 2187 images from 68 subjects were used for testing, resulting in a total of ~2.4 million matches. + +|3a Fig. 3(a) IRIS ROC on ND-LG4000| 3b Fig. 3(b) IRIS DET on ND-LG4000| +|---|---| + +Fig. 3(a) shows the ROC for ND-LG4000 dataset, where Fig. 3(b) shows the DET in log-scale for the same dataset. Note that both plots are mathematically equivalent, where ROC shows the verification rate at a given FMR, while DET shows in log-scale the FNMR, or (1.0 - verification rate), at a given FMR. + + +## WLD-InHouse-v2 dataset + +**_Disclaimer: This data was not collected from Worldcoin users during field operations, but stems primarily from paid participants in a dedicated workstream, separately administered by a respected partner._** + +The WLD-InHouse-v2 dataset was internally collected to simulate more challenging environments in both indoor and outdoor settings using the Orb [5]. This dataset comes from a research workstream which is isolated from Worldcoin’s operations and includes more than 2k images from 98 subjects with high resolution 1440x1080, resulting in a total of ~2.6 million matches. + +|4a Fig. 4(a) IRIS* ROC on WLD-InHouse-v2| 4b Fig. 4(b) IRIS* DET on WLD-InHouse-v2| +|---|---| + +> **Note**: IRIS* is an optimised version of IRIS to accommodate high-resolution images captured by the Orb. + +Because of the high quality image capture enabled by the Orb and the effectiveness of iris encoding enabled by IRIS*, we are able to achieve high accuracy with FNMR less than 0.0001 at FMR of 0.000001 in WLD-InHouse-v2 dataset. + +## References + +1. Phillips, P. , Bowyer, K. , Flynn, P. , Liu, X. and Scruggs, W. (2008), The Iris Challenge Evaluation 2005, IEEE Second International Conference on Biometrics: Theory, Applications and Systems (BTAS 08), Arlington, VA, (https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=890057) +2. Bowyer, K. , Flynn, P. (2016), The ND-IRIS-0405 Iris Image Dataset, (https://arxiv.org/abs/1606.04853) +3. Othman, N. , Dorizzi, B. , Garcia-Salicetti, S. (2016), OSIRIS: An open source iris recognition software, Pattern Recognition Letters, vol.82, part2, pp.124-131, 2016. (https://github.com/tohki/iris-osiris) +4. The ND-CrossSensor-Iris-2013 Dataset (Accessed 2018), (https://cvrl.nd.edu/projects/data/#nd-crosssensor-iris-2013-data-set) +5. The Worldcoin Foundation blog (2023), Opening The Orb: A look inside Worldcoin’s biometric imaging device, (https://worldcoin.org/blog/engineering/opening-orb-look-inside-worldcoin-biometric-imaging-device) diff --git a/docs/performance_card/comparison_1a.png b/docs/performance_card/comparison_1a.png new file mode 100644 index 0000000..853fa85 Binary files /dev/null and b/docs/performance_card/comparison_1a.png differ diff --git a/docs/performance_card/comparison_1b.png b/docs/performance_card/comparison_1b.png new file mode 100644 index 0000000..c5592fb Binary files /dev/null and b/docs/performance_card/comparison_1b.png differ diff --git a/docs/performance_card/comparison_2a.png b/docs/performance_card/comparison_2a.png new file mode 100644 index 0000000..33f74e8 Binary files /dev/null and b/docs/performance_card/comparison_2a.png differ diff --git a/docs/performance_card/comparison_2b.png b/docs/performance_card/comparison_2b.png new file mode 100644 index 0000000..641761d Binary files /dev/null and b/docs/performance_card/comparison_2b.png differ diff --git a/docs/performance_card/ice2005_results.png b/docs/performance_card/ice2005_results.png new file mode 100644 index 0000000..85c1b18 Binary files /dev/null and b/docs/performance_card/ice2005_results.png differ diff --git a/docs/performance_card/lg4000_3a.png b/docs/performance_card/lg4000_3a.png new file mode 100644 index 0000000..70cf81b Binary files /dev/null and b/docs/performance_card/lg4000_3a.png differ diff --git a/docs/performance_card/lg4000_3b.png b/docs/performance_card/lg4000_3b.png new file mode 100644 index 0000000..e6673b6 Binary files /dev/null and b/docs/performance_card/lg4000_3b.png differ diff --git a/docs/performance_card/wld_dataset_4a.png b/docs/performance_card/wld_dataset_4a.png new file mode 100644 index 0000000..74853ba Binary files /dev/null and b/docs/performance_card/wld_dataset_4a.png differ diff --git a/docs/performance_card/wld_dataset_4b.png b/docs/performance_card/wld_dataset_4b.png new file mode 100644 index 0000000..f04866f Binary files /dev/null and b/docs/performance_card/wld_dataset_4b.png differ