Test data and benchmarking

Results

You can obtain these numbers by running by running the Benchmark.

Data set	Items	Coverage	Identified	Accuracy
3.23	3	0.48%	8 / 8	100.00%
Elinvynia	6	0.96%	9 / 9	100.00%
Krolya	131	21.06%	265 / 265	100.00%
gathered	553	88.91%	6320 / 6321	99.98%
reported	18	2.89%	28 / 28	100.00%
staticf0x	52	8.36%	110 / 110	100.00%
staticf0xElin	126	20.26%	278 / 278	100.00%
Total	557 / 622	89.55%	7018 / 7019	99.99%

Legend:

• Coverage column shows how many items from the database were covered, e.g. if 100 distinct unique items are in the data set (with 601 items in total), the coverage will be 16.6%. The total coverage shows the percentage across all data sets, so that no unique item is counted twice.
• reported data set is from all reported issues (either on GitHub or Discord), after fixing the issues and verifying the results
• gathered data set comes from screenshots gathered using Unique Matcher during testing, all manually verified
• All other data sets are from individual contributors gathered manually (usually having 1+ unidentified variant and 1+ identified variant)

Notes:

• Only drop-enabled and natural drops are counted
• Unless otherwise specified, the data is gathered using the latest version of Unique Matcher

Benchmark

You can run python3 benchmark.py to start a benchmark that will calculate the accuracy of a data set and timing data of the matching. The tool will let you pick which data set to run. If you pick multiple data sets, the benchmark will produce the table in the section above.

The data set needs to be located in tests/test_data/contains and the structure is as follows:

example
└── Bones_of_Ullr
    ├── Screenshot from 2023-08-25 18-02-47.png
    └── Screenshot from 2023-08-25 18-02-52.png

Missing items

The following items are not yet covered in the test data. If you have a screenshot of these unidentified and identified, please consider contributing the screenshots to help improve Unique Matcher's accuracy.

Base	Found	Missing items
Agate Amulet	3/4	Voll's Devotion
Assassin Bow	1/2	Darkscorn
Assassins Mitts	1/2	Storm's Gift
Carved Wand	1/2	The Poet's Pen
Cloth Belt	5/7	Sunblast, Bound Fate
Cobalt Jewel	1/3	Dead Reckoning, Unending Hunger
Colosseum Plate	0/1	Pragmatism
Coral Amulet	4/5	Ikiaho's Promise
Crimson Jewel	0/5	Might of the Meek, Inspired Learning, Firesong, Immutable Force, Bloodnotch
Ebony Tower Shield	1/2	Tukohama's Fortress
Elegant Round Shield	1/2	The Squire
Engraved Greatsword	0/1	Rakiata's Dance
Fiend Dagger	1/2	Arakaali's Fang
Fishing Rod	0/1	Reefbane
Goathide Boots	1/2	Abberath's Hooves
Gold Amulet	3/6	The Untouched Soul, The Ascetic, Bisco's Collar
Graceful Sword	0/1	Nametaker
Grand Mana Flask	0/1	Zerphi's Last Breath
Heavy Belt	6/7	Kaom's Binding
Hellions Paw	0/1	Bloodseeker
Imperial Bow	1/2	Lioneye's Glare
Imperial Skean	1/2	Divinarius
Iron Circlet	1/2	Asenath's Mark
Karui Maul	0/1	Marohi Erqi
Karui Sceptre	1/2	Maata's Teaching
Lacquered Garb	1/2	Cloak of Defiance
Majestic Plate	0/1	Utula's Hunger
Maple Round Shield	0/1	The Flawed Refuge
Moonstone Ring	4/6	Tawhanuku's Timing, Anathema
Murder Mitts	2/3	Thunderfist
Necromancer Silks	1/2	Fleshcrafter
Paua Amulet	1/3	Defiance of Destiny, Atziri's Foible
Primordial Staff	1/2	The Burden of Shadows
Prophet Crown	2/3	The Brine Crown
Regicide Mask	1/2	Akoya's Gaze
Riveted Boots	0/1	Ralakesh's Impatience
Sages Robe	1/2	Dialla's Malefaction
Saints Hauberk	1/2	Doryani's Prototype
Sapphire Flask	0/1	Taste of Hate
Siege Axe	0/1	Soul Taker
Slaughter Knife	0/1	Bino's Kitchen Knife
Spidersilk Robe	1/2	The Covenant
Spine Bow	1/2	Voltaxic Rift
Studded Belt	1/2	Ryslatha's Coil
Titanium Spirit Shield	0/1	Rathpith Globe
Two-Point Arrow Quiver	1/2	Rigwald's Quills
Two-Stone Ring	4/5	Call of the Brotherhood
Vaal Claw	1/2	Essentia Sanguis
Viridian Jewel	5/6	Ancestral Vision
Visored Sallet	0/1	The Trickster's Smile

History of accuracy

App version	Total accuracy
0.7.0	99.99%
0.6.1	99.81%
0.6.0	99.66%
0.5.4	99.60%
0.5.1	99.60%

Older versions are not displayed due to improper benchmarking.

Performance

Measured by python3 tools/benchmark.py --github --json, combined json: jq -s ". | add" benchmark-*.json > combined.json.

Ran on Ryzen 5600 with 8 threads (8 items processed in parallel).

By identification method

Method	Count	Mean	Std	Min	Max
ITEM_NAME	574	438ms	55ms	283ms	648ms
SOLARIS_CIRCLET	6	422ms	22ms	386ms	445ms
TEMPLATE_MATCH	2485	380ms	37ms	248ms	574ms
ONLY_UNIQUE_FOR_BASE	3929	364ms	33ms	250ms	809ms
HISTOGRAM_MATCH	25	351ms	34ms	293ms	485ms

By item being identified

Identified	Count	Mean	Std	Min	Max
Yes	575	438ms	55ms	283ms	648ms
No	6444	370ms	35ms	248ms	809ms

By number of sockets

Only for template matching and histogram matching.

Sockets	Count	Mean	Std	Min	Max
6	357	408ms	36ms	262ms	506ms
4	369	389ms	35ms	293ms	564ms
3	385	385ms	34ms	299ms	574ms
1	56	381ms	32ms	297ms	443ms
0	1343	369ms	33ms	248ms	551ms

Bases most likely to experience worse accuracy

This table shows min_val of template matching algorithm on the worst 10 item bases, sorted by 75% percentile. The lower the min_val, the better, maximum is 1.0, items are only considered as identified if min_val <= 0.96. As seen in the table, Eternal Sword in some cases is getting close to that min_val threshold.

Item base	Count	Mean	Std	Min	25%	50%	75%	Max
Eternal Sword	23	0.606707	0.188741	0.16595	0.525276	0.616685	0.734043	0.936182
Death Bow	16	0.545318	0.100011	0.38545	0.493082	0.527202	0.566099	0.796749
Spiraled Wand	14	0.445211	0.0957488	0.284219	0.373543	0.461773	0.526432	0.568439
Ezomyte Staff	21	0.447989	0.0749969	0.336764	0.3983	0.439883	0.496357	0.594609
Spine Bow	8	0.415173	0.0375281	0.359883	0.385623	0.414918	0.45061	0.461652
Ezomyte Axe	30	0.411596	0.0877233	0.283284	0.366445	0.391909	0.434933	0.69234
Gold Amulet	27	0.371971	0.0422855	0.289493	0.361236	0.36732	0.398582	0.448826
Stibnite Flask	2	0.304988	0.146431	0.201446	0.253217	0.304988	0.35676	0.408531
Cobalt Jewel	1	0.35136	nan	0.35136	0.35136	0.35136	0.35136	0.35136
Coral Amulet	28	0.249799	0.108789	0.111632	0.176555	0.226215	0.342137	0.446503