Data quality

[jvwong]

We've implemented some features to mitigate curation mistakes including:

• High accuracy grounding search
• Flagging potential issues at submission time

But mistakes happen, and monitoring and assessing the quality of the data is an issue that has been time and again with reviewers.

Entity

Mixing entities from different species

Below are (potential) mistakes in the production data set. In the instances below, I just checked whether there were genes from different species, and whether those were obvious mistakes. I did not verify whether the entities from each doc were accurate.

1. Doc: https://biofactoid.org/document/a4dd6b45-af1b-407c-9f0e-8d6ff9ae557e

• Observed: "PHLP2" is grounded to C elegans
• Expected: "PHLPP2" (human: https://www.ncbi.nlm.nih.gov/gene/23035)
• Reason: probably typo

2. Doc: https://biofactoid.org/document/12d96fc7-eadc-427a-9a86-6b9c7bdc6ec2

• Observed: "calcineurin" is from D. melanogaster
• Expected: "calcineurin" from human (there are three isozymes of the catalytic subunit, each encoded by a separate gene (PPP3CA, PPP3CB, and PPP3CC))
• Reason: Case where standard names are probably divorced from common usage

3. Doc: https://biofactoid.org/document/f28a7f27-8ee5-4a39-bd5b-e41f9a4e5c83

• Observed: "MRTF" is from mouse
• Expected: "MRTF" in human
• Reason: probably since human matches (MRTFA, MRTFB) come up 4th/5th in grounding search

4. Doc: https://biofactoid.org/document/aa18ee18-a78f-4d46-a244-9a6dc97bb238

• Observed: "Cofilin" is from human
• Expected: "Cofilin" is from mouse
• Reason: Mouse version is 4th in grounding

Disconnected entity

1. Doc: https://biofactoid.org/document/034d7fcc-80a0-4b78-8b1d-4fd8652dc4f8

• Observed: "mdm2" is not involved in interaction
• Expected: "mdm2" is involved in interaction with CtBp monomer
• Reason: The intention is to show that p53 is prevented from MDM2 binding by CtBP monomer

2. Doc: https://biofactoid.org/document/2753be81-1b79-4582-adaa-aea258ba24b2

• Observed: "PUM1" is not involved in interaction
• Expected: "PUM1" should not be duplicated
• Reason: hiding entity behind duplicate is visually ok

Duplicate articles (by pmid)

1. Cypher/ZASP is a novel A-kinase anchoring protein. J Biol Chem 288 2013

• https://biofactoid.org/document/12d96fc7-eadc-427a-9a86-6b9c7bdc6ec2
• https://biofactoid.org/document/19d56861-93bf-4f02-b8ff-136cf8a24d60

2. Targeted interplay between bacterial pathogens and host autophagy. Autophagy 15 2019

• https://biofactoid.org/document/270ced12-74fd-4932-8031-b8ae4a25a9e3
• https://biofactoid.org/document/b43d3125-42a8-4ef8-b8a9-fc49f5b76309

Entity / as process, compartment

1. Doc: https://biofactoid.org/document/104efd69-c6e5-48e2-95ed-b921af7b3bf3

• Observed: "ADT" implying androgen deprivation therapy
• Expected: Inhibition of androgen via LHRH agonists (e.g. Triptorelin, Leuprolide)
• Reason: Wanted to convey a stimulus/process

2. Doc: https://biofactoid.org/document/2d80235e-650c-4338-815f-d8906559ac41

• Observed: "Lysosome"
• Expected: Cellular compartment
• Reason: Wanted organelle

Misc Entity

a. Doc: https://biofactoid.org/document/047c96a4-1006-4052-aa97-9e42ed8e6a9e
- Observed: "Ribosome" (grounded to ribose)
- Expected: Complex, labelled ribosome (i.e. https://www.ebi.ac.uk/QuickGO/term/GO:0005840)
- Reason: ?

FYI this gets merged at the BioPAX level:

b. Doc: https://biofactoid.org/document/047c96a4-1006-4052-aa97-9e42ed8e6a9e
- Observed: "Nascent RPS2" (grounded to CHEBI:10216)
- Expected: Ground to RPS2 in same network (ncbi:6187)
- Reason: Extra token 'Nascent' entirely throws off search.

2. Doc: https://biofactoid.org/document/7f77ddcc-bbc9-4c04-b416-d7775798f301

• Observed: "Estrogen-responsive Enhancers (Homo sapiens)" (grounded to Yamogenin 3-O-neohesperidoside)
• Expected: ESR1, type = DNA
  • In a related issue, I notice that we're not displaying the 'type' of entity in general, which is likely important information as this is a proxy for the different symbols scientists typically use to convey (b-sheet for protein, arrow for promoter).
• Reason: Didn't inspect ground at all

4. Doc: https://biofactoid.org/document/034d7fcc-80a0-4b78-8b1d-4fd8652dc4f8
   a. Grounding distraction
   • Observed: "CtBP monomer" is grounded to some chemical "xanomeline"
   • Expected: "CtBP monomer" is grounded to a gene gene CTBP2
   • Reason: The 'monomer' token alters the ranking
   • Comments: Can NLP help?
     • 1. Ignore extra token(s)
     • 2. Use the extra tokens if they fit a category (e.g. stoichiometry; phosphorylation...state)
          b. Stoichiometry
   • Observed: "NADH dependent dimerisation" CtBP appears twice in complex
   • Expected: Single CtBP (will be merged in BioPAX anyways)
   • Reason: No support for stoichiometry

5. Doc: https://biofactoid.org/document/ee9ff968-088d-45be-8f63-3bfb9c5fd64e

• Observed: Complex "PPP1CB:NUAK1:PNUTS"
• Expected: No complex (Interactions get erased in BioPAX)
• Reason: Didn't know that complex is effectively an entity, but BioPAX doens't specify intra-complex interactions

Interaction

1. Doc: https://biofactoid.org/document/8ba702d2-3a88-4d8a-b93f-b614a347c2c3

• Observed: IRGM activates NLRP3
• Expected: IRGM inhibits NLRP3
• Reason: ?

Other

Cellular compartment

1. Doc: https://biofactoid.org/document/ed747856-15b8-4a31-82a0-05f57cca66d1

• Observed: miR223 with YBX1 and YBAP1 in mitochondria and exosome, respectively
• Expected: no compartment
• Reason: Out of scope

[jvwong]

November 2022 Update: Quality analysis of Biofactoid contributions

Code helper: https://github.com/jvwong/quality-control

Table I: Documents were either randomly sampled or previously identified as problematic ("targeted")

Type	Count
targeted	17
random	19
documents	165

Table 2: Analysis of randomly sampled documents

Feature	Accurate	Total	Percent
Mechanism	45	48	93.8%
Entity ground	68	71	95.8%
Edge type	48	48	100.0%

Table 3: Analysis of "targeted" documents that were manually identified as problematic

Feature	Accurate	Total	Percent
Mechanism	60	63	95.2%
Entity ground	53	76	69.7%
Edge type	60	63	95.2%

Table 4: Summary of entity error types (Nov 2022)

Error Type	Count.
Token/Modifier	4
Process	1
Complex/Family	4
Species	7
Spelling	2
Other	3
Total	26

See Google Sheet "Biofactoid: Quality Control" for full details.

Entity grounding issues

What general kinds of things did I see with respect to entity/participant grounding errors?

• Species mix up: Same name, different organism
  • Often with mouse (e.g. p53)
  • Sometimes cross-species is valid (host - pathogen LC3b / yhjj)
• Prefixes: Adjectives or modifers to entity name
  • e.g. pre-miR-2137
  • e.g. Stoichiometry ("CtBP monomer")
  • e.g. "Nascent RPS2"
  • e.g. Material type "RPS2 mRNA"
• Processes
  • e.g. cell cycle arrest genes
• Complex support
  • e.g. "Ribosome"
  • Complex-only docs (9867a569-5f37-4296-818f-d16cf3dbf0a3)
  • Misinterpretation of complex (strict)
  • Intra-complex interactions
• Family
  • e.g. PKC
  • e.g. PI3K
  • Class of chemicals (diacylglycerol, varying R sidechains)
• Other
  • Nodes
    • empty
    • disconnected
  • Temporal or dynamics (Co-translation PDCD2)
  • Spelling mistakes (e.g. Calcinurin [Calcineurin], PHLP2 [PHLPP2])
  • Mistaken a modification for a complex (ubiquitin Rheb)

[jvwong]

January 2024 Update: Ongoing screen

Source data Google Sheet

Document ID	Element ID	Label	Ground ID	Ground Name	Expected ID	Expected Name	Error Type
d0e52e42-fff4-45b1-a630-aecfa9d56f45	485565ad-dadd-4f9d-9aa3-5c1266bbeb49	Particulate matter 2.5	CHEBI:177893	variculatriene A	ENVO:01000415	fine respirable suspended particulate matter	Concept
2efa50b6-b6a4-497f-911a-9be49cc6d60c	b16048b5-ddfd-4371-a2ce-fc23400315d2	BJG-05-039	fplx:CYP	CYP	ligandId=12266	BJG-05-039	Record
579d3c4c-afd7-49c4-9351-1dd31909026f	60b6a6c8-a0ce-44a6-997d-316b1d4592d5	Plasma membrane	CHEBI:156268	asperaculane G	GO:0005886	plasma membrane	Concept
780577bc-ee59-433f-96a8-02d2a480981e	68fb5641-c087-4d31-bbac-7172f78134dc	Plasma membrane	CHEBI:156268	asperaculane G	GO:0005886	plasma membrane	Concept
27b91cfe-9c41-448d-b182-3fb7ae613bec	d23223ac-6fb8-409f-9c9b-52b8d7b1d1d1	cell cycle genes	ncbi:6191	RPS4X	-	-	Concept
27b91cfe-9c41-448d-b182-3fb7ae613bec	6ce2939c-18b5-403e-aa20-17f51ef942e7	Cell cycle progression	ncbi:9236	CCPG1	GO:1901989	positive regulation of cell cycle phase transition	Concept
d1f479b2-6652-40e7-ab2e-fc8d052190e6	bc156d24-e2ce-4a43-8a74-dbc63bdb9eb7	Glucocorticoid Receptor pThr524	CHEBI:91834	4-methyl-3-[[1-oxo-2-(propylamino)propyl]amino]-2-thiophenecarboxylic acid methyl ester	ncbi:2908	nuclear receptor subfamily 3 group C member 1	Multiconcept
88a6830a-e9e2-4df1-b64e-71ed8274ceff	e2b42107-1afb-4283-b11b-35cb6556666e	Exercise	CHEBI:94590	3,3-diethyl-1H-pyridine-2,4-dione	-	-	Concept
88a6830a-e9e2-4df1-b64e-71ed8274ceff	b519451b-9d62-4d66-8ac1-faa6589019e3	GLUT4 Vesicle	CHEBI:9380	syringin	GO:0032593	insulin-responsive compartment	Concept
7f77ddcc-bbc9-4c04-b416-d7775798f301	52faf1dd-824b-48f1-bdfa-f61942de2852	Estrogen-responsive Enhancers	CHEBI:10087	Yamogenin 3-O-neohesperidoside	-	-	Concept
a646470f-a91a-40c2-8041-2f5e5a082df5	936b35c0-46e3-49f2-b818-f87ae87f621e	Epicardial EMT	CHEBI:155896	benzyl 2,5-dihydroxybenzoate	GO:0003347	epicardial cell to mesenchymal cell transition	Concept
02714f9b-926b-41b5-9999-99477deec935	1b58ccae-3426-4595-aca1-82eef6229d3b	abnormal mitral valve	fplx:RAL	RAL	MP:0000286	abnormal mitral valve morphology	Concept
52aee1d1-1264-4f23-8d89-6c0fd5c12fd3	52c6c507-2185-419e-9d26-0a2ca70ff2fd	GRP78 cell surface translocation	fplx:Cation_channels	Cation_channels	ncbi:3309	HSPA5	Multiconcept
52aee1d1-1264-4f23-8d89-6c0fd5c12fd3	503850a8-e4fe-427d-b532-b109d41e05e0	Hyperglycemia	fplx:DGK	DGK	SNOMED:80394007	Hyperglycemia (disorder)	Concept
52aee1d1-1264-4f23-8d89-6c0fd5c12fd3	67ff77e3-b349-4bc5-84ad-647e1ffde775	↑ Extracellular Matrix Protein Overproduction/ Fibrosis	fplx:ERK	ERK	NCIT:C3044	Fibrosis	Concept
52aee1d1-1264-4f23-8d89-6c0fd5c12fd3	ed3e7b1c-3041-43ba-af21-fd0710d5a1bf	Activated Alpha 2-macroglobulin	fplx:MAPK	MAPK	ncbi:2	alpha-2-macroglobulin	Multiconcept
52aee1d1-1264-4f23-8d89-6c0fd5c12fd3	af81e44d-eeb3-4721-8b25-4d90b557dd67	Latent TGFβ1 Complex	fplx:TFDP	TDFP	GO:0038045	large latent transforming growth factor-beta complex	Concept
37be6137-8e26-4c32-977c-8af9fcae3ccb	2653d1d7-e98f-4db6-afa7-cbb707c80dd3	neutrophil extracellular traps	fplx:ERK	ERK	NCIT:C180931	Neutrophil Extracellular Trap	Concept
37be6137-8e26-4c32-977c-8af9fcae3ccb	2637cf25-b8ac-422a-ad79-3d3c39fff531	neutrophils	fplx:Neuropilins	Neuropilins	CL:0000096	mature neutrophil	Concept
11d26818-d198-4fc5-bb82-59db73762b20	212ae483-bad1-4ada-a0ce-fb4aa815768f	ischemia	fplx:Chemokine	Chemokine	SNOMED:52674009	Ischemia (disorder)	Concept
7ad7b201-be22-4390-b30b-9b126dc3cec5	c3a1d202-a319-4450-9648-861b5b494458	Gogli fragmentation	CHEBI:155922	elemental niobium	CMPO:0000152	fragmented golgi phenotype	Concept
7ad7b201-be22-4390-b30b-9b126dc3cec5	066349e1-8105-4f67-b2a2-10fef37726c2	Heat stress	CHEBI:156346	preaustinoid A3	MCO:0000172	heat stress treatment	Concept
742b3751-883e-4ff3-8475-b3c1c2a4ab90	c1405406-b926-4323-a851-b586447c627b	PI3K/AKT signal	CHEBI:156139	N-acetyl-L-thialysine	-	-	Pathway
6b9b90a4-a657-4453-8833-60ad98d86550	90f3571c-c5d0-4633-a07b-ad356816d891	C75 (FASN inhibitor)	CHEBI:156465	asnovolin H(1−)	pubchem:24892789	C75	Record
6b9b90a4-a657-4453-8833-60ad98d86550	24b1ea1f-a907-4d9f-91a9-923b3f8a16d0	OGT Human	CHEBI:155921	elemental lanthanum	ncbi:8473	O-linked N-acetylglucosamine (GlcNAc) transferase	Multiconcept
c26af6d-32a2-49cf-8c71-26ec0cde4d2aa	abb1f2c7-0d11-4941-a25e-75725179b82a	Heart failure	CHEBI:15569	N-(carboxyaminomethyl)urea	NCIT:C50577	Heart Failure	Concept
999205de-8768-483b-9203-6a775f533099	330c76f1-97ba-43ef-828e-8c8de45244d7	BMS-345541	CHEBI:93113	LSM-3417	CHEBI:91340	N'-(1,8-dimethyl-4-imidazo[1,2-a]quinoxalinyl)ethane-1,2-diamine	Synonym
999205de-8768-483b-9203-6a775f533099	523a018f-9f3d-468b-a7e8-99f2a28f0ddf	Cirmtuzumab	CHEBI:9220	Sphagnum acid	NCIT:C118284	Zilovertamab	Concept
cdd26bce-9d82-48e0-8380-7326c765b2af	cc8a7406-85b8-4239-b007-8bbf368426bb	Pre-miR-2137	CHEBI:94646	N-[2-amino-6-[(4-fluorophenyl)methylamino]-3-pyridinyl]carbamic acid ethyl ester	ncbi:100316779	microRNA 2137	Multiconcept
1b257dab-095b-4a63-8e57-60edbf32ff9a	2262b563-5291-4d72-8dee-87fe6d5f41c5	14-3-3 rat	fplx:p14_3_3	p14_3_3	-	-	Multiconcept / Concept
726c4e0e-a844-4d1f-887c-9c354821d61e	11bea867-74c7-476c-85b4-0f27c5b5c187	mtRNA	ncbigene:5442	POLRMT	CHEBI:33697	ribonucleic acid	Multiconcept
726c4e0e-a844-4d1f-887c-9c354821d61e	f9ca50a0-c480-4c00-b936-83178616e888	nuclear chromatin	fplx:NFY	NFY	GO:0000785	chromatin	Multiconcept / Concept
cc83f9b6-489f-4d88-9947-ec4c5d5a97da	8a64fe6b-7612-4ffe-ae8e-4d97c2074e4b	FAO dependent pathway	fplx:PKA	PKA	-	-	Pathway
cc83f9b6-489f-4d88-9947-ec4c5d5a97da	88eb3c55-291a-4e5b-9196-888506de1ec3	SLC16A9/Carnitine	fplx:PPP3C	PPP3C	ncbi:220963	SLC16A9	Multiconcept
31909bbe-c871-4cdd-873e-591a718da550	bffe91c8-4569-4755-91e7-c353dbd49c92	replisome	fplx:TOP	TOP	GO:0030894	replisome	concept