Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore invalid attributes when scraping #990

Open
fbacall opened this issue Jun 12, 2024 · 0 comments
Open

Ignore invalid attributes when scraping #990

fbacall opened this issue Jun 12, 2024 · 0 comments

Comments

@fbacall
Copy link
Member

fbacall commented Jun 12, 2024

When scraping resources, the various model validations can sometimes be overly strict.

For example when parsing a Bioschemas LearningResource, if a resource is marked-up with:

"educationalLevel": "Introductory"

which TeSS uses for the difficulty_level field, it will fail validation because TeSS uses a controlled vocab for this field and "Introductory" is not a valid option. This validation failure prevents the material being registered entirely.

Instead, TeSS could just ignore that one field and register the material anyway.

  • Look into a way of stripping invalid attributes from a model and re-attempting the creation
  • Ensure these "soft failures" are logged somewhere as warnings (in the source's ingestion log)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant