Sources filtered by criterion

[sahalali]

This ticket is created to address the functionality described in culturecreates/tech-support-tickets#94

Expectation: @sahalali to update

Description copied from ticket:

Signe Laval would like to import (using an aggregator) certain events from the La Vitrine source in Artsdata. The events in question would be a filtered subset of the La Vitrine events, the ones that feature Lavallois artists. In other words the La Vitrine aggregator from Artsdata would only include events with Lavallois artists performing.

Your 'La Vitrine' aggregator would only include events that meet this criteria:
-performed by "lavallois artist"

The Signe Laval CMS would be the source of the latest list of Lavallois artists (people and organizations) via these 2 taxonomies used in people/orgs:
Orgs: the dynamic taxonomy is called "Ville de l'organisation:
People: the dynamic taxonomy is called "Ville de résidence"
NOTE: the names of the taxonomies are subject to change.
These taxonomies are used to identify which people/Organizations in the Signe Laval CMS are Lavallois. So this information would have to be available to identify which events to import from Artsdata.

Could we automate entry of lavallois artist event's that are in arts data in my cms ?
Like if Adam Jonhson as an event at la place des arts it can enter straight up in brouillon in my cms !

[troughc]

https://placedesarts.com/fr/evenement/half-moon-run-osm-pop

[troughc]

https://kg.artsdata.ca/entity?uri=http%3A%2F%2Fkg.artsdata.ca%2Fresource%2FK23-1785

[dev-aravind]

Assigning to @saumier to provide a sample source for @dev-aravind to test the filters.

[saumier]

@dev The sample event is now linked to the artists Adam Johnson
ad:K5-513. Please try to load the source placedesarts-com from Artsdata.

[dev-aravind]

@saumier

Proposed aggregator algorithm to filter entities in Artsdata:

• Fetch all the entities in the source for which we have a filter for (if we are filtering events by some places we will fetch all the places first)

• Filter the entities by filters mentioned in the mapping file, for example if the filter is :

  {
  "entityType" : "PLACE",
  "artsdataFilters":[
  {
  "inputProperty":["address.addressRegion.@none"],
  "includePatterns" : ["NB"],
  "excludePatterns" : []
  }
  ]
  }

we will get places from the region "NB" only

• Fetch events and check if the location in each individual event is present in any of the filtered places.

• If yes, the event gets imported, else it gets skipped.

I would have to write 2 SPARQLs to make this work, one to get all the places in case of a place filter and one to get all the people/organization in case of a organizer/performer/sponsor filter. Please review this and let me know what you think.

[saumier]

@dev-aravind Thanks for writing down your psuedocode. It is very clear.

I have a few concerns. Currently the SPARQL to fetch events from Artsdata is the same for all sources. It is tempting to customize this SPARQL. For example, the SPARQL could fetch only the events?event s:location/s:address/s:addressRegion ?region . filter(str(?region) = "NB"). When I discussed this direction with Suhail last year we decided to keep one SPARQL for all sources so that we could maintain the SPARQL without having to manage many duplicate SPARQLs with small differences per calendar. So I like your idea of 2 additional SPARQLs for places and agents(person or organization). But is there a more elegant solution (less code)? There are several ideas that come to mind but I haven't thought through them. One idea is to load the data into a local graph in Aggregator (using Javascript RDF) and then apply the filter in a single step. Another idea is divide the calls to Artsdata into 2 steps: get a list of events IDs, then load the data for each event. If we use those 2 steps, the first could be filtered to only list the events to import (i.e. events in NB), and the second could always load the event data in the same way to be easy to maintain. The second step takes as input the list of events from the first step. Do you think that would work?

Something else I'd like you to know about. The current SPARQL to load events from Artsdata will change sometime in the next 6 months. The plan is to simplify it by using the minted events in Artsdata. We are going through a transition at the moment by using the SPARQL to merge on-the-fly events to use minted IDs.

To summarize, I am proposing the following pattern:

1. SPARQL to get the list of events (same for all calendars)
2. SPARQL to filter the list of events (i.e. custom filter(str(?region) = "NB"). It could be generated from the calendar's filter configuration.
3. SPARQL to get event data for each event in the list (same for all calendars).

[dev-aravind]

@saumier The event is now imported into the staging CMS.

[saumier]

@sahalali Lets review.