Merge Duplicates - Phase 1 - events

[troughc]

First: sorry if this ticket already exists - I could not find it in our board.

Final LINK
https://www.figma.com/design/DBnSSMT9Etfm6guDKKSp8O/Culture-Creates---Footlight-Calendar?node-id=10157-100372&m=dev

New feature for Admins only to merge duplicates. This feature will be available in all workspaces lists (events, people, places, organizations). The Figma illustrates the events view as an example/model for how to implement
Of note: the checkboxes (left of the thumbnail image in the list views) will always be displayed for Admins. The checkboxes are not displayed for other user profiles.

French translations have been added in FIGMA

Tasklist :

Events:

• 1. Make the event listing page selectable.
• 2. Introduce Merge duplicates and cancel button in the footer.Create the footer component to reuse.
• 3. Add routing to event forms from the listing pages.
• 4. Api integration to get the details of selected events.
• 5. Create the number component to show the no.of duplicates found and add dropdown.
• 6. Add "review merged content" component to the right.
• 7. Add the logic to show "review merged content " or "remove unwanted versions".
• 8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
• 9. If image duplicates exists ,they should be compared based on the Uri.
• 10. Add the date component to show duplicates.
• 11. Add the tree selected component to show duplicates.
• 12. Add the api integration to update the respective event.
• 13. Add the api integrations to delete unwanted events.

People :

• 1. Make the people listing page selectable.
• 2. Reuse Merge duplicates and cancel button in the footer.
• 3. Add routing to person forms from the listing pages.
• 4. Api integration to get the details of selected person.
• 5. Integration to show the duplicates found.
• 6. Add "review merged content" component to the right.
• 7. Add the logic to show "review merged content " or "remove unwanted versions".
• 8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
• 9. If image duplicates exists ,they should be compared based on the Uri.
• 10. Add the tree selected component to show duplicates.
• 11. Add the api integration to update the respective person.
• 12. Add the api integrations to delete unwanted people.
• 13. Show the social media link.Let the user choose to remove the link.

Place :

• 1. Make the Place listing page selectable.
• 2. Reuse Merge duplicates and cancel button in the footer.
• 3. Add routing to place form.
• 4. Api integration to get the details of selected place.
• 5. Integration to show duplicate found.
• 6. Add "review merged content" component to the right.
• 7. Add the logic to show "review merged content " or "remove unwanted versions".
• 8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
• 9. If image duplicates exists ,they should be compared based on the Uri.
• 10. Add the tree selected component to show duplicates.
• 11. Compare the street name,city,province ,country,coordinates and update the respective postal address.
• 12. Add the api integration to update the respective place
• 13. Add the api integrations to delete unwanted people.

Organization :

• 1. Make the organization listing page selectable.
• 2. Reuse Merge duplicates and cancel button in the footer.
• 3. Add routing to organization form.
• 4. Api integration to get the details of selected organization.
• 5. Integration to show duplicate found.
• 6. Add "review merged content" component to the right.
• 7. Add the logic to show "review merged content " or "remove unwanted versions".
• 8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
• 9. If image and logo duplicates exists ,they should be compared based on the Uri.
• 10. Add the tree selected component to show duplicates.
• 11. Add the api integration to update the respective organization
• 12. Add the api integrations to delete unwanted organization.

Tasks

Preview Give feedback

1. Add regression test in the merge duplicates page for events #1484
   +0
   

[troughc]

@saumier @sahalali here is the issue for the merge duplicates feature

[sahalali]

@troughc
Please take note of the following points:

1. How should we compare events to find duplicates? For instance, should duplicates be identified based on Artsdata ids, titles, dates, or a combination of these?
2. When merging two or more events, the priority for which event should be preserved. For example, if event A and event B are merged, the priority associated with event A should be retained, and event B should be deleted.
3. It seems necessary to enable the selection of events from multiple pages as users may find it difficult to filter events for merging within the same page.
4. Users should have the option to select events based on multiple searches. For instance, they should be able to merge events with entirely different search parameters.
5. An option to deselect events should be included.
6. Should users be restricted from merging events in the Published state, or merging Draft events with Published events?
7. Should the merge option be restricted based on user roles?
8. I think it will be helpful for the user if a 'tag' can be added to each value to differentiate the source event. It can be a message on hover or a coloured tag.

[AbhishekPAnil]

Frontend estimates : 166H

Events: 46H

1. Make the event listing page selectable.
2. Introduce Merge duplicates and cancel button in the footer.Create the footer component to reuse.
3. Add routing to event forms from the listing pages.
4. Api integration to get the details of selected events.
5. Create the number component to show the no.of duplicates found and add dropdown.
6. Add "review merged content" component to the right.
7. Add the logic to show "review merged content " or "remove unwanted versions".
8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
9. If image duplicates exists ,they should be compared based on the Uri.
10. Add the date component to show duplicates.
11. Add the tree selected component to show duplicates.
12. Add the api integration to update the respective event.
13. Add the api integrations to delete unwanted events.

People : 40H

1. Make the people listing page selectable.
2. Reuse Merge duplicates and cancel button in the footer.
3. Add routing to person forms from the listing pages.
4. Api integration to get the details of selected person.
5. Integration to show the duplicates found.
6. Add "review merged content" component to the right.
7. Add the logic to show "review merged content " or "remove unwanted versions".
8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
9. If image duplicates exists ,they should be compared based on the Uri.
10. Add the tree selected component to show duplicates.
11. Add the api integration to update the respective person.
12. Add the api integrations to delete unwanted people.
13. Show the social media link.Let the user choose to remove the link.

Place : 40H

1. Make the Place listing page selectable.
2. Reuse Merge duplicates and cancel button in the footer.
3. Add routing to place form.
4. Api integration to get the details of selected place.
5. Integration to show duplicate found.
6. Add "review merged content" component to the right.
7. Add the logic to show "review merged content " or "remove unwanted versions".
8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
9. If image duplicates exists ,they should be compared based on the Uri.
10. Add the tree selected component to show duplicates.
11. Compare the street name,city,province ,country,coordinates and update the respective postal address.
12. Add the api integration to update the respective place
13. Add the api integrations to delete unwanted people.

Organization : 40H

1. Make the organization listing page selectable.
2. Reuse Merge duplicates and cancel button in the footer.
3. Add routing to organization form.
4. Api integration to get the details of selected organization.
5. Integration to show duplicate found.
6. Add "review merged content" component to the right.
7. Add the logic to show "review merged content " or "remove unwanted versions".
8. Compare the values in the respective fields show each and every field where there is a duplicate.(The selected field can also be edited).
9. If image and logo duplicates exists ,they should be compared based on the Uri.
10. Add the tree selected component to show duplicates.
11. Add the api integration to update the respective organization
12. Add the api integrations to delete unwanted organization.

[troughc]

@saumier can you please review my answers below when you get back? Let us know if you agree or have other insights.

1. How should we compare events to find duplicates? For instance, should duplicates be identified based on Artsdata ids, titles, dates, or a combination of these?
   This phase 1 workflow is assuming that the person/Admin searching for duplicates is using the search bar to show events by name (title). I suppose they could be searching by organization. This means that all events with same/similar titles (or same organization) would be displayed in the event list right? Then, the Admin explicitly (manually) selects the ones they want to compare and clicks the Merge duplicates button.

QUESTION: If the Artsdata ID pre-exists is it even possible to add a duplicate to the CMS? If yes, you can base the comparison of finding duplicates on an event or event series have the same artsdata ID.
NOTE: even if they have different Artsdata IDs, I think they could be duplicates. I cover this in your next question.

Bigger picture (when we do automatic detection of duplicates): I suggest that the duplicate search comparison for EVENTS should be based on title, date(s) and location. For Places, based on name and address/postal code for organizations and people, based on name.

• If date and location are the same and title is same or close, it's a good indicator that it is a duplicate.
• Even if the title is different, it can still be a duplicate.
  Note: it can also still be a duplicate if there is an event series (with multiple dates) versus a single event (with one date) - a person could have added one event in an event series instead of adding all the events. In this case, the location and a single date would be the same (and also the title).

2. When merging two or more events, the priority for which event should be preserved. For example, if event A and event B are merged, the priority associated with event A should be retained, and event B should be deleted.
   Okay. How do you determine which event is event A and which event is event B?
   If there is no Artsdata ID it probably doesn't matter what you preserve...would not the saved (merged) event simply overwrite (and delete) all the selected/displayed duplicates?
   If there is at least one Artsdata ID for one of the selected events:

• If event A has an Artsdata ID and Event B does not, I think that the merged date title and location would have to match the Artsdata ID event for it to be the event which is retained. If it does not match, I think you should save the merged event as a a new event (without an Artsdata ID). This new event would get a new Artsdata ID. @saumier is this correct?
• If events A, B , C... all have DIFFERENT Artsdata IDs, only one would be preserved in the CMS if the user chooses to merge (and save) them all (the user may choose NOT to merge). As for which Artsdata ID to choose: I think the one that most closely matches (same date, title, location, and maybe other properties like event type, description, performer/organizer). @saumier can you please help with this?

3. It seems necessary to enable the selection of events from multiple pages as users may find it difficult to filter events for merging within the same page.
   Sounds okay. Although I would hope the Admin is able filter down duplicates (by name) to one page.

4. Users should have the option to select events based on multiple searches. For instance, they should be able to merge events with entirely different search parameters.
   How would you do this?

5. An option to deselect events should be included.
   Good idea. Uou could add the breadcrumb 'back to previous screen' to take you back to your duplicate search in the list view. And a warning that you didn't save your changes.
   NOTE: You can always exit the merge activity by clicking the right navigation.

6. Should users be restricted from merging events in the Published state, or merging Draft events with Published events?
   No. Only Admins can merge.

7. Should the merge option be restricted based on user roles?
   Yes, only Admins can merge duplicates.

8. I think it will be helpful for the user if a 'tag' can be added to each value to differentiate the source event. It can be a message on hover or a coloured tag.
   Hmm, this may complicate the UI and user experience. Also, I am not sure how we would do this in fields that take multiple inputs like Event type. In this case, we will display all merged options in one field for the Admin to edit - and some options will be the same across different events - and we will display the duplicated option only once.
   As a prerequisite, the admin should already be familiar with the event before doing the work.
   I suggest we implement as is and see if we get feedback that they want more detailed information. We can enhance the feature if necessary.

[troughc]

@AbhishekPAnil I would like to get feedback from @saumier when he returns, but I am assigning this back to you so you are not blocked from doing your estimate. thanks!

[saumier]

If event A has an Artsdata ID and Event B does not, I think that the merged date title and location would have to match the Artsdata ID event for it to be the event which is retained. If it does not match, I think you should save the merged event as a a new event (without an Artsdata ID). This new event would get a new Artsdata ID. @saumier is this correct?
If events A, B , C... all have DIFFERENT Artsdata IDs, only one would be preserved in the CMS if the user chooses to merge (and save) them all (the user may choose NOT to merge). As for which Artsdata ID to choose: I think the one that most closely matches (same date, title, location, and maybe other properties like event type, description, performer/organizer). @saumier can you please help with this?

@troughc @sahalali In all cases of entities resulting from a merge, I think the Artsdata IDs in CMS that are different should be added to the merged entity without any special consideration. It will help traceability. This is also true for sameAs URIs that are not visible to the user in the UI and cannot be selected by the user (like Footlight Console URIs, Caligram URIs, ...). Even if an event has 2 or 3 or 4 ... Artsdata IDs after the merge. For there to be more than 1 Artsdata ID there has been an error somewhere in Artsdata. I don't think CMS should be responsible for choosing the right Artsdata ID and there is no visibility in CMS of the ID of Artsdata IDs which makes it currently impossible for a user to select one. Let Artsdata be responsible to link or mint on its own and create the sameAs between Artsdata IDs if needed. In the CMS UI that has a link to Artsdata, it can link to the first arbitray Artsdata ID if there is more than one Artsdata ID in the merged event.

[AbhishekPAnil]

cc @AbhishekPAnil For this pass add select all checkbox to the top of the listing pages.

[AbhishekPAnil]

@troughc Should a user be able to share the list of selected duplicates ?Like the event listing with the selected filters we have right now!

[AbhishekPAnil]

One thing to mention @AbhishekPAnil @dev-aravind: remember that when an Admin merges duplicates that are places organizations or people, the final entity needs to link to ALL the other entities that each duplicate was linked to. For example if you merge two places because they are duplicates, all events that have either place as a location should continue to be linked to the final, chosen entity.

Question: is this what you are doing?

cc @sahalali @saumier

This has been noted .

[dev-aravind]

Task for @dev-aravind:

Modify all the entities update APIs to remap duplicate entity references to the merged one.

Organizations - Referenced in Events
Places - Referenced in Events, Places, Organizations
People - Referenced in Events

[dev-aravind]

@AbhishekPAnil The backend changes can be found in this branch. Please test it and let me know if you need any changes.

[troughc]

@AbhishekPAnil I replied to your question in the Figma.

[AbhishekPAnil]

cc @AbhishekPAnil Will be displaying the dates separately even if the events have same dates.

[sahalali]

@AbhishekPAnil What is the status of this ticket?

[AbhishekPAnil]

@AbhishekPAnil What is the status of this ticket?

This is in progress. Currently working on integrating the language changes that syam did.I am doing this ticket in parallel.

[sahalali]

If event A has an Artsdata ID and Event B does not, I think that the merged date title and location would have to match the Artsdata ID event for it to be the event which is retained. If it does not match, I think you should save the merged event as a a new event (without an Artsdata ID). This new event would get a new Artsdata ID. @saumier is this correct?
If events A, B , C... all have DIFFERENT Artsdata IDs, only one would be preserved in the CMS if the user chooses to merge (and save) them all (the user may choose NOT to merge). As for which Artsdata ID to choose: I think the one that most closely matches (same date, title, location, and maybe other properties like event type, description, performer/organizer). @saumier can you please help with this?

@troughc @sahalali In all cases of entities resulting from a merge, I think the Artsdata IDs in CMS that are different should be added to the merged entity without any special consideration. It will help traceability. This is also true for sameAs URIs that are not visible to the user in the UI and cannot be selected by the user (like Footlight Console URIs, Caligram URIs, ...). Even if an event has 2 or 3 or 4 ... Artsdata IDs after the merge. For there to be more than 1 Artsdata ID there has been an error somewhere in Artsdata. I don't think CMS should be responsible for choosing the right Artsdata ID and there is no visibility in CMS of the ID of Artsdata IDs which makes it currently impossible for a user to select one. Let Artsdata be responsible to link or mint on its own and create the sameAs between Artsdata IDs if needed. In the CMS UI that has a link to Artsdata, it can link to the first arbitray Artsdata ID if there is more than one Artsdata ID in the merged event.

@saumier I have question, what if the event date / title is mismatching with the Artsdata entity.
Example: If the merged entity doesn't match the date or title in Artsdata after merge.

[AbhishekPAnil]

cc @dev-aravind
Please add support for

1. making rest of the merged events obsolete.
2. merging events into one event.

[saumier]

@saumier I have question, what if the event date / title is mismatching with the Artsdata entity.
Example: If the merged entity doesn't match the date or title in Artsdata after merge.

@sahalali My opinion has not changed. I still think the merge should not worry about anything else but merging. Since the CMS has no UI to edit or change sameAs URIs there should not be any business logic in how to treat them other than merge. When the data is loaded into Artsdata it will be handled there. In most cases the script that links and unlinks CMS to Artsdata will take care of it. That script is where the business logic should be focused on and only there. The other advantage of merging all sameAs identifiers that that it will appear in the logs and change history even if it gets unlinked in the next step.

[AbhishekPAnil]

Cc @AbhishekPAnil Add the remaining estimates for the event -merge duplicates and create separate tickets for the other workspaces and re-estimate the same.

[AbhishekPAnil]

Remaining estimates :
Development : 24-28 hrs.
Testing : 20 hrs