New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(api-v2): Remove ForbiddenResource #1615

Merged

benjamingeer merged 31 commits into develop from wip/1543-forbidden-resource

Apr 7, 2020

benjamingeer commented Feb 28, 2020 •

edited

Loading

User story: https://dasch.myjetbrains.com/youtrack/issue/DSP-36

Discussion:

https://discuss.dasch.swiss/t/how-to-deal-with-forbidden-resources-and-values-consistently/153

https://discuss.dasch.swiss/t/knora-api-js-lib-pr-changing-public-api/158/5

This PR removes ForbiddenResource to make Gravsearch more consistent with the rest of API v2.

Instead of using ForbiddenResource, Gravsearch now returns "knora-api:mayHaveMoreResults": true if the number of main resources returned, plus the number of main resources hidden because of permissions, equals the page size.

This has a couple of other advantages:

The client doesn't need to know the maximum page size (which is a server configuration setting that the client doesn't have access to) to find out if it can request another page.
The response is more concise.

The rest of API v2 now also distinguishes between a resource not being found (NotFoundException, HTTP 404) and the user not having permission to see it (ForbiddenException, HTTP 403).

This PR also unifies the processing of search responses and other resource responses in ConstructResponseUtilV2.

Tasks:

Change knora-base.ttl and system-data.ttl.
Change ConstructResponseUtilV2 and its dependencies.
Update tests and test data.
Add a client test response that includes knora-api:mayHaveMoreResults.
Update API docs.
Update design docs.
Increase the knora-base version number.

Resolves #1543.

Benjamin Geer added 4 commits

February 19, 2020 18:41


          feat(api-v2): Refactor to use ForbiddenResource consistently (ongoing).

3dbb59c


          Merge branch 'develop' into wip/1543-forbidden-resource

b6d3cf2

# Conflicts:
#	webapi/src/main/scala/org/knora/webapi/util/StringFormatter.scala


          test: Fix tests.

49f4214


          feat(api-v2): Use ForbiddenResource consistently (ongoing).

benjamingeer mentioned this pull request

Ben's PR history #571

Open

benjamingeer self-assigned this

benjamingeer added API/V2 breaking/api labels

benjamingeer added this to the 2020.1 milestone

Benjamin Geer added 5 commits

February 28, 2020 11:29


          Merge branch 'develop' into wip/1543-forbidden-resource

15c3be5


          refactor(api-v2): Move ForbiddenResource from triplestore to app.

cae954d


          refactor(test): Simplify code.

d6c8fc0


          refactor(api-v2): Simplify response processing (ongoing).

7f1d91f


          Merge branch 'develop' into wip/1543-forbidden-resource

623630c

benjamingeer mentioned this pull request

Consider simplifying permissions so they can be calculated in SPARQL #1618

Open

Collaborator

daschbot commented Mar 2, 2020

This pull request has been mentioned on Discuss DaSCH. There might be relevant details there:

https://discuss.dasch.swiss/t/how-to-deal-with-forbidden-resources-and-values-consistently/153/1

Benjamin Geer added 4 commits

March 3, 2020 13:24


          feat(api-v2): Add ForbiddenValue (ongoing).

5d1b28e


          feat(api-v2): Remove ForbiddenResource (ongoing).

e86b20b


          test(api-v2): Fix compile error.

c25808e


          feat(api-v2): Get rid of ForbiddenResource (ongoing).

d8d1c38

benjamingeer changed the title ~~feat(api-v2): Use ForbiddenResource consistently~~ feat(api-v2): Get rid of ForbiddenResource


          feat(api-v2): Remove ForbiddenResource (ongoing).

da23fc6

benjamingeer changed the title ~~feat(api-v2): Get rid of ForbiddenResource~~ feat(api-v2): Remove ForbiddenResource

Benjamin Geer added 5 commits

March 5, 2020 15:31


          feat(api-v2): Remove ForbiddenResource (ongoing).

baeff8a


          feat(api-v2): Remove ForbiddenResource (ongoing).

144863d


          refactor(api-v2): Move checkResourceIris into ReadResourcesSequenceV2.

f3ed00c


          style(api-v2): Clean up a few things.

5de2b47


          docs(gravsearch): Update docs.

ba61578

Collaborator

daschbot commented Mar 6, 2020

This pull request has been mentioned on Discuss DaSCH. There might be relevant details there:

https://discuss.dasch.swiss/t/how-to-deal-with-forbidden-resources-and-values-consistently/153/7

benjamingeer requested a review from tobiasschweizer

March 6, 2020 08:51

benjamingeer mentioned this pull request

fix(gravsearch): Prevent duplicate results #1626

Merged

Benjamin Geer added 4 commits

March 12, 2020 09:12


          test(gravsearch): Add client test data with paging.

40fa184


          docs(gravsearch): Update design docs.

b46c4f0


          feat(upgrade): Increment knora-base version.

8e81c67

- Simplify Gravsearch paging.
- Add design docs.


          fix(knora-ontologies): Remove extra rdfs:comment.

65b9380

Collaborator

daschbot commented Mar 26, 2020

This pull request has been mentioned on Discuss DaSCH. There might be relevant details there:

https://discuss.dasch.swiss/t/knora-api-js-lib-pr-changing-public-api/158/2

Collaborator

daschbot commented Mar 27, 2020

This pull request has been mentioned on Discuss DaSCH. There might be relevant details there:

https://discuss.dasch.swiss/t/knora-api-js-lib-pr-changing-public-api/158/5


          Merge branch 'develop' into wip/1543-forbidden-resource

1347d64

# Conflicts:
#	docs/src/paradox/05-internals/design/api-v2/gravsearch.md
#	webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala

benjamingeer requested a review from SepidehAlassi

April 3, 2020 12:15


          Merge branch 'develop' into wip/1543-forbidden-resource

6db2513

Collaborator

daschbot commented Apr 6, 2020

This pull request has been mentioned on Discuss DaSCH. There might be relevant details there:

https://discuss.dasch.swiss/t/knora-api-js-lib-pr-changing-public-api/158/9

SepidehAlassi suggested changes

View reviewed changes

Contributor

SepidehAlassi left a comment

Please resolve the failing test related to JS-Lib. As you have mentioned in the "Proposal: Analysis of Dependencies ..." document,

No subsystem should have to delay merging a PR until some other system can support that PR.

I suggest that, in another branch/PR, you either remove these tests or replace them with tests to check the compatibility of releases, not the compatibility of commits, as you suggested yourself. After that PR is merged to the develop branch, with a control merge, the problem of failing tests of this PR must be solved.

docs/src/paradox/05-internals/design/api-v2/gravsearch.md Outdated Show resolved Hide resolved

.../main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala Show resolved Hide resolved

.../main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala Show resolved Hide resolved

.../main/scala/org/knora/webapi/messages/v2/responder/resourcemessages/ResourceMessagesV2.scala Show resolved Hide resolved

webapi/src/main/scala/org/knora/webapi/util/ConstructResponseUtilV2.scala Outdated

                           val transformedValuePropertyAssertions: RdfPropertyValues = resource.valuePropertyAssertions.map {
                               case (propIri, values) =>
-                                  val transformedValues = values.map {
-                                      value =>
+                                  val transformedValues: Seq[ValueRdfData] = values.foldLeft(Vector.empty[ValueRdfData]) {

Contributor

SepidehAlassi Apr 6, 2020

Please extract this foldleft operation into a function to increase readability. It is 26 lines with 3 levels of indentation (nested conditions)

Author

benjamingeer Apr 7, 2020

The whole splitMainResourcesAndValueRdfData method has always been a headache for me to read: too long and too much nesting. Each time I have to work on it, I try to make it a bit clearer, but it really needs to be split up into much smaller pieces. I was going to ask you to refactor it with Tobias, since he's the author. :) For now, I've just moved the nestResources method out to the top level, and I've moved the foldLeft operation into another function, transformValuesByNestingResources (in 6e96460). But even after doing that, splitMainResourcesAndValueRdfData is still 273 lines long. :(

Contributor

SepidehAlassi Apr 7, 2020

For now, I've just moved the nestResources method out to the top level, and I've moved the foldLeft operation into another function, transformValuesByNestingResources (in 6e96460). But even after doing that, splitMainResourcesAndValueRdfData is still 273 lines long. :(

It took me really long time to understand what the splitMainResourcesAndValueRdfData does, but at least now with transformValuesByNestingResources it is a tiny bit easier to read.

webapi/src/main/scala/org/knora/webapi/util/ConstructResponseUtilV2.scala Outdated

                                                           nestedResource = Some(dependentResource)
                                                       )
+                                                  } else if (dependentResourceIrisNotVisible.contains(dependentResourceIri)) {

Contributor

SepidehAlassi Apr 6, 2020

then this means that previously there was no difference between the deleted dependant resource and forbidden one, in both cases link value was returned, but now if the dependant resource is forbidden (i.e. user does not have permission to see), then it is skipped. If the dependant resource is deleted, the link value is however returned. Have I understood it right?
If so, when a user requests a resource, the link value property itself will not be visible to the user either if the user does not have permission to see the dependent resource, right?

Author

benjamingeer Apr 7, 2020

now if the dependant resource is forbidden (i.e. user does not have permission to see), then it is skipped. If the dependant resource is deleted, the link value is however returned. Have I understood it right?

Yes. Do you think this makes sense? Or should we do the same thing in both cases?

If so, when a user requests a resource, the link value property itself will not be visible to the user either if the user does not have permission to see the dependent resource, right?

Right, it should not be visible. I will check and see if that's what really happens.

Author

benjamingeer Apr 7, 2020 •

edited

Loading

It turns out that the link value property was not filtered out if we filtered out all its values. I've fixed this and added a test in 35fafe4. Good catch, thanks!

Author

benjamingeer Apr 7, 2020

We have to return a LinkValue that points to a deleted resource, so the user can change the link to point to a different resource. In the case of a LinkValue that points to a resource that the user doesn't have permission to see, I filtered it out because @loicjaouen said he preferred it that way. I think it makes sense because there isn't anything that the user could do with that link.

Contributor

SepidehAlassi Apr 7, 2020 •

edited

Loading

Yes. Do you think this makes sense? Or should we do the same thing in both cases?

yes, I believe it makes sense as it is now. In this way, the difference between a deleted dependant resource and a forbidden one is clear and the user will not be confused by seeing a link value but not its dependant resource.

Author

benjamingeer Apr 7, 2020

OK great. In b990663 I've added a test that gets a link to a deleted resource, to check that the link value is returned correctly.

Contributor

SepidehAlassi Apr 7, 2020

Looks good to me!

webapi/src/main/scala/org/knora/webapi/util/ConstructResponseUtilV2.scala Outdated Show resolved Hide resolved

webapi/src/main/scala/org/knora/webapi/util/ConstructResponseUtilV2.scala Show resolved Hide resolved

webapi/src/main/scala/org/knora/webapi/util/StringFormatter.scala Outdated Show resolved Hide resolved

Author

benjamingeer commented Apr 6, 2020

I suggest that, in another branch/PR, you either remove these tests or replace them with tests to check the compatibility of releases, not the compatibility of commits, as you suggested yourself.

I actually have no idea what these tests do, how they work, or even where the code is that runs them. @tobiasschweizer and @subotic set this up and I don't know if it's documented anywhere. I could probably figure out how to turn them off, but maybe @tobiasschweizer and @subotic would be better able to deal with this properly.

After that PR is merged to the develop branch, with a control merge, the problem of failing tests of this PR must be solved.

Does this mean I can't merge this PR until someone writes a whole new testing framework for compatibility between Knora and knora-api-js-lib? That sounds like it could take a long time.

Contributor

tobiasschweizer commented Apr 7, 2020

@SepidehAlassi

I suggest that, in another branch/PR, you either remove these tests or replace them with tests to check the compatibility of releases, not the compatibility of commits, as you suggested yourself.

I actually have no idea what these tests do, how they work, or even where the code is that runs them. @tobiasschweizer and @subotic set this up and I don't know if it's documented anywhere. I could probably figure out how to turn them off, but maybe @tobiasschweizer and @subotic would be better able to deal with this properly.

After that PR is merged to the develop branch, with a control merge, the problem of failing tests of this PR must be solved.

Does this mean I can't merge this PR until someone writes a whole new testing framework for compatibility between Knora and knora-api-js-lib? That sounds like it could take a long time.

I think we should have a look at this today. We could decide on a good time at the end of the Scrum meeting.

Contributor

SepidehAlassi commented Apr 7, 2020 •

edited

Loading

Does this mean I can't merge this PR until someone writes a whole new testing framework for compatibility between Knora and knora-api-js-lib? That sounds like it could take a long time.

No, what I mean is that: Since the problems with tests should be resolved anyway before the merge, I suggest you solve the problem in another branch that can be reviewed fast, (even if solving the test problems means removing the tests) then merge it to develop first, then merge the develop branch into this branch.

Author

benjamingeer commented Apr 7, 2020

I suggest you solve the problem in another branch that can be reviewed fast, (even if solving the test problems means removing the tests) then merge it to develop first, then merge the develop branch in this branch.

OK will do.


          refactor: Fixes for review.

6e96460

Author

benjamingeer commented Apr 7, 2020

@tobiasschweizer

I think we should have a look at this today. We could decide on a good time at the end of the Scrum meeting.

OK great, thanks.

Author

benjamingeer commented Apr 7, 2020

@tobiasschweizer You asked for a copy-and-paste of the failing JS lib tests in this PR:

**************************************************
*                    Failures                    *
**************************************************

1) workspace-project App perform a fulltext search
  - Expected '15' to equal '21'.

2) workspace-project App perform a fulltext search count query
  - Expected '15' to equal '21'.

3) workspace-project App perform a label search
  - Expected '15' to equal '21'.

4) workspace-project App perform an extended search
  - Expected '23' to equal '25'.

Contributor

tobiasschweizer commented Apr 7, 2020 •

edited

Loading

@tobiasschweizer You asked for a copy-and-paste of the failing JS lib tests in this PR:

**************************************************
*                    Failures                    *
**************************************************

1) workspace-project App perform a fulltext search
  - Expected '15' to equal '21'.

2) workspace-project App perform a fulltext search count query
  - Expected '15' to equal '21'.

3) workspace-project App perform a label search
  - Expected '15' to equal '21'.

4) workspace-project App perform an extended search
  - Expected '23' to equal '25'.

Ok, this is because knora-api returns less results now. I can easily fix this in a separate knora-api-js-lib PR, so then the tests pass again in both repos. In the meantime please go ahead as agreed.

edit: I think I don't have to check for a certain number of results at all, just have to check that there is a number returned.

Benjamin Geer added 2 commits

April 7, 2020 11:38


          fix(api-v2): If we filtered out all the link values of a property, fi…

35fafe4

…lter out the property.


          test(api-v2): Test getting a resource with a link to a deleted resource.

b990663

Author

benjamingeer commented Apr 7, 2020

@SepidehAlassi Is this PR OK to merge now? If so, please don't forget to click the "approve" button.

Contributor

SepidehAlassi commented Apr 7, 2020

@SepidehAlassi Is this PR OK to merge now? If so, please don't forget to click the "approve" button.

yes, I believe it is. :-)

SepidehAlassi approved these changes

View reviewed changes

Author

benjamingeer commented Apr 7, 2020

@SepidehAlassi Many thanks for the review!

benjamingeer merged commit 992596e into develop

benjamingeer deleted the wip/1543-forbidden-resource branch

April 7, 2020 17:22

tobiasschweizer mentioned this pull request

Test Framework: Do Not Check for Exact Number of Search Results in E2E Tests dasch-swiss/dsp-js-lib#171

Merged

benjamingeer mentioned this pull request

feat(upgrade): Update repository on startup #1643

Merged

10 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels