Unauthorized Error when Loading Large Datasets in Metadata Editor

[vchendrix]

When loading a dataset with over 100 data files in the Metadata Editor, users may encounter an error message "You are not authorized to edit this data set." However, this error message is not always displayed, and the page may appear to load normally. Upon inspecting the JavaScript console, a 401 Unauthorized error is visible.

Steps to Reproduce:

1. Load a dataset with over 100 data files in the Metadata Editor.
2. Observe the error message "You are not authorized to edit this data set." (if displayed).
3. Check the JavaScript console for a 401 Unauthorized error.

Expected Behavior:

The Metadata Editor should load the dataset without errors, regardless of the number of data files.

Actual Behavior:

The Metadata Editor displays an error message or fails to load the dataset, with a 401 Unauthorized error visible in the JavaScript console.

Additional Context:

• The issue may be related to the concurrent access of the meta endpoint for all 100 files.
• The error is intermittent, and the page may load normally on some attempts.

Screen shot of error

Screencast of Issue
https://drive.google.com/file/d/1gMWnKKXP0esWlOtjtaNQHS67xHUJ3YQ4/view?usp=sharing

Example Error Information:

• Request URL: https://data.ess-dive.lbl.gov/catalog/d1/mn/v2/meta/ess-dive-452f59b6282987d-20231130T213150897232
• Request Method: GET
• Status Code: 401 Unauthorized

[mburrus]

I'm seeing this error on another private dataset with over 100 files: https://data.ess-dive.lbl.gov/view/ess-dive-161d0c0f88a0849-20240815T185407332338

Error messages I'm getting:
It's a mix of 200, 401, 501, and (failed)net::ERR_HTTP2_SERVER_REFUSED_STREAM errors across the data file PIDs.

Screenshot:

More Context

• I was able to open an edit session for this dataset on September 30th and did so without hitting this error. On that attempt, I was trying to re-save the access policy from the share panel as a way for the files to inherit the metadata access policy. Not all the data files were updated and a number of them responded with a red exclamation point.
• Before this attempt, I had tried to update the rightsHolder from the backend and hit some kind of concurrency error.

[mbjones]

@vchendrix @mburrus Can you describe the access policies on all of the objects in this package?

• Does your logged in ORCID have write access to all of the objects in this package?
• Are all of the objects set with identical access policies?
  • if yes, then does the logged in ORCID have write access to all objects (metadata, resource map, data)?
  • If no, then:
    • are the metadata file and resource map file writable by the logged in ORCID?
    • are all of the data files the same policy, and what permissions does your ORCID have?

@robyngit @rushirajnenuji the http2 error may be a new side-effect of enabling HTTP/2 as a more efficient protocol on servers lately. Maybe @vchendrix can let us know if they support clinet requests for only HTTP/1.1 or also HTTP/2. Some of our test servers at NCEAS use HTTP/2, but AFAIK none of our production servers have it enabled yet.

[vchendrix]

@vchendrix @mburrus Can you describe the access policies on all of the objects in this package?

• Does your logged in ORCID have write access to all of the objects in this package?
  Yes.

• Are all of the objects set with identical access policies?
  Yes.
  files_in_dataset.csv

• if yes, then does the logged in ORCID have write access to all objects (metadata, resource map, data)?
  Yes.

curl -H "Authorization: Bearer $ESS_DIVE_AUTH_TOKEN" "https://data.ess-dive.lbl.gov/catalog/d1/mn/v2/query/solr?q=id:ess-dive-3a48ab5f69ecf8d-20240108T174327967&fl=id,writePermission&wt=json" 
{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"id:ess-dive-3a48ab5f69ecf8d-20240108T174327967",
      "fl":"id,writePermission",
      "wt":"javabin",
      "version":"2"}},
  "response":{"numFound":1,"start":0,"numFoundExact":true,"docs":[
      {
        "id":"ess-dive-3a48ab5f69ecf8d-20240108T174327967",
        "writePermission":["CN=ess-dive-admins,DC=dataone,DC=org",
          "CN=watershed-function-sfa-admin,DC=dataone,DC=org",
          "CN=urn:node:ESS_DIVE,DC=dataone,DC=org"]}]
  }}

• If no, then:

  • are the metadata file and resource map file writable by the logged in ORCID?
  • are all of the data files the same policy, and what permissions does your ORCID have?

@robyngit @rushirajnenuji the http2 error may be a new side-effect of enabling HTTP/2 as a more efficient protocol on servers lately. Maybe @vchendrix can let us know if they support clinet requests for only HTTP/1.1 or also HTTP/2. Some of our test servers at NCEAS use HTTP/2, but AFAIK none of our production servers have it enabled yet.

Our services support HTTP/2 and HTTP/1.1

% curl -s -I -X HEAD  https://data.ess-dive.lbl.gov 
HTTP/2 200 
date: Tue, 15 Oct 2024 18:21:02 GMT
content-type: text/html
content-length: 10352
set-cookie: INGRESSCOOKIE=19cdfff91257311df6e1f2f92cc10ee1|47d24e7c0dbc2412b1cf3a747b30e59a; Path=/; Secure; HttpOnly
x-frame-options: SAMEORIGIN
last-modified: Fri, 23 Aug 2024 21:21:03 GMT
etag: "2870-620605a3a8dc0"
accept-ranges: bytes
access-control-allow-origin: 
access-control-allow-headers: Authorization, Content-Type, Origin, Cache-Control
access-control-allow-methods: GET, POST, PUT, OPTIONS
access-control-allow-credentials: true
strict-transport-security: max-age=15724800; includeSubDomains

(base) val@vchendrix ~ % curl -s -I -X HEAD --http1.1 https://data.ess-dive.lbl.gov
HTTP/1.1 200 OK
Date: Tue, 15 Oct 2024 18:21:53 GMT
Content-Type: text/html
Content-Length: 10352
Connection: keep-alive
Set-Cookie: INGRESSCOOKIE=29a1440a13a68111b1a3b64412631550|47d24e7c0dbc2412b1cf3a747b30e59a; Path=/; Secure; HttpOnly
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 23 Aug 2024 21:21:03 GMT
ETag: "2870-620605a3a8dc0"
Accept-Ranges: bytes
Access-Control-Allow-Origin: 
Access-Control-Allow-Headers: Authorization, Content-Type, Origin, Cache-Control
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS
Access-Control-Allow-Credentials: true
Strict-Transport-Security: max-age=15724800; includeSubDomains

[mburrus]

Hi @mbjones I have a follow up on the dataset that Val provided details for.

Considering that the unauthorized error is intermittently appearing and sometimes the user can edit the dataset, I told the user that they should go ahead and reload the edit session until it works. They were able to load the edit session eventually, but then they encountered an unexpected error message when they attempted to submit changes and their dataset was corrupted.

Here are the steps they took:

1. Reload the submit URL until it works
2. Change some metadata fields
3. Upload 6 new files. 4/6 files uploaded successfully with a check mark. 2/6 files failed to upload and had a red exclamation mark.
4. Click "Submit Dataset"
5. Submission failed. See error message on top of webpage that says: The requested identifier <PID> is already used by another data object and therefore can not be used for this object...
6. Go back to view landing page and see that files are no longer listed in the file table. I confirmed that the resource map is now missing.

Here's the quote from the user:

While it seemed that refreshing a few times worked to load the dataset, upon submission of my edits, I received an error for two of the new files I was attempting to upload (see screenshots). It now appears that some of the edits were saved (e.g., abstract, title), however, I am no longer able to see the files at the top of my dataset under "Files in this dataset" ( I see these listed at the bottom of my dataset).

Screenshots:

[vchendrix]

@mbjones Looking a little more into this issue when loading the data table in the editor. Looked at the /meta calls that were returning 401 errors and they don't seem to be authenticating the token correctly. The token is there and it is valid but the following Metacat error is logged.

Error for https://data.ess-dive.lbl.gov/catalog/d1/mn/v2/meta/ess-dive-5c5a631453d321e-20231130T213147717572

2024-10-18T21:41:24.459477371Z metacat 20241018-21:41:24: [ERROR]: D1ResourceHandler: Serializing exception with code 401: READ not allowed on ess-dive-0c7f0edc620a810-20231130T213140216564 for subject[s]: public; authenticatedUser; http://orcid.org/0000-0001-9061-8952;  [edu.ucsb.nceas.metacat.restservice.D1ResourceHandler:serializeException:591]
org.dataone.service.exceptions.NotAuthorized: READ not allowed on ess-dive-0c7f0edc620a810-20231130T213140216564 for subject[s]: public; authenticatedUser; http://orcid.org/0000-0001-9061-8952; 
2024-10-18T21:41:24.459482481Z 	at edu.ucsb.nceas.metacat.dataone.D1AuthHelper.prepareAndThrowNotAuthorized(D1AuthHelper.java:461) ~[metacat.jar:?]

[robyngit]

they encountered an unexpected error message when they attempted to submit changes and their dataset was corrupted.

@mburrus was this also for a data package with 100+ files? We are investigating whether MetacatUI is creating faulty resource maps. If that's the case, it would be uploaded successfully but not added to the solr index as a resource map. So the object would exist in Metacat, but would appear like a lone data object. I'm wondering if you would be able to check whether any objects like this exist for the users that lost their resource maps in the process of updating their datasets?

[vchendrix]

@robyngit After some investigation with @taojing2002 on Friday, we discovered that there may be some issues with with the CN resolving the user's permissions if they have edit permissions by way of a group as evidenced by my previous comment.

We did a quick little test where I gave myself permissions directly and the 401 errors went away. However, another issue still remains. I am still getting many errors in the network tab of Chrome: "(failed) net::ERR_FAILED" for many of the 100 simultaneous HTTP GET calls that are sent.

The net::ERR_FAILED error in Chrome can be caused by various issues, including network problems, server-side issues, or client-side issues. Given that there are 100 simultaneous HTTP GET calls, it is possible that the load balancer or web server is treating this as a DDoS attack or simply cannot handle the load. We do not have access to the load balancer or ingress logs to determinie if this is the case. However, this could be addrressed with Client-Side Throttling to limit the number of simultaneous requests. This can help prevent overwhelming the server.

Another thought, I have noticed that the metadata from these calls is being used to populate the data table columns. Is it possible to retrieve this information from solr in less calls instead?

Network Errors results in Size and Type metadata missing