Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for syncing Assessments from OneTrust to Transcend #376

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 77 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
1dfa3c3
create flattenOneTrustAssessment helper
abrantesarthur Jan 9, 2025
de282db
flatten questionReponses
abrantesarthur Jan 10, 2025
10c1998
create helper getOneTrustRisk
abrantesarthur Jan 10, 2025
f68e17c
update return type of getOneTrustRIsk
abrantesarthur Jan 10, 2025
9333325
convert OneTrust interfaces to codecs
abrantesarthur Jan 10, 2025
559ab87
fix codecs
abrantesarthur Jan 10, 2025
35bab06
fix codecs
abrantesarthur Jan 10, 2025
5d9def9
create flattenOneTrustSectionHeaders helper
abrantesarthur Jan 10, 2025
4822fba
improve codecs and create flattenOneTrustQuestions helper
abrantesarthur Jan 10, 2025
5636c78
improve variable names
abrantesarthur Jan 10, 2025
2ca1f64
create extractProperties helper
abrantesarthur Jan 11, 2025
531a240
implement flattenOneTrustNestedQuestionsOptions
abrantesarthur Jan 11, 2025
ab072df
improve flattenList
abrantesarthur Jan 11, 2025
edd5c70
improve flattenList
abrantesarthur Jan 11, 2025
22ae8b1
update flattenOneTrustNestedQuestionsOptions to use aggregateObjects
abrantesarthur Jan 11, 2025
460f25c
more changes
abrantesarthur Jan 11, 2025
ccd64ae
update const flattenOneTrustSectionHeaders = (
abrantesarthur Jan 11, 2025
f90fcd5
update const flattenOneTrustSectionHeaders = (
abrantesarthur Jan 11, 2025
3b08994
commit
abrantesarthur Jan 11, 2025
e80c95e
commit
abrantesarthur Jan 11, 2025
ec8e77c
create more helpers and add tests
abrantesarthur Jan 11, 2025
35d5fd7
create enrichRiskStatisticsWithDefault
abrantesarthur Jan 11, 2025
c0e9d26
fix bug
abrantesarthur Jan 11, 2025
930a1ba
remove extra test
abrantesarthur Jan 11, 2025
6538e41
create more codecs
abrantesarthur Jan 12, 2025
bb8dd28
flatten risks
abrantesarthur Jan 12, 2025
e104337
flatten approvers, respondents, and primaryEntityDetails
abrantesarthur Jan 12, 2025
ee8b072
update flattenOneTrustAssessment type
abrantesarthur Jan 12, 2025
d6052b1
create DEFAULT_ONE_TRUST_COMBINED_ASSESSMENT
abrantesarthur Jan 12, 2025
b1c5f6b
add comments
abrantesarthur Jan 12, 2025
ae91ee2
update
abrantesarthur Jan 12, 2025
80c3f72
improve createDefatulCodec
abrantesarthur Jan 12, 2025
2c12a0e
add missing fields to codec
abrantesarthur Jan 13, 2025
24841f1
fix codecs
abrantesarthur Jan 13, 2025
8480eaf
undo changes to cli-pull-ot
abrantesarthur Jan 13, 2025
bd5842c
improve codecs
abrantesarthur Jan 13, 2025
b26cb56
done
abrantesarthur Jan 13, 2025
3ac69be
update writeOneTrustAssessment to write in csv format
abrantesarthur Jan 13, 2025
f9be7b6
import assessment types from privacy types
abrantesarthur Jan 13, 2025
ad32c0a
update type-utils
abrantesarthur Jan 13, 2025
2b05b5d
remove enrich helpers
abrantesarthur Jan 14, 2025
74666c2
import createDefaultCodec from type-utils
abrantesarthur Jan 14, 2025
da299c3
rename cli-pull-ot -> cli-sync-ot
abrantesarthur Jan 14, 2025
799e41c
add dryRun argument
abrantesarthur Jan 14, 2025
30f3a4c
add dryRun argument
abrantesarthur Jan 14, 2025
239a72d
move some logic from writeOneTrustAssessment to cli-sync-ot
abrantesarthur Jan 14, 2025
ae5522f
ship more improvements to writeOneTrustAssessment
abrantesarthur Jan 14, 2025
4b428dc
reorganize folder structure
abrantesarthur Jan 14, 2025
13e80e9
create syncOneTrustAssessments and enrichOneTrustAssessment helpers
abrantesarthur Jan 14, 2025
d11609e
call syncOneTrustAssessments from cli-sync-ot
abrantesarthur Jan 14, 2025
7647c02
create oneTrustAssessmentToJson helper
abrantesarthur Jan 14, 2025
5dae37d
create oneTrustAssessmentToCsv helper
abrantesarthur Jan 14, 2025
4256788
create oneTrustAssessmentToCsvRecord helper
abrantesarthur Jan 14, 2025
6ec35ba
update readme
abrantesarthur Jan 14, 2025
5f94a17
move constants to heleprs
abrantesarthur Jan 14, 2025
df76064
add transcendUrl to the list of arguments
abrantesarthur Jan 14, 2025
b424c03
add ability to sync to Transcend
abrantesarthur Jan 14, 2025
414e84e
update error messages
abrantesarthur Jan 14, 2025
70956b6
update extractProperties documentation
abrantesarthur Jan 14, 2025
a8500c6
add fixme comment
abrantesarthur Jan 14, 2025
ae821ae
write docs for aggregateObjects
abrantesarthur Jan 14, 2025
ca7f66b
document flattenObject
abrantesarthur Jan 14, 2025
7692cd4
remove some TODOs
abrantesarthur Jan 14, 2025
eda427b
add TODOs
abrantesarthur Jan 14, 2025
f8ef92e
remove TODOs
abrantesarthur Jan 14, 2025
e4176be
add a fixme
abrantesarthur Jan 14, 2025
4409674
fix merge conflicts
abrantesarthur Jan 14, 2025
66bfc71
simplify flattenOneTrustAssessment
abrantesarthur Jan 14, 2025
b34e3f9
improve flattenOneTrustSections
abrantesarthur Jan 14, 2025
b65036b
improve flattenOneTrustSectionHeaders
abrantesarthur Jan 14, 2025
bd5e882
simplify flatten functions
abrantesarthur Jan 14, 2025
7b4ee5b
import aggregateObjects and flattenObject from @transcend/type-utils
abrantesarthur Jan 14, 2025
dedf1b8
use transposeObjectArray from @transcend/type-utils
abrantesarthur Jan 14, 2025
7a83891
improve flattenOneTrustAssessment
abrantesarthur Jan 14, 2025
148b0e4
update Readme
abrantesarthur Jan 14, 2025
de53f5e
update cli-sync-ot docs
abrantesarthur Jan 14, 2025
34de61b
update package.version
abrantesarthur Jan 14, 2025
4deaa2d
update commments
abrantesarthur Jan 16, 2025
cf23df8
remove potential bugs from flattenOneTrustAssessment
abrantesarthur Jan 16, 2025
fa17027
add fixmes
abrantesarthur Jan 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .pnp.cjs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.
Binary file not shown.
52 changes: 39 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@
- [Authentication](#authentication-34)
- [Arguments](#arguments-33)
- [Usage](#usage-34)
- [tr-sync-ot](#tr-sync-ot)
- [tr-build-xdi-sync-endpoint](#tr-build-xdi-sync-endpoint)
- [Authentication](#authentication-35)
- [Arguments](#arguments-34)
Expand Down Expand Up @@ -176,7 +177,7 @@ yarn add -D @transcend-io/cli

# cli commands available within package
yarn tr-pull --auth=$TRANSCEND_API_KEY
yarn tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
yarn tr-sync-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
yarn tr-push --auth=$TRANSCEND_API_KEY
yarn tr-scan-packages --auth=$TRANSCEND_API_KEY
yarn tr-discover-silos --auth=$TRANSCEND_API_KEY
Expand Down Expand Up @@ -217,7 +218,7 @@ npm i -D @transcend-io/cli

# cli commands available within package
tr-pull --auth=$TRANSCEND_API_KEY
tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
tr-sync-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
tr-push --auth=$TRANSCEND_API_KEY
tr-scan-packages --auth=$TRANSCEND_API_KEY
tr-discover-silos --auth=$TRANSCEND_API_KEY
Expand Down Expand Up @@ -577,9 +578,9 @@ tr-pull --auth=./transcend-api-keys.json --resources=consentManager --file=./tra

Note: This command will overwrite the existing transcend.yml file that you have locally.

### tr-pull-ot
### tr-sync-ot

Pulls resources from a OneTrust instance. For now, it only supports retrieving OneTrust Assessments. It sends a request to the [Get List of Assessments](https://developer.onetrust.com/onetrust/reference/getallassessmentbasicdetailsusingget) endpoint to fetch a list of all Assessments in your account. Then, it queries the [Get Assessment](https://developer.onetrust.com/onetrust/reference/exportassessmentusingget) and [Get Risk](https://developer.onetrust.com/onetrust/reference/getriskusingget) endpoints to enrich these assessments with more details such as respondents, approvers, assessment questions and responses, and assessment risks. Finally, it syncs the enriched resources to disk in the specified file and format.
Pulls resources from a OneTrust and syncs them to a Transcend instance. For now, it only supports retrieving OneTrust Assessments. It sends a request to the [Get List of Assessments](https://developer.onetrust.com/onetrust/reference/getallassessmentbasicdetailsusingget) endpoint to fetch a list of all Assessments in your account. Then, it queries the [Get Assessment](https://developer.onetrust.com/onetrust/reference/exportassessmentusingget) and [Get Risk](https://developer.onetrust.com/onetrust/reference/getriskusingget) endpoints to enrich these assessments with more details such as respondents, approvers, assessment questions and responses, and assessment risks. Finally, it syncs the enriched resources to disk in the specified file and format.

This command can be helpful if you are looking to:

Expand All @@ -596,22 +597,47 @@ In order to use this command, you will need to generate a OneTrust OAuth Token w

To learn how to generate the token, see the [OAuth 2.0 Scopes](https://developer.onetrust.com/onetrust/reference/oauth-20-scopes) and [Generate Access Token](https://developer.onetrust.com/onetrust/reference/getoauthtoken) pages.

If syncing the resources to Transcend, you will also need to generate an API key on the Transcend Admin Dashboard (https://app.transcend.io/infrastructure/api-keys).

The API key needs the following scopes when pushing the various resource types:

| Resource | Scope |
| ----------- | ------------------ |
| assessments | Manage Assessments |

#### Arguments

| Argument | Description | Type | Default | Required |
| ---------- | ------------------------------------------------------------------------------------------------- | ------- | ----------- | -------- |
| auth | The OAuth access token with the scopes necessary to access the OneTrust Public APIs. | string | N/A | true |
| hostname | The domain of the OneTrust environment from which to pull the resource (e.g. trial.onetrust.com). | string | N/A | true |
| file | Path to the file to pull the resource into. Its format must match the fileFormat argument. | string | N/A | true |
| fileFormat | The format of the output file. For now, only json is supported. | string | json | false |
| resource | The resource to pull from OneTrust. For now, only assessments is supported. | string | assessments | false |
| debug | Whether to print detailed logs in case of error. | boolean | false | false |
| Argument | Description | Type | Default | Required |
| ------------- | ------------------------------------------------------------------------------------------------- | ------------ | ------------------------ | -------- |
| hostname | The domain of the OneTrust environment from which to pull the resource (e.g. trial.onetrust.com). | string | N/A | true |
| oneTrustAuth | The OAuth access token with the scopes necessary to access the OneTrust Public APIs. | string | N/A | true |
| transcendAuth | The Transcend API Key to with the scopes necessary to access Transcend's Public APIs. | string | N/A | false |
| transcendUrl | URL of the Transcend backend. Use https://api.us.transcend.io for US hosting. | string - URL | https://api.transcend.io | false |
| file | Path to the file to pull the resource into. Its format must match the fileFormat argument. | string | N/A | false |
| fileFormat | The format of the output file. | string | csv | false |
| resource | The resource to pull from OneTrust. For now, only assessments is supported. | string | assessments | false |
| dryRun | Whether to export the resource to a file rather than sync to Transcend. | boolean | false | false |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth trying to convey that some arguments are conditionally required? E.g. I assume if you have --dryRun then file is required, and if you don't then transcendAuth/transcendUrl are required

| debug | Whether to print detailed logs in case of error. | boolean | false | false |

#### Usage

```sh
# Syncs all assessments from the OneTrust instance to Transcend
tr-sync-ot --hostname=trial.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --transcendAuth=$TRANSCEND_API_KEY
```

Alternatively, you can set dryRun to true and sync the resource to disk:

```sh
# Writes out file to ./oneTrustAssessments.csv
tr-sync-ot --hostname=trial.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --dryRun=true --file=./oneTrustAssessments.csv
```

You can also sync to disk in json format:

```sh
# Writes out file to ./oneTrustAssessments.json
tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=trial.onetrust.com --file=./oneTrustAssessments.json
tr-sync-ot --hostname=trial.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --dryRun=true --fileFormat=json --file=./oneTrustAssessments.json
```

### tr-push
Expand Down
8 changes: 4 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"author": "Transcend Inc.",
"name": "@transcend-io/cli",
"description": "Small package containing useful typescript utilities.",
"version": "6.13.0",
"version": "6.14.0",
"homepage": "https://github.com/transcend-io/cli",
"repository": {
"type": "git",
Expand All @@ -28,7 +28,6 @@
"tr-pull-consent-metrics": "./build/cli-pull-consent-metrics.js",
"tr-pull-consent-preferences": "./build/cli-pull-consent-preferences.js",
"tr-pull-datapoints": "./build/cli-pull-datapoints.js",
"tr-pull-ot": "./build/cli-pull-ot.js",
"tr-push": "./build/cli-push.js",
"tr-request-approve": "./build/cli-request-approve.js",
"tr-request-cancel": "./build/cli-request-cancel.js",
Expand All @@ -42,6 +41,7 @@
"tr-retry-request-data-silos": "./build/cli-retry-request-data-silos.js",
"tr-scan-packages": "./build/cli-scan-packages.js",
"tr-skip-request-data-silos": "./build/cli-skip-request-data-silos.js",
"tr-sync-ot": "./build/cli-sync-ot.js",
"tr-update-consent-manager": "./build/cli-update-consent-manager-to-latest.js",
"tr-upload-consent-preferences": "./build/cli-upload-consent-preferences.js",
"tr-upload-cookies-from-csv": "./build/cli-upload-cookies-from-csv.js",
Expand All @@ -68,9 +68,9 @@
"@transcend-io/handlebars-utils": "^1.1.0",
"@transcend-io/internationalization": "^1.6.0",
"@transcend-io/persisted-state": "^1.0.4",
"@transcend-io/privacy-types": "^4.103.0",
"@transcend-io/privacy-types": "^4.105.0",
"@transcend-io/secret-value": "^1.2.0",
"@transcend-io/type-utils": "^1.5.0",
"@transcend-io/type-utils": "^1.8.0",
"bluebird": "^3.7.2",
"cli-progress": "^3.11.2",
"colors": "^1.4.0",
Expand Down
76 changes: 0 additions & 76 deletions src/cli-pull-ot.ts

This file was deleted.

73 changes: 73 additions & 0 deletions src/cli-sync-ot.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env node
import { logger } from './logger';

import colors from 'colors';
import { parseCliSyncOtArguments, createOneTrustGotInstance } from './oneTrust';
import { OneTrustPullResource } from './enums';
import { syncOneTrustAssessments } from './oneTrust/helpers/syncOneTrustAssessments';
import { buildTranscendGraphQLClient } from './graphql';

/**
* Pull configuration from OneTrust down locally to disk
*
* Dev Usage:
* yarn ts-node ./src/cli-sync-ot.ts --hostname=customer.my.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --transcendAuth=$TRANSCEND_API_KEY
*
* Standard usage
* yarn cli-sync-ot --hostname=customer.my.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --transcendAuth=$TRANSCEND_API_KEY
*/
async function main(): Promise<void> {
const {
file,
fileFormat,
hostname,
oneTrustAuth,
transcendAuth,
transcendUrl,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so transcendAuth + transcendUrl are optional? If they don't pass it in we'll just save to disk?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, with a caveat: the criteria for saving to disk is that --dryRun=true, in which case we the parseCliSyncOtArguments helper also ensures that transcendAuth + transcendUrl were passed in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait i thought dryrun=true would mean don't upload to Transcend

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, you're right! I meant dryRun=false means ensuring that transcendAuth and transcendUrl were passed in!

resource,
debug,
dryRun,
} = parseCliSyncOtArguments();

// use the hostname and auth token to instantiate a client to talk to OneTrust
const oneTrust = createOneTrustGotInstance({ hostname, auth: oneTrustAuth });

try {
if (resource === OneTrustPullResource.Assessments) {
await syncOneTrustAssessments({
oneTrust,
file,
fileFormat,
dryRun,
...(transcendAuth && transcendUrl
? {
transcend: buildTranscendGraphQLClient(
transcendUrl,
transcendAuth,
),
}
: {}),
});
}
} catch (err) {
logger.error(
colors.red(
`An error occurred syncing the resource ${resource} from OneTrust: ${
debug ? err.stack : err.message
}`,
),
);
process.exit(1);
}

// Indicate success
logger.info(
colors.green(
`Successfully synced OneTrust ${resource} to ${
dryRun ? `disk at "${file}"` : 'Transcend'
}!`,
),
);
}

main();
28 changes: 28 additions & 0 deletions src/codecs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2073,3 +2073,31 @@ export const PathfinderPromptRunMetadata = t.partial({
export type PathfinderPromptRunMetadata = t.TypeOf<
typeof PathfinderPromptRunMetadata
>;

/** The columns of a row of a OneTrust Assessment form to import into Transcend. */
const OneTrustAssessmentColumnInput = t.intersection([
t.type({
/** The title of the column */
title: t.string,
}),
t.partial({
/** The optional value of the column */
value: t.string,
}),
]);

/** A row with information of the OneTrust assessment form to import into Transcend */
const OneTrustAssessmentRowInput = t.type({
/** A list of columns within this row. */
columns: t.array(OneTrustAssessmentColumnInput),
});

/** Input for importing multiple OneTrust assessment forms into Transcend */
export const ImportOnetrustAssessmentsInput = t.type({
/** 'The rows of the CSV file.' */
rows: t.array(OneTrustAssessmentRowInput),
});
/** Type override */
export type ImportOnetrustAssessmentsInput = t.TypeOf<
typeof ImportOnetrustAssessmentsInput
>;
13 changes: 13 additions & 0 deletions src/graphql/gqls/assessment.ts
Original file line number Diff line number Diff line change
Expand Up @@ -283,3 +283,16 @@ export const ASSESSMENTS = gql`
}
}
`;

export const IMPORT_ONE_TRUST_ASSESSMENT_FORMS = gql`
mutation TranscendCliImportOneTrustAssessmentForms(
$input: ImportOnetrustAssessmentsInput!
) {
importOneTrustAssessmentForms(input: $input) {
assessmentForms {
id
title
}
}
}
`;
Loading
Loading