forked from data-integrations/wrangler
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
16 changed files
with
760 additions
and
1 deletion.
There are no files selected for viewing
121 changes: 121 additions & 0 deletions
121
wrangler-transform/src/e2e-test/features/WranglerUI/DataTypeParsers.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| # Copyright © 2024 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: datatype parsers | ||
|
|
||
| @BQ_SOURCE_TS_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse timestamp directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "create_date" and apply directive: "Parse" as "Simple date" with: "yyyy-MM-dd" option | ||
| Then Expand dropdown column: "update_date" and apply directive: "Parse" as "Simple date" with: "yyyy-MM-dd" option | ||
| Then Enter directive from CLI "parse-timestamp :time" | ||
| Then Enter directive from CLI "parse-as-currency :price :newprice" | ||
| Then Enter directive from CLI "format-as-currency :newprice :format_price" | ||
| Then Enter directive from CLI "diff-date :create_date :update_date :diff_date" | ||
| Then Enter directive from CLI "timestamp-to-datetime :update_date" | ||
| Then Enter directive from CLI "rename :newprice :id" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_Timestamp" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully | ||
|
|
||
|
|
||
| @BQ_SOURCE_DATETIME_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse datetime directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "timestamp" and apply directive: "Parse" with directive type: "Datetime" and select: "Custom format" and enter: "yyyy-MM-dd'T'HH:mm:ssX'['z']'" | ||
| Then Enter directive from CLI "current-datetime :create_date" | ||
| Then Enter directive from CLI "datetime-to-timestamp :timestamp" | ||
| Then Enter directive from CLI "format-datetime :create_date 'y'" | ||
| Then Enter directive from CLI "format-date :timestamp yyyy-mm-dd" | ||
| Then Enter directive from CLI "rename timestamp timecolumn" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_DatetimeNew" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
65 changes: 65 additions & 0 deletions
65
wrangler-transform/src/e2e-test/features/WranglerUI/ParseAsAvro.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Copyright © 2024 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: Wrangler - Run time scenarios for Parse Avro | ||
|
|
||
| @BQ_SOURCE_AVRO_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse avro directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "body" and apply directive: "Parse" as "Avro" | ||
| Then Enter directive from CLI "filter-rows-on condition-true name == null || name =~ '^\W*$'" | ||
| Then Enter directive from CLI "cut-character name nameNew 2-3" | ||
| Then Expand dropdown column: "value" and apply directive: "ChangeDataType" as "Integer" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_avro" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
71 changes: 71 additions & 0 deletions
71
wrangler-transform/src/e2e-test/features/WranglerUI/ParseAsCsv.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Copyright © 2024 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: Wrangler - Run time scenarios for parse csv using UI | ||
|
|
||
| @BQ_SOURCE_CSV_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse csv directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "body" and apply directive: "Parse" as "CSV" with: "Comma" option | ||
| Then Expand dropdown column: "body_3" and apply directive: "FillNullOrEmptyCells" as "shubh" | ||
| Then Enter directive from CLI "rename body_1 new_id" | ||
| Then Enter directive from CLI "quantize body_4 body_q 1:2=20,3:4=40" | ||
| Then Expand dropdown column: "body_4" and apply directive: "ChangeDataType" as "Integer" | ||
| Then Enter directive from CLI "columns-replace s/^new_//g" | ||
| Then Enter directive from CLI "set-headers :abc" | ||
| Then Enter directive from CLI "change-column-case uppercase" | ||
| Then Enter directive from CLI "cleanse-column-names " | ||
| Then Enter directive from CLI "split-to-rows :id '#'" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_csv" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
70 changes: 70 additions & 0 deletions
70
wrangler-transform/src/e2e-test/features/WranglerUI/ParseAsExcel.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # Copyright © 2024 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: Parse as excel | ||
|
|
||
| @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse Excel directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTableExcel" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTableExcel" | ||
| Then Enter directive from CLI "parse-as-excel :body '0' true" | ||
| Then Expand dropdown column: "name" and apply directive: "CopyColumn" as "copiedname" | ||
| Then Enter directive from CLI "merge name bkd uniquenum ','" | ||
| Then Enter directive from CLI "rename bkd rollno" | ||
| Then Expand dropdown column: "fwd" and apply directive: "DeleteColumn" | ||
| Then Select checkbox on two columns: "id" and "rollno" | ||
| Then Expand dropdown column: "id" and apply directive: "SwapTwoColumnNames" | ||
| Then Enter directive from CLI "split-to-rows :name 'o'" | ||
| Then Enter directive from CLI "filter-rows-on condition-false rollno !~ '2.0'" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_excel" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
Oops, something went wrong.