forked from data-integrations/wrangler
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
15 changed files
with
788 additions
and
1 deletion.
There are no files selected for viewing
121 changes: 121 additions & 0 deletions
121
wrangler-transform/src/e2e-test/features/WranglerUI/DataTypeParsers.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| # Copyright © 2023 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: datatype parsers | ||
|
|
||
| @BQ_SOURCE_TS_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse timestamp directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "create_date" and apply directive: "Parse" as "Simple date" with "yyyy-MM-dd" option | ||
| Then Expand dropdown column: "update_date" and apply directive: "Parse" as "Simple date" with "yyyy-MM-dd" option | ||
| Then Enter directive from CLI "parse-timestamp :time" | ||
| Then Enter directive from CLI "parse-as-currency :price :newprice" | ||
| Then Enter directive from CLI "format-as-currency :newprice :format_price" | ||
| Then Enter directive from CLI "diff-date :create_date :update_date :diff_date" | ||
| Then Enter directive from CLI "timestamp-to-datetime :update_date" | ||
| Then Enter directive from CLI "rename :newprice :id" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_Timestamp" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully | ||
|
|
||
|
|
||
| @BQ_SOURCE_DATETIME_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse datetime directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "timestamp" and apply directive:"Parse" with directive type: "Datetime" and select "Custom format" and enter "yyyy-MM-dd'T'HH:mm:ssX'['z']'" | ||
| Then Enter directive from CLI "current-datetime :create_date" | ||
| Then Enter directive from CLI "datetime-to-timestamp :timestamp" | ||
| Then Enter directive from CLI "format-datetime :create_date 'y'" | ||
| Then Enter directive from CLI "format-date :timestamp yyyy-mm-dd" | ||
| Then Enter directive from CLI "rename timestamp timecolumn" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_DatetimeNew" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
65 changes: 65 additions & 0 deletions
65
wrangler-transform/src/e2e-test/features/WranglerUI/ParseAsAvro.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Copyright © 2023 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: Wrangler - Run time scenarios for Parse Avro | ||
|
|
||
| @BQ_SOURCE_AVRO_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse avro directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "body" and apply directive: "Parse" as "Avro" | ||
| Then Enter directive from CLI "filter-rows-on condition-true name == null || name =~ '^\W*$'" | ||
| Then Enter directive from CLI "cut-character name nameNew 2-3" | ||
| Then Expand dropdown column: "value" and apply directive: "Change data type" as "Integer" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_avro" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
118 changes: 118 additions & 0 deletions
118
wrangler-transform/src/e2e-test/features/WranglerUI/ParseAsCsv.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| # Copyright © 2023 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @Wrangler | ||
| Feature: Wrangler - Run time scenarios for parse csv using UI | ||
|
|
||
| @BQ_SOURCE_CSV_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST | ||
| Scenario: To verify User is able to run a pipeline using parse csv directive | ||
| Given Open Wrangler connections page | ||
| Then Click plugin property: "addConnection" button | ||
| Then Click plugin property: "bqConnectionRow" | ||
| Then Enter input plugin property: "name" with value: "bqConnectionName" | ||
| Then Replace input plugin property: "projectId" with value: "projectId" | ||
| Then Enter input plugin property: "datasetProjectId" with value: "projectId" | ||
| Then Override Service account details in Wrangler connection page if set in environment variables | ||
| Then Click plugin property: "testConnection" button | ||
| Then Verify the test connection is successful | ||
| Then Click plugin property: "connectionCreate" button | ||
| Then Verify the connection with name: "bqConnectionName" is created successfully | ||
| Then Select connection data row with name: "dataset" | ||
| Then Select connection data row with name: "bqSourceTable" | ||
| Then Verify connection datatable is displayed for the data: "bqSourceTable" | ||
| Then Expand dropdown column: "body" and apply directive: "Parse" as "CSV" with "Comma" option | ||
| Then Expand dropdown column: "body_3" and apply directive: "Fill null or empty cells" as "shubh" | ||
| Then Enter directive from CLI "rename body_1 new_id" | ||
| Then Enter directive from CLI "quantize body_4 body_q 1:2=20,3:4=40" | ||
| Then Expand dropdown column: "body_4" and apply directive: "Change data type" as "Integer" | ||
| Then Enter directive from CLI "columns-replace s/^new_//g" | ||
| Then Enter directive from CLI "set-headers :abc" | ||
| Then Enter directive from CLI "change-column-case uppercase" | ||
| Then Enter directive from CLI "cleanse-column-names " | ||
| Then Enter directive from CLI "split-to-rows :id '#'" | ||
| Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" | ||
| Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds | ||
| Then Expand Plugin group in the LHS plugins list: "Sink" | ||
| Then Select plugin: "BigQuery" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery2" | ||
| Then Click plugin property: "useConnection" | ||
| Then Click on the Browse Connections button | ||
| Then Select connection: "bqConnectionName" | ||
| Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" | ||
| Then Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Enter input plugin property: "table" with value: "bqTargetTable" | ||
| Then Validate "BigQuery" plugin properties | ||
| Then Close the Plugin Properties page | ||
| Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection | ||
| Then Save the pipeline | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_csv" | ||
| Given Open Wrangler connections page | ||
| Then Expand connections of type: "BigQuery" | ||
| Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" | ||
| Then Click plugin property: "Delete" button | ||
| Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully | ||
|
|
||
| # Given Open Datafusion Project to configure pipeline | ||
| # Then Select checkbox on two columns "city" and "firstname" | ||
| # Then Expand the dropdown to apply column transformations for column: "city" | ||
| ## Then Apply directive: "Delete selected columns" | ||
| # Then Expand the dropdown to apply column transformations for column: "city" | ||
|
|
||
| # Then Expand dropdown column: "city" and apply directive: "Parse" as "CSV" with "Tab" option | ||
| # Then Expand dropdown column: "city" and apply directive: "Delete column" | ||
| ## Then Apply directive: "Calculate" as "Character count" with "ankit" option | ||
| # Then Apply directive: "Extract fields" as "Using patterns" with "Date" option | ||
| # Then Apply directive: "Send to error" as "value is ISBN13" | ||
| # Then Apply directive: "Find and replace" as "anit" and enter text "baseball" | ||
|
|
||
| # Then Apply directive:"Set counter" and select "If condition is true" with condition "jexl ext" with option "1" and name "ankit" | ||
| # Then Apply directive:"Define variable" as "Custom condition" and enter "sasdf" with option "lastname" and value "ankit" | ||
| # Then Apply directive: "Filter" with value "value is empty" | ||
| # Then Apply directive:"Set counter" as "Always" and enter "ankit" and "1" | ||
| # Then Apply directive: "Filter" as "Custom condition" with "ankit" option | ||
| # Then Click on "Apply" button | ||
| # Then Apply directive: "Pa" as "" | ||
| # Then Apply directive: "Filter" | ||
| # Then Click on "Apply" button | ||
| # Then Apply directive: "Filter" as "Custom condition" and enter text "ankit" | ||
| # Then Click on scroll down option | ||
| # Then Apply directive: "Calculate" as "Character count" with "abc" option | ||
| # Then Apply directive:"Parse" as "Simple date" and enter "abcddfd" and option "Custom format" | ||
| # Then Apply directive: "Parse" as "Excel" with "5" option | ||
| # Then Apply directive: "Copy column" with value "adfsfsdfds" | ||
| # Then Apply directive: "Custom transform" | ||
| # Then Apply directive: "Change data type" as "Boolean" | ||
| # Then Click on "Extract" button | ||
| # Then Apply directive: "Decode" as "Base64" | ||
| # Then Click on "Apply" button | ||
| # Then Expand the dropdown to apply column transformations for column: "city" | ||
| # Then Apply directive: "Delete column" | ||
| # Then Expand the dropdown to apply column transformations for column: "city" | ||
| # Then Apply directive: "change-data-type-directive" as "String" | ||
| # Then Expand the dropdown to apply column transformations for column: "city" | ||
| # Then Apply directive: "format-directive" as "UPPERCASE" | ||
| # Then Expand the dropdown to apply column transformations for column: "city" | ||
| # Then Apply directive: "Parse" as "CSV" with "Tab" option | ||
| # Then Click on "Apply" button | ||
| # Then Click on scroll down option | ||
| # Then Apply directive: "encode-directive" as "Base64" | ||
| # Then Apply directive:"Format" as "Concatenate" and enter "ankit" at "at the beginning" | ||
|
|
Oops, something went wrong.