Workflow widget node binding DSS result set node
Visualis, as a visual report system, has been connected with DSS workflow at present. You can create a visualis node by dragging it for visual development. For the traditional use of visualisation, the visualization component widget needs to create a view like component view to provide the data source of graphic rendering. For widgets, as long as the result set is a structured data set, it can be used as the data source of widgets for visual graphics development.
If you need to use visualis nodes in DSS, you need to refer to [visualis appconn installation and deployment document] (). At present, the data nodes of DSS, as long as the nodes that can generate structured data result sets, can support successful binding with visualis widget nodes. For the description of binding DSS data nodes to widgets, please refer to the following table:
| Node name | Task type | remarks |
|---|---|---|
| sql | Spark SQL task | Multiple result sets are not supported |
| pyspark | Python spark task | See remarks |
| hql | Hive SQL task | Multiple result sets are not supported |
For SQL nodes and HQL nodes, as long as they are not multi result set queries, after the execution is completed, the result set generated by them is registered in the CS service of linkis and a temporary table is generated. The dataframe result set is registered in the service and stored as a temporary table Note when using the pyspark node as the upstream table. When using spark Python to implement data query and as the data source of the widget, you need to generate a dataframe result set and call the show method, where the widget will display the dimension information of a DF. For a table, it belongs to a multi-dimensional table, that is, multiple columns.
df = spark.sql("select * from default.demo")
show(df) The following figure shows how to use the visualis node upstream of binding in DSS.
In the DSS workflow, after dragging the data development node, it will generate a_ tmp_ sql_ 5643_ The temporary table of RC1. When dragging a widget node to bind a data development node, in the DSS workflow JSON, the JSON of the widegt node will be set to configure the bindviewkey as the upstream bound data development node nodeid. The DSS backend will find its CS cache table through the bound nodeid, and when requesting to synchronously create a widget node of visualis, pass the CS ID of its CS table as the data source used by the widget node. The following figure shows the workflow parameter JSON generated by binding the upstream SQL node when the DSS drags and creates the widget node. Where bindviewkey is the node ID of the upstream SQL node.
{
"title": "widget_2919",
"bindViewKey": "22418bea-caec-4129-93ba-ce1938274b1c",
"desc": ""
} The creation process can be shown in the following figure:
DSS supports widget, display and dashboard nodes. Their crud and execution are connected with visualis. The implementation details of connecting with DSS are as follows. The logic related to the appconn of visualis that needs to be implemented on the DSS side is:
- Implement the projectcreationoperation in the specification, which is called when the DSS project is created. Call the project creation interface in the controller of visualis through HTTP. (the function of displaying the project list on the native homepage of visualis according to the DSS permission has not been realized yet)
- Implement the crud related operation interface of ref in the specification. It is called when the DSS node is created. First determine the specific node type to be created through parameters, and then call the widget, display or widget creation interface in the controller of visualis through HTTP.
- The creation of a widget does not call the default interface of the controller, but specifically defines the /widget/smartcreate interface. In the widgetresultfulapi, it handles some additional logic that needs to be processed after CS access and virtual view transformation. In addition, the widget itself will save the CSID of the current workflow as the basis for querying context information. Therefore, when the CSID itself changes, it is necessary to call the widget/setcontext interface to update the CSID recorded in the corresponding widget, otherwise the upstream table will not be found.
- In visualis, dashboard is actually a multi-layer structure, represented as dashboard portal dashboard. By default, a two-layer interface is created for node docking, that is, under the dashboard portal with the same name as the node, there is only one dashboard with the same name as the node. Therefore, when creating, it is necessary to call the two creation interfaces in turn to make them have the same name and are related to each other.
- Display in visualis is actually a two-layer structure, which is represented by display and display slide. They are one-to-one relationships. Therefore, when creating, you need to call the two creation interfaces in turn and make them interrelated.
- Implement the visualisrefexecutionoperation interface in the specification. When the DSS node executes, call the corresponding interface in visualis through HTTP to obtain the results.
- For widget execution, call the /visualis/widget/{id}/getdata interface in widgetresultfulapi to obtain the widget execution result. This interface is specifically implemented for interfacing with DSS. By parsing the config field of the widget, it simulates the query related parameters of the front-end splicing, and calls the background query interface to obtain the execution results. The result set of widget execution is the result set of spark SQL query it submits.
- To query the display/dashboard, call the preview interface of the corresponding controller to obtain the corresponding screenshot binary file as the record of the result set. The metadata of the result set needs to be obtained again through a special interface. The interface is defined in widgetresultfulapi. The interface format is /widget/{type}/{id}/metadata, where the type of display is display and the type of dashboard is portal. Note the ID here, and the dashboard should pass the ID of the corresponding dashboard portal. The content returned by the metadata interface is the JSON interface, which records the correspondence between the names of all widgets added to the display/dashboard and their fields and update time.
- Implements the operation interface for importing and exporting refs in the specification. When the DSS node performs import and export operations, it calls the corresponding interface in visualis through HTTP.
- The import/export interface in projectrestfulapi is the implementation of the import and export function of visualis. The export interface receives the project ID and the ID of the corresponding widget, display or dashboard as parameters, exports all the information as JSON structure, and returns resourceid and version after uploading to BML. The import interface receives the resourceid and version of the project ID and BML as parameters. After downloading the JSON structure from BML, it restores it to a specific entity and returns the corresponding relationship between the new and old IDs.
- It should be noted that on the appconn side, after successful import, the ID information in the original jobcontent needs to be updated and returned to the workflow for update. The visualis side requires the following related modifications:
- Access SSO specification. Since the visualis front-end and DSS share the user state, it is only necessary to implement SSO when the back-end interfaces call each other. The visualisuserinterceptor needs to be implemented to operate the user information in the HTTP session. Implement visualisssofilterinitializer, which is used to add SSO filter provided by DSS to the link of HTTP request processing of visualis. The modifyhttprequestwrapper is implemented to copy the cookie information provided by the DSS request to the cookie on the visualis side.
- Front end modification. In order to support unified front-end access in multiple environments, the front-end page captures the parameter env={env} through the URL, converts the parameter into a route label, and puts it into all subsequent interface requests starting from this page, so that the gateway can forward the request to the visualis background instance corresponding to the corresponding dev/prod environment according to the label.