-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into feat/ucs2-support
Signed-off-by: Trae Yelovich <[email protected]>
- Loading branch information
Showing
51 changed files
with
1,146 additions
and
91 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,8 @@ artifacts | |
| lib | ||
| node_modules | ||
| out | ||
| dist | ||
| typedoc | ||
| tsconfig.tsbuildinfo | ||
| zowe.config.json | ||
| zowe.schema.json | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,114 @@ | ||
| | API | Backend | Middleware | SDK | CLI | VSCE | | ||
| | ---------------- | ------- | ---------- | --- | --- | ---- | | ||
| | List data sets | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | List members | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | List USS files | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | List jobs | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | List spool files | ✅ | ✅ | ✅ | ❌ | ✅ | | ||
| | Read data set | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | Read USS file | ✅ | ✅ | ✅ | ❌ | ❌ | | ||
| | Read spool file | ✅ | ✅ | ✅ | ❌ | ✅ | | ||
| | Write data set | ✅ | ✅ | ✅ | ❌ | ❌ | | ||
| | Write USS file | ✅ | ✅ | ✅ | ❌ | ❌ | | ||
| | Get job JCL | ✅ | ✅ | ✅ | ❌ | ✅ | | ||
| # API Matrix | ||
|
|
||
| ### Legend | ||
|
|
||
| - ✅ Supported | ||
| - 🚧 Partially supported | ||
| - ❌ Not supported | ||
| - ➖ Not applicable | ||
| - _italic_ Target for MVP | ||
|
|
||
| ## Data Sets | ||
|
|
||
| | Operation | z/OSMF | FTP | Backend | Middleware | SDK | CLI | VSCE | | ||
| | ------------------------ | --------------- | --- | --------------- | ---------- | --- | --- | ---- | | ||
| | _List data sets_ | ✅ | ✅ | 🚧 <sup>1</sup> | ✅ | ✅ | ✅ | ✅ | | ||
| | _List data set members_ | ✅ | ✅ | 🚧 <sup>1</sup> | ✅ | ✅ | ✅ | ✅ | | ||
| | _Read data set/member_ | ✅ | ✅ | 🚧 <sup>2</sup> | ✅ | ✅ | ✅ | ✅ | | ||
| | _Write data set/member_ | ✅ | ✅ | 🚧 <sup>2</sup> | ✅ | ✅ | ❌ | ❌ | | ||
| | _Create data set_ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | _Delete data set_ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | _Delete data set member_ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Migrate data set | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Recall data set | 🚧 <sup>3</sup> | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | | ||
| | Delete migrated data set | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Rename data set | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Copy data set | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Invoke AMS (VSAM) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Search data sets | 🚧 <sup>4</sup> | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
|
|
||
| 1. Not all attributes are retrieved | ||
| 2. Streaming is not supported for large files | ||
| 3. Does not support some migration utilities like CA Disk | ||
| 4. Limited options compared to ISPF `srchfor` | ||
|
|
||
| ## USS Files | ||
|
|
||
| | Operation | z/OSMF | FTP | Backend | Middleware | SDK | CLI | VSCE | | ||
| | ------------------------ | ------ | --- | --------------- | ---------- | --- | --- | ---- | | ||
| | _List files/directories_ | ✅ | ✅ | 🚧 <sup>1</sup> | ✅ | ✅ | ✅ | ✅ | | ||
| | _Read USS file_ | ✅ | ✅ | 🚧 <sup>2</sup> | ✅ | ✅ | ✅ | ❌ | | ||
| | _Write USS file_ | ✅ | ✅ | 🚧 <sup>2</sup> | ✅ | ✅ | ❌ | ❌ | | ||
| | _Create file/directory_ | ✅ | ✅ | 🚧 <sup>3</sup> | ✅ | ❌ | ❌ | ❌ | | ||
| | _Delete file/directory_ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | Copy file/directory | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Move file/directory | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | _`chmod` file/directory_ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | | ||
| | _`chown` file/directory_ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | _`chtag` file/directory_ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Invoke `extattr` | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
| | Get ACL entries | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
| | Set ACL entries | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
| | Link file/directory | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
| | Unlink file/directory | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
| | List z/OS file systems | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Create z/OS file system | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Delete z/OS file system | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Mount file system | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Unmount file system | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
|
|
||
| 1. Not all attributes are retrieved | ||
| 2. Streaming is not supported for large files | ||
| 3. Recursive option is not supported | ||
|
|
||
| ## Jobs | ||
|
|
||
| | Operation | z/OSMF | FTP | Backend | Middleware | SDK | CLI | VSCE | | ||
| | ------------------ | ------ | --- | --------------- | ---------- | --- | --- | ---- | | ||
| | _Get job status_ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | | ||
| | _List jobs_ | ✅ | ✅ | 🚧 <sup>1</sup> | ✅ | ✅ | ✅ | ✅ | | ||
| | _List spool files_ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | _Read spool file_ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | _Get job JCL_ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | _Submit job_ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | Hold job | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Release job | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Change job class | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Cancel job | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | _Delete job_ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
|
|
||
| 1. Listing by prefix and status is not supported | ||
|
|
||
| ## Console | ||
|
|
||
| | Operation | z/OSMF | FTP | Backend | Middleware | SDK | CLI | VSCE | | ||
| | ------------------------ | ------ | --- | --------------- | ---------- | --- | --- | ---- | | ||
| | Issue MVS command | ✅ | ❌ | ✅ <sup>1</sup> | ✅ | ✅ | ✅ | ❌ | | ||
| | Get MVS command response | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Get MVS detect result | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
|
|
||
| 1. Requires APF authorization | ||
|
|
||
| ## TSO | ||
|
|
||
| | Operation | z/OSMF | FTP | Backend | Middleware | SDK | CLI | VSCE | | ||
| | ------------------------ | ------ | --- | ------- | ---------- | --- | --- | ---- | | ||
| | Start TSO address space | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Start TSO app | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Issue TSO command | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | | ||
| | Get TSO command response | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Send TSO message | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Send TSO app message | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Ping TSO address space | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Receive TSO message | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Receive TSO app message | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Stop TSO address space | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
|
|
||
| ## Other | ||
|
|
||
| | Operation | z/OSMF | FTP | Backend | Middleware | SDK | CLI | VSCE | | ||
| | -------------------- | ------ | --- | ------- | ---------- | --- | --- | ---- | | ||
| | Read system log | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ➖ | | ||
| | Get server status | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Change user password | ✅ | ❌ | ❌ | ❌ | ❌ | ➖ | ➖ | | ||
| | Issue SSH command | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # Client Design | ||
|
|
||
| ## SDK Package | ||
|
|
||
| The SDK defines a `ZSshClient` class with a `request` method that communicates with the [middleware component](../server/ioserver_architecture.md) over SSH. | ||
|
|
||
| For each command supported by the middleware, a `Request` and `Response` type is defined in the `doc` folder. For example, to list data sets: | ||
|
|
||
| ```typescript | ||
| export namespace ListDatasets { | ||
| export interface Request extends IRpcRequest { | ||
| command: "listDatasets"; | ||
| pattern: string; | ||
| attributes?: boolean; | ||
| } | ||
|
|
||
| export interface Response extends IRpcResponse { | ||
| items: Dataset[]; | ||
| returnedRows: number; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Once these interfaces are defined, the command can be invoked using the generic method `ZSshClient.request`: | ||
|
|
||
| ```typescript | ||
| using client = await ZSshClient.create(session); | ||
| const request: ListDatasets.Request = { | ||
| command: "listDatasets", | ||
| pattern: "IBMUSER.*", | ||
| }; | ||
| const response = await client.request<ListDatasets.Response>(request); | ||
| ``` | ||
|
|
||
| Each command is also exposed through the `AbstractRpcClient` class, which makes it more convenient and concise to invoke commands. For example, to list data sets there is a `listDatasets` method defined within the `ds` object: | ||
|
|
||
| ```typescript | ||
| public get ds() { | ||
| return { | ||
| listDatasets: (request: Omit<ListDatasets.Request, "command">): Promise<ListDatasets.Response> => | ||
| this.request({ command: "listDatasets", ...request }), | ||
| ... | ||
| }; | ||
| } | ||
| ``` | ||
|
|
||
| Now the example of invoking a command can be simplified as follows: | ||
|
|
||
| ```typescript | ||
| using client = await ZSshClient.create(session); | ||
| const response = await client.ds.listDatasets({ pattern: "IBMUSER.*" }); | ||
| ``` | ||
|
|
||
| > [!NOTE] | ||
| > To transmit binary data, it must be Base64 encoded since the z/OS OpenSSH server does not support sending raw bytes without codepage conversion. Use the methods `ZSshUtils.encodeByteArray` before sending binary data and `ZSshUtils.decodeByteArray` after receiving it. | ||
| ## CLI Plug-in | ||
|
|
||
| The Zowe CLI plug-in harnesses the SDK to access mainframe resources over SSH. The `src` directory is structured as follows: | ||
|
|
||
| - `imperative.ts` - Imperative configuration for the plug-in | ||
| - `SshBaseHandler.ts` - Base command handler that creates an SSH session and processes command output | ||
| - `**/*.definition.ts` - Definitions of commands and command groups | ||
| - `**/*.handler.ts` - Command handlers that implement functionality using the SDK | ||
|
|
||
| ## VS Code Extension | ||
|
|
||
| TODO |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| # ioserver | ||
|
|
||
| ioserver acts as the middleware between the client and server. It is written in Go and leverages an input channel to support asynchronous request processing. | ||
|
|
||
| ## Architectural overview | ||
|
|
||
| ioserver mediates all requests and dispatches requests to appropriate command handlers. These command handlers are defined in a map and are accessed by the command name. Those command handlers can perform actions in Go, or execute the backend layer (`zowex`) to access data. The response data is composed and serialized as JSON before being returned to the caller through stdout. | ||
|
|
||
| ## Request and response processing | ||
|
|
||
| The ioserver process is instantiated by the client through SSH, which opens a communication channel over stdio. When a request is received from the client over stdin, ioserver attempts to parse the input as JSON. If the JSON response is valid, the server looks for the `command` property of the JSON object and attempts to identify a matching command handler. If a command handler is found for the given command, the handler is executed and given the JSON object for further processing. | ||
|
|
||
| The command handlers can expect a stronger request type than what is expected during initial command processing. Appropriate request and response types can be exposed for use with these handlers. In the event of a JSON deserialization error, the command handler stops execution and returns early, returning an error response with any additional context. Once the JSON is successfully deserialized into the desired type, command processing continues and the handler can perform any actions necessary to create, receive, update or delete data. | ||
|
|
||
| Once the command handler has the required data for a response, the handler "marshals" (serializes) the response type as JSON and prints it to stdout. The output is redirected to the SSH client, where it is interpreted as a response and processed according to the corresponding response type. | ||
|
|
||
| See the example JSON request below for listing data sets: | ||
|
|
||
| ```json | ||
| { | ||
| "command": "listDatasets", | ||
| "pattern": "DS.PAT.*" | ||
| } | ||
| ``` | ||
|
|
||
| This request is deserialized by ioserver and `zowex` is invoked to get the list of data sets matching the given pattern. Then the response is composed, serialized as JSON and returned to the caller, for example: | ||
|
|
||
| ```json | ||
| { | ||
| "items": [ | ||
| { | ||
| "dsname": "DS.PAT.AB", | ||
| "dsorg": "PO", | ||
| "volser": "MIGRAT" | ||
| }, | ||
| { | ||
| "dsname": "DS.PAT.ABC", | ||
| "dsorg": "PO-E", | ||
| "volser": "WRKD01" | ||
| }, | ||
| { | ||
| "dsname": "DS.PAT.ABCD", | ||
| "dsorg": "PS", | ||
| "volser": "WRKD03" | ||
| } | ||
| ], | ||
| "returnedRows": 3 | ||
| } | ||
| ``` | ||
|
|
||
| ## Handling encoding for resource contents | ||
|
|
||
| Modern text editors expect a standardized encoding format such as UTF-8. `ioserver` implements processing for reading/writing data sets, USS files and job spools (read-only) with a given encoding. | ||
|
|
||
| ### Read | ||
|
|
||
| When an encoding is not provided, we use the system default codepage as the source encoding for resource contents (`IBM-1047`, in most cases). We convert the contents from the source encoding to UTF-8 so the contents can be rendered and edited within any modern text editor. | ||
|
|
||
| ```json | ||
| { | ||
| "command": "readFile", | ||
| "path": "/u/users/you/file.txt", | ||
| "encoding": "ISO8859-1" | ||
| } | ||
| ``` | ||
|
|
||
| Response: | ||
|
|
||
| ```jsonc | ||
| { | ||
| "encoding": "ISO8859-1", | ||
| "path": "/u/users/you/file.txt", | ||
| "data": [104, 101, 108, 108, 111] // "hello" in ASCII | ||
| } | ||
| ``` | ||
|
|
||
| ### Write | ||
|
|
||
| The contents of write requests are interpreted as UTF-8 data. We convert the UTF-8 bytes to the given encoding so the data can be read by z/OSMF and other mainframe utilities. If no encoding is provided, we convert the contents from `UTF-8` to the system default codepage. | ||
|
|
||
| ```jsonc | ||
| { | ||
| "command": "writeFile", | ||
| "path": "/u/users/you/file.txt", | ||
| "encoding": "IBM-939", | ||
| "contents": [ | ||
| 165, 131, 191, 165, 132, 192, 165, 131, 171, 165, 131, 161, 165, 131, 175, | ||
| 33 | ||
| ] // "Hello!" in Japanese, encoded as UTF-8 | ||
| } | ||
| ``` | ||
|
|
||
| Response: | ||
|
|
||
| ```json | ||
| { | ||
| "success": true | ||
| } | ||
| ``` | ||
|
|
||
| ### Data transmission | ||
|
|
||
| To send and receive converted contents between `ioserver` and `zowex`, we pipe the bytes of a write request to `zowex` and interpret the stdout from `zowex` during a read request. The environment variable `_BPXK_AUTOCVT` is temporarily disabled within the command handlers during write operations to prevent additional conversions of piped data. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| # Backend Design |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.