[FEATURE] Define required input and provided output keys

[dbwiddis]

Is your feature request related to a problem?

Workflow validation will verify that the required input of a step is provided by either its predecessor steps.

There is presently no mechanism to identify what output a step provides without running it.

What solution would you like?

Create a separate JSON file that maps workflow step types to their required input and output. The format of this file would be:

{
  "create_index": {
    "inputs": [
      "name",
      "settings"
    ],
    "outputs": [
      "index_name"
    ]
  },
  "create_ingest_pipeline": {
    "inputs": [
      "name",
      "processors"
    ],
    "outputs": [
      "pipeline_id"
    ]
  }
}

This separate definition file would be used to validate the template when it is first loaded (to make sure the required inputs are provided) and as part of unit testing for each step to ensure that the step fails if any are missing, and ensure returned output provides the required keys.

What alternatives have you considered?

Ideally these inputs and outputs would be encoded on methods on the steps themselves.

However, the front-end would also require this information in order to provide real-time validation to users trying to connect building blocks. Still, there are some options if they are included as methods on the WorkflowStep interface:

• use them for additional validation (and visibility to developers) in unit testing, and also test the JSON matches
• auto-generate the JSON in response to a REST API for the front-end to update its own information

Do you have any additional context?

This decision (proposed file vs. REST API vs. another idea) is a blocker for #18

Additionally if we consider a per-step configuration, this might be a better place for default timeouts, see #45 and #66.

[joshpalis]

To help make validation of inputs easier, I've created a use case template format proposal #117. I will action that issue along with this