Simple data driven UI configuration views in DHIS2

[jbee]

Intro

DHIS2 has many apps with UI views that mostly represent a simple edit form for a backend configuration of sorts.

Examples are

• system settings
• job configuration parameters
• translations

Currently the apps that contain the views to edit these server configurations are fully or partially "hard coded" requiring to update the app whenever the server configuration changes. This creates a fair amount of "busy work" to keep server and app in sync. This issue gets harder with server and app versions being more and more independently updated.

To both save time and solve the version compatibility issues this proposal suggests to add endpoints to the server that describe the form as a data model. An app would only fetch this model and render it without adding any logic on its own. Thereby the app always shows what the server actually supports and app and server are always in sync as long as they both support the same model.

Components

A list of components that were identified by looking at the system settings and the scheduler app.

Input Components

Type	Comment
Dropdown (single select)	A classic drop-down
Dropdown (multi select)	Each entry has a checkbox
Dropdown (text + options)	A text field that allows any value with a dropdown for quick select of common values
Checkbox
Input (text)
Input (pattern)
Input (numeric)
Button (file select)	with button to preview
Button (action)	a button that triggers a form independent action when clicked
List (multi select)	A list of unselected and a list of selected items with buttons to move items from one to the other
Button (save/update)

In addition any of the inputs might first display a warning or confirmation dialog before actually applying a change.

Inputs either map to a request parameter given by name or a request body object JSON path.

Layout Components

Type	Comment
tabs (vertical)	like menu, each tab opening a new view
box	a visual grouping using a border around a list of inputs
title	a divider sectioning of the following set of inputs from above one with a caption label
optional group	a set of inputs might first become available when a specific input checkbox is checked

Information Components

Type	Comment
table	a table to display tabular information

Solution (Sketch)

The proposal is given in the form of a model to have a more clear communication.
The actual shape of the model may vary from the proposal given here as the topic progresses.
However, it is important to keep the model equally simple as in this sketch to avoid the issue of ambiguity and too much complexity for the renderer code. The idea is strongly based upon saving time and resources by focusing on simple UIs only.
UIs that are too complex to cover with this approach should be custom build.
This just tries to find the sweet spot for simple forms that do not require custom solution where the benefit of saving time and resources if more important than to have a more problem specific view.

Model

ViewObject:

 { 
  "layout": "vertical-tabs" | "horizontal-tabs" | "dropdown", 
  "pages": [FormObject]
}

FormObject:

{
  "name": String, // unique name of the form
  "label": String, // shown on the tab or in the dropdown
  "icon"?: String, // name or URL for an icon that would be shown next to the label
  "action": String, // URL
  "method": String, // HTTP method, POST etcetera
  "parameter"?: String, // in case the selected page itself is reflected as a parameter or path
  "path"?: String,
  "body"?: String,
  "submit": "form" | "input", // entire form is submitted or each edit of an input field individually
  "button": String, // the text on the button that submits the form
  "components": [ComponentObject]
}

ComponentObject:

{
  "type": "dropdown" | "input" | "checkbox" | "file" | "selectlist" | "hidden" | "button" | "table" | "title"
  "group"?: String, // unique name for the group
  "style": "box"!
  // Input Components only
  "name": String, // unique name for the input in the form
  "value"?: String, // the pre-filled value
  "label": String,
  "description"?: String,
  "parameter": String,
  "path": String,
  "body": String,
  "required": true!  | false,
  "requires-true"?: [String], // names of components in this form
  "requires-false"?: [String], // names of components in this form
  // "input" type only (and "dropdown" with manual Entry=true)
  "text": "free"! | "numeric" | "pattern"
  "pattern"?: RegEx,
  "minLength"?: Number,
  "maxLength"?: Number,
  "minimum"?: Number,
  "maximum"?: Number,
  // "dropdown" type only
  "manualEntry": true | false!,
  // "dropdown", "selectlist" type only
  "multiselect": true | false!,
  "options": {String, String}, // value => label map
  // "selectlist" type only
  "sortable": true! | false, // allow reordering of the selected items
  // "button" type only
  "action": String, // URL
  "method": String // HTTP method for action
  // "title" type only
  "title": String,
  // "table" type only
  "cells": [[String]]
}

• Properties marked with ? are optional
• Default values are marked with !, a property with a default does not to need to be stated as the default is assumed
• Components are rendered in the order they are given

Groups

Components may declare a group name. Groups are only for visual grouping.
All components of the same group belong to that group and are rendered as a group of the style declared.
To find these groups a client might map the list of components to a map with the group as the key and the value being a list of the components of the group retaining the order.

Conditional Inputs

Inputs may be disabled based on the value of another checkbox input. To link them a input may declare the requires-true or requires-false components by their name that need to be true/false in order for the input to become enabled.
If multiple components are named these combine with AND logic.

Integration in Apps

An app that wants to use a model driven UI for some part of their interface would use the same UI component "model based form" that is build to render the model. All this component needs as an configuration is the server URL that yields the model to render. This URL always returns a ViewObject.
From there on the component is completely independent from the surrounding UI it might be embedded into.
It does not use state or other input other than the model returned by the server.
It does not update state in the surrounding UI.

State & Feedback

UIs build this way are "stateless". A button that performs an independent action might ask to refresh the view by fetching and rendering the model again. Here it would be the nice solution if the current inputs of form components are carried over to the re-rendering so that from a users perspective clicking a button is not necessarily a destructive action for the inputs made.
But based on the logic implemented on the server the model might change significantly after a button has been pressed.
The client, however, does not need to know this. It simply remembers the current values by their names, clears the view, re-renders it and applies the old remembered values to the new form if inputs of the name still exist.
The same process applies after the form has been submitted, except that no inputs need to be remembered as their change should be reflected in the new model send by the server. Clients are not smart about this. They simply restart fresh.

When an action button or a form is submitted the response is either shown as success or failure.
Both cases cause a refresh (re-render). A failure just tries to refill the user input as remember like it is described above.

Validation

The proposal keep the validation capabilities quite minimal. More sophisticated validation properties can always be added.
This does not change how the overall approach operates so going into details of validation methods is unnecessary at this point.

Example 1: New Job Configuration

{
  "layout": "dropdown",
  "pages": [
    { 
     "name": "ANALYTICS_TABLE",
     "path": "jobType",
     "label": "Analytics Table",
     "submit": "form",
     "button": "Save New Job",
     "method": "POST",
     "action": "/api/jobConfigurations",
     "components": [
       { "name": "name", "body": "name", "label": "Name", "type": "input", "maxLength": 255 },
       { "name": "cron", "body": "cronExpression", "label": "CRON expression", "type": "dropdown", "manualEntry": true,
          "options": { "0 0 * ? * *", "Every hour", ... }},
       { "name": "lastYears", "body": "jobParameters.lastYears", "label": "Last years", "type": "input", "text": "numeric", "required": false },
       { "name": "skipTableTypes", "body": "jobParameters.skipTableTypes", "label": "Skip table types", "type": "dropdown", "multiselect": true,
          "options": {"EVENT": "Event", ...} },
       { "name": "skipResourceTables": "body": "jobParameters.skipResourceTables", "label": "Skip resource tables", "type": "checkbox" }
     ]
    },
   {
    "name": "DATA_INTEGRITY",
    ...
   }
  ]
}

The slight difference to the current UI would be that the user first selects the job type because that is the selection for the page/form to show. Each of the pages is an independent form that contains all needed inputs to successfully POST a new job of the type.

Example 2: System Settings

{
  "layout": "vertical-tabs",
  "pages": [
    {
      "name": "general",
      "label": "General",
      "icon": "material-gears",
      "submit": "input",
      "method": "POST",
      "action": "/api/systemSettings/{key}",
      "components": [
        { "path": "key", "name": "keyAnalyticsMaxLimit", "parameter": "value", "label": "Maximum number of analytics records", "type": "input", "text": "numeric", "value"="200000" },
        { "path": "key", "name": "keySqlViewMaxLimit", "parameter": "value", "label": "Maximum number of SQL view records", "type": "dropdown", 
          "options":  {"-1", "Unlimited", "50000": "50.000", ...} },
        ...
      ]
    }
  ]
}

For settings each input is its own functional form. A change to the value is submitted directly as indicated by submit: input.
All inputs have both a path of value key (to indicate that the name is inserted for the {key} placeholder in the action template) and a parameter to indicate that the input value is given as a request parameter of name value.
That name is used for path and value is used for parameter in case both are present would be hard coded logic and a limitation of the model. If only one of them is present they use the value.

[cjmamo]

Nice. Perhaps this data model could be described in JSON Forms.

[jbee]

I can have a look. This is still very early. First I want to analyse the cases I am aware of and find a semantic model that would support them. Then we can look at the technical solution to express the semantic model. I add more details soon (I hope)..

[kabaros]

Thanks @jbee - From a UI point of view, I do not see the solution as simple and I will try to frame my thoughts here.

Complexity of the UI “State”:


This solution is not stateless. From a BE perspective, it might look like it, but in fact, it is the exact opposite of that. It shifts the state (and complexity) to a higher level. Rather than the state becoming the state of the form elements, now it is shifted up to mean the actual form, its elements, their config and values, and layout components, all of them need to be loaded asynchronously, updated, mapped to UI components (which would then still have to have their old state for managing the input values, the selected items etc.. unless we really bend how the FE framework works). This new higher-level state is a more difficult state since it’s dynamic and abstract, so it is hard to rationalise - for you, you can rationalise it looking at the JSON, we’d need to do the actual work of mapping it to UI elements just to rationalise it.

This is a much more complicated UI architecture. That complexity is not the issue as such, but for it to be justified, the return has to be worth it and I do not think this is the case here, or at least, the compromise - for UX and FE maintainability - hasn’t been taken into account. These are competing requirements and stakeholders here:

• Speed (BE perspective): For BE, the most important is to be able to roll out changes quickly. This is understandable especially for a system that is conceptually very tied to backend (like Scheduler).

• UX: we want a user experience that is decent - for example, for configuring some job params, we want to use a different component (Transfer component for example) because this makes sense UX-wise, we might want more intelligent client-side validation, is it really ok to save only on clicking the button - then how do we make sure that no changes are lost if someone forgets to click the button (if we have to keep track of the state of the form and show a reminder then this is another layer of state and complexity) …. This solution has massive impact on UX (even more so than FE stack) so that aspect needs to come into play - cc @cooper-joe

• Maintainability (FE perspective): we want to keep the codebase maintainable, easy to understand and debug.

This solution favours the first perspective (BE) but massively affects the other two. Now, this could be the right balance (especially for something like job config) but it’s a conversation that needs to be had. But then if we agree that we need to optimise for speed then in my opinion, this solution affects maintainability too much that we should not go that route, but there are other things we can/should do to make rolling changes faster - some of them are technical, but also a lot of it is process-related (and I can expand on that, I just don’t want this post to be too long)

Personal Experience

I’d rather stick to the more objective points, but I feel I should also share my personal experience here. This is a route I went before at least three times in my career - to have a completely Backend driven UI. The first time I did, I was the person who decided it and it was when I just shifted from being a backend developer to doing FE. This made total sense for me then, but I massively underestimated the complexity of the FE - yeah 80% was trivial to build, but the last 20% was awful and made the whole codebase and architecture crack with exceptions. The last time I did this - it kind of worked - but this was in the context of a much larger project, very corporate-y and five of us were just working on the frontend of this solution. It worked but it was also such a waste of resources. Again it started simple and trivial but got complicated - things like wanting to have a wizard instead of a tab, elements showing conditionally, validation based on multiple elements, how regexes are defined in Java vs JS, then how to make all of this responsive - if BE specifies horizontal tabs, we’d still want vertical on mobile - and make it accessible etc... In short, every time I went this route, I regretted it :)

I’m happy to expand on my thoughts on a direct call, and propose some alternatives - I understand and empathise with the frustration from a BE perspective here, but I hope that I can give you a sense of how all of this is not trivial from a FE perspective and we can find a middle ground.

[cooper-joe]

My opinion and experience are similar to that which @kabaros outlines above. Note that this perspective is based on the design, rather than technical, outcomes.
A backend as the source of truth that informs the UI sounds nice in theory, but in practice results in headaches trying to hack for the 10-20% of edge cases not covered. This inevitably ends up in UX compromises because the ideal solution is blocked by the technical implementation. In my experience, any time saved is lost in time spent hacking when a new field/feature/etc. is added.

[jbee]

I started a new discussion as my thoughts have changed somewhat in why and how I want to do it which is here #344