Proposal: State schema

[karth295]

I feel like there's a lot of this in statebus code:

conns = fetch('/connections')
conns.all ||= []

When I fetch remote state, fetch immediately returns with {key: '/connections'}, and later I will rerender with the "all" array. It seems like you should be able to tell statebus the schema for state.

Maybe something like this:

bus.schema =
  '/connections':
    all: Array
  '/connection':
    id: String
    video: Boolean
    audio: Boolean

Side benefits:

1. This helps document the state available in statebus
2. Statebus could verify that saves obeyed the types
   a. When you connect to a bus via the web protocol, it could send you its schema.

The schema should definitely be optional, and should only check types on state and keys within state that are defined in the schema.

[karth295]

Maybe a little bit better:

bus('/connection').schema =
  all: Array

[toomim]

I haven't finished digesting this yet. But briefly: there's a schema validator in statebus v5. It can be used like:

bus('foo').to_save = (o) =>
      if not bus.validate(o, {
        key: 'string'
        '?phone': 'string'
        '?resident': 'boolean'
        '?pools': 'array'
        'required': 'exact-matching-string'
      }) then bus.save.abort(o)

A question mark at the start of a key means it's optional.

[toomim]

Also this is just a temporary API design. We should figure out what the final API should look like.

[toomim]

Ok, so let's first check my understanding. This proposed schema could perform these three functions:

1. document the state
2. validate changes to state
3. set default values

(Is that correct? I think the github issue could be clarified to list all three points.) I certainly agree with the importance of these three problems! I've also noticed the bad smell of writing conns.all ||= [] around my code.

One way to address these three issues is by defining a schema. However, the schema doesn't solve the problems fully.

Problem 1: Documenting state

Solution option 1: A schema

• Programmer has to write it
• Can't document all the state, such as nested subtrees, or union types, without making the language more complex, and eventually turing-complete.
• Doesn't tell you the values of attributes at runtime.
• Only works for statebus sites that use our schema library
• Is only visible to people reading source code. Statebus state will be used by people over the network, e.g. strangers connecting to your website and re-using its state.

Solution option 2: A state dashboard (example: go to https://consider.it and type "idkfa")

• Shows you ALL the state
• Shows the actual values inside of it
• Can detect and visualize patterns over the space (for instance, the dashboard presents arrays of objects as a spreadsheet)
• Can be extended to detect and visualize patterns of values over time (for instance, using a drop-menu with three options if it sees that an attribute is always one of three values)
• Is available to anyone over the network, whether they can see the source code or not.

We do need to port the state dashboard code to v5, and spiff it up.

Problem 2: Validating changes to state

Solution option 1: Schema

• Can't validate everything; programmer still has to write validation logic
• Then he'll have to do validation in two places—the schema, and his validation function. The validation function will depend on the schema doing some validation, that he doesn't write by hand, so he'll have to coordinate the code between the two, which is dangerous for critical security logic. Changes to one could impact the other. And if he writes all the logic in the validation routine, then he doesn't need the schema's validation anymore.

Solution option 2: Validation helpers

• Programmer can write turing-complete code, can validate anything
• We give him helpers like the validate() function above

Problem 3: Setting default values

Solution option 1: Schema

• Can set some default values (like initializing arrays with [] as above), but the programmer would still need to write code to set other default values. For instance, the toomim bros. mining ordering page sets values for a default mining order object, that looks like this:

default_item = ->
  { quantity: 1, months: 1, type: 'sp31', power: systems.sp31.power }

default_order = ->
  key: 'ls/order'
  payment: 'bitcoin'
  renewal: true
  wash_resident: false
  items: [ default_item() ]
  adjustment: { desc: '', amt: 0.0 }
  email: '[email protected]'
  name: 'Dudesicle'
  duedate: date_struct(new Date())
  pay_address: 'N/A'
  comment: null
  code: null

## .. and later on ..

dom.BODY = ->
  # ...
    # ...

    # If there's initialization data provided, let's use it
    if initial_data
      # Clear the old order
      save(default_order())

      # Now load the new order data in on top of it
      for k of initial_data
        order[k] = urldata[k]
      save(order)

    # If there's nothing loaded at all, let's give it some default
    else if not order.email
      save(default_order())

This can't be set with the current schema proposal, because it sets custom defaults, using code to compute them.

Solution option 2: Default helpers

• We could specifically make code like the above easier to write, like with a "default" function:

bus('foo').default = (k) => {
   return {val: 'default'}
}

• We could make a wrapper that defines lots of defaults in one place:

set_defaults({
  foo: {val: 'default'}
  cons: {all: []}
  dynamic: () => {
    if (something)
      {foo: 'bar'}
    else
      {bar: 'buz'}
  }
})

I think we want to solve each of these three problems, but we should be careful just introducing a schema feature into the API, because then programmers will feel like they are supposed to specify their state with the schema, and they'll invest work into the schema, and then they'll run into limits with how much it helps them, and have to write code anyway to finish getting their job done, and then additionally keep the schema up-to-date.

On the other hand, we don't need to write a "schema" API at all right now. We can just add some helpers to our own code that make it easier to set defaults and validate changes as we currently need to in our projects, solving our own problems. As these helper functions become generally useful, we can add them to the statebus distro. If they become so mature and useful that we think everyone will want to use them all the time, then we can make a schema API and encourage programmers to use it. But I think for now we will do best by solving each of these 3 problems independently, without prematurely codifying a limited standard.

[toomim]

Oh, I'm sorry, I mis-understood your point 2a:

a. When you connect to a bus via the web protocol, it could send you its schema.

This addresses the "have to read source code" issue. On the other hand, it does require the client and server to cooperate, and I think we get the same benefit from a state dashboard dynamically looking at the actual state, rather than asking programmers and servers to predefine what the state might look like.

[toomim]

Another way to look at the problem is that we can infer a schema dynamically at runtime. That's what the state dashboard does. Additionally, it also visualizes the current state, within that schema, and lets the user modify the state.

State inference can also be useful for detecting bugs. When you change the code, you can re-run your tests, and we could automatically show you any changes to the inferred schema. If these aren't changes you want, then your code probably has bugs. This can detect the same types of bugs that a schema would detect, but without requiring the programmer to write the schema.

So we could split the problems cited in this issue up into these sub-projects:

• Improve schema inference
• Port state dashboard to v5
• Improve state dashboard
• Implement defaults helpers
• Improve validation helpers

[karth295]

Sounds good to me -- maintaining a schema doesn't sound very fun.

I don't think you've addressed the main problem of setting default state on the server and using that on the client. The server and client will have to coordinate for that to work.

[toomim]

I don't think you've addressed the main problem of setting default state on the server and using that on the client. The server and client will have to coordinate for that to work.

Ah, interesting point! Off the top of my head, we will be able to address this once we finish server-side rendering. The server will serialize its state into JSON along with the rendered HTML of the page and the client code. The JSON will not just be the default state—it'll actually contain the current version of that state on the server. Thus the client will only need to set defaults for the state that doesn't exist on the server.

[toomim]

Actually, consider that even without server-side rendering, you don't need to set a default for the state on both the client and the server; just on one of them.

If you set a default on the server, then the client code will crash e.g. on for c in fetch('conns').all and (because it knows it's waiting for the conns fetch) will wait for the server to send its valid default state to the client before throwing an error. Once that is sent to the client, the client will re-render and be fine.

If you set a default on the client, then even if the server doesn't set the default state, the client will have it and will be fine. Then whenever the client saves that state, the server will get a good solid version of the state and so it won't ever have needed the default.

Defaults are only a problem if neither the server nor the client set a default. But only one of them needs to. If you define it on the client, you might get the page to load a little faster. If you define it on the server, you can simplify your client code. And once we finish server-side rendering, you will get the best of both worlds.