Proposal: optional_subfield for use style nested optional decoding

[jhillyerd]

Problem Summary

The new use decoder API lacks an easy way to cherry-pick a nested value when the path to that value may have missing or null elements. It is possible with the current API by mixing optional_field, optional, and at – but the method is not simple or easily discoverable.

Cherry-picking optional nested values is very helpful when the Gleam type you wish to decode into does not closely match the schema you are decoding from, or if you don’t have a schema and are iterating on the design of your type & decoder.

Proposed Solution Summary

Add an optional_subfield function to the dynamic/decode module, similar to the existing subfield function, but allowing for a default value, signature:

pub fn optional_subfield(
  field_path: List(name),
  default: t,
  field_decoder: Decoder(t),
  next: fn(t) -> Decoder(final),
) -> Decoder(final)

In addition, we can consider modifying the existing optionally_at function to handle “null” and not just missing values. Its current behavior might be considered a bug, but altering it might also be considered an API change.

Problem Detail

Writing a decoder to optionally extract a value for “c” from the below JSON samples is more difficult than one would expect.

{"id": 1}
{"id": 1, "a": null}
{"id": 1, "a": {"b": null}}
{"id": 1, "a": {"b": {"c": null}}}
{"id": 1, "a": {"b": {"c": “target value”}}}

There are several difficulties:

1. There is no optional subfield function for the new use decoder API, so one must start with optional_field and then pass in regular Decoder to navigate deeper into the tree
2. optionally_at does not handle null values along the path, only missing ones (see failure example below)
3. While one_of + at can be a concise way to decode a nested value, it does not allow the caller to differentiate a missing value from an invalid one, masking errors (see failure example below)
4. The solution I found, chaining optional and at results in a nested set of Options that need to be flattened

Each of the current field type decoders has a matching optional variant, except one. This was confusing to me:

• at has optionally_at
• field has optional_field
• subfield has no optional variant

Problem Example Code

This example illustrates the existing way to decode this, along with some assertions you may use to test alternatives if you feel I’ve missed something. It should be usable in a fresh gleam project after running gleam add gleam_json.

The example below succeeds, but the combination of optional_field, optional, and at is difficult to read, non-obvious, and requires a matching chain of option.flatten calls to unwrap the result.

import gleam/dynamic/decode.{
  at, field, int, optional, optional_field, string, success,
}
import gleam/io
import gleam/json
import gleam/list
import gleam/option.{type Option, None, Some, flatten}

pub type Thing {
  Thing(id: Int, c: Option(String))
}

pub fn main() {
  let thing_decoder = {
        use id <- field("id", int)
        use str_val <- optional_field(
          "a",
          None,
          optional(at(["b"], optional(at(["c"], optional(string))))),
        )


        success(Thing(id:, c: str_val |> flatten |> flatten))
  }

  let none1 = "{\"id\": 1}"
  let none2 = "{\"id\": 1, \"a\": null}"
  let none3 = "{\"id\": 1, \"a\": {\"b\": null}}"
  let none4 = "{\"id\": 1, \"a\": {\"b\": {\"c\": null}}}"

  [none1, none2, none3, none4]
  |> list.each(fn(o) {
        io.println("input: " <> o)
        let assert Ok(Thing(1, None)) = json.parse(o, thing_decoder)
  })

  let some1 = "{\"id\": 1, \"a\": {\"b\": {\"c\": \"string\"}}}"
  io.println("input: " <> some1)
  let assert Ok(Thing(1, Some("string"))) = json.parse(some1, thing_decoder)

  let error1 = "{\"id\": 1, \"a\": {\"b\": {\"c\": 7}}}"
  io.println("input: " <> error1)
  let assert Error(json.UnableToDecode([
        decode.DecodeError("String", "Int", ["b", "c"]),
  ])) = json.parse(error1, thing_decoder)
}

Failing Examples

Trying to use optionally_at instead:

  let thing_decoder = {
    use id <- field("id", int)
    use str_val <- then(optionally_at(["a", "b", "c"], None, optional(string)))
    
    success(Thing(id:, c: str_val))
  }

Will fail to decode when a is null:

input: {"id": 1, "a": null}
runtime error: let assert

Pattern match failed, no pattern matched the value.

unmatched value:
  Error(UnableToDecode([DecodeError("Dict", "Atom", ["a"])]))

Trying to use one_of instead:

  let thing_decoder = {
    use id <- field("id", int)
    use str_val <- then(
      one_of(at(["a", "b", "c"], string) |> decode.map(Some), [success(None)]),
    )

    success(Thing(id:, c: str_val))
  }

Will decode a None, instead of failing when given a number instead of a string:

runtime error: let assert

Pattern match failed, no pattern matched the value.

unmatched value:
  Ok(Thing(1, None))

Solution Detail

My ideal solution would allow the decoder block above to be written as:

  let thing_decoder = {
    use id <- field("id", int)
    use str_val <- optional_subfield([“a”, “b”, “c”], None, optional(string))

    success(Thing(id:, c: str_val))
  }

The optional_subfield function can be implemented by copying the existing optional_field function, and modifying it to use index instead of bare_index. The resulting function body is only 8 lines of code, which should not present a heavy maintenance burden.

A potential implementation, with tests can be seen here: jhillyerd/stdlib@0f09583

[lpil]

Hello! This would be written today like so:

use value <- decode.then(decode.optionally_at(
  ["a", "b", "c"],
  None,
  decode.optional(decode.string),
))

We could possibly add this new function, but it's best to wait some time to get an understanding of how the current API is going. The stdlib is very conservative so we delaying any additions is good.

[jhillyerd]

Thanks, I was looking for a function glue regular decoders to use, I didn't realize then was it -- I may send a docs PR to add an example of that. I've updated my failing examples above to use then so they are easier to read.

However, then + optionally_at doesn't actually work in the scenario I gave, it fails when it encounters a null: UnableToDecode([DecodeError("Dict", "Atom", ["a"])]). This feels like a bug/oversight to me, given we often can't control how the JSON given to us has been encoded, sometimes values may be null and other times they will not be present.

[lpil]

Could you give an example of when it would fail? I thought I tested with all your examples above, but perhaps I am misremembering.

[jhillyerd]

Yes, given a decoder containing:

use str_val <- decode.then(decode.optionally_at(
  ["a", "b", "c"],
  None,
  optional(string),
))

And the following 3 JSON inputs:

{"a": {}}
{"a": {"b": {}}}
{"a": {"b": null}}

The first two will decode to None as expected, but the third case will fail:

runtime error: let assert

Pattern match failed, no pattern matched the value.

unmatched value:
  Error(UnableToDecode([DecodeError("Dict", "Atom", ["a", "b"])]))

I am running these tests with Erlang, not sure if JS would behave any differently.