Change name of component schema item, delete schema item properties, create subset based on a schema

[mikeschinkel]

Hi all,

I attended the 2024-11-05 Overlays meeting and mentioned some use-cases where I had struggled to use Overlay and they asked me to elaborate after which they suggested I post here about them.

I had a client project to develop a Kubernetes External DNS provider in Golang and was given an OpenAPI document that defined their DNS' API. I used that OpenAPI to generate Go code using oapi-codegen and found that I wanted to transform the document before generated code from it.

I basically found three (3) conceptually simple things I wanted in order to facilitate generating Golang code more optimized for my use-case:

1. Rename a schema and change one of the properties of the renamed schema
2. Delete selected schema properties
3. Subset the OpenAPI doc based on a criteria, in my case the values of the endpoints

Specifically I wanted those because:

1. The names used in the OpenAPI schema generated wordy and confusing function and variable names in Golang,
2. The properties were either a security concern or were superfluous, and
3. I did not want to generate a huge amount of code to support all those other endpoints that the project would never use.

I struggled with getting overlay to meet my needs so just to get the project done I fell back to using jq and a shell script. Here is that script for reference (not great code, but it works):

#!/usr/bin/env bash

ORIGINAL="../openapi.json"
INPUT="./input.json"
OUTPUT="./openapi.json"
transform() {
  local query="$1"
  jq -r "${query}" "${INPUT}" > "${OUTPUT}"
  cp "${OUTPUT}" "${INPUT}"
}
cp "${ORIGINAL}" "${INPUT}"

##======================================
## Change name of component schema item
# Add Login as a duplicate of Body_login_login_post schema
transform ".components.schemas.Login=.components.schemas.Body_login_login_post"
# Give Login a proper title
transform ".components.schemas.Login.title=\"Login Post\""
# Now delete the Body_login_login_post schema
transform "del(.components.schemas.Body_login_login_post)"
# Replace all references to Body_login_login_post with Login
# shellcheck disable=SC2016
transform 'walk(if type=="object" and ."$ref" then ."$ref" |= sub("#/components/schemas/Body_login_login_post";"#/components/schemas/Login") else . end)'

##======================================
## Delete schema item properties
# Delete all Login properties that are not `username` or `password`
transform '.components.schemas.Login.properties |= with_entries(select(.key=="username" or .key=="password"))'
# Delete "view" properties for DNSandIPRequest and DNSRequest
transform '.components.schemas.DNSandIPRequest.properties |= with_entries(select(.key!="view"))'
transform '.components.schemas.DNSRequest.properties |= with_entries(select(.key!="view"))'

##======================================
## Create a subset of the OpenAPI doc
# Collect all endpoints we need; /login and those starting with /ipam/ 
transform '.paths = (.paths | with_entries(select(.key | startswith("/ipam/") or . == "/login"))) |.'
# Collect all schemas still in use 
REFS="$(jq -c '[..|objects|select(has("$ref"))|.["$ref"]|select(startswith("#"))|sub("#/components/schemas/";"")]|unique|sort' "${INPUT}")"
# Strip the unused schemas
# shellcheck disable=SC2154
transform "$(printf '.components.schemas = (.components.schemas | to_entries | map(select(.key as $key | %s | index($key)))|from_entries)' "${REFS}")"

##======================================
## Now generate the Go code for the API
go run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@d232e9e --config=./config.yaml ./openapi.json

This project was several months ago so I don't remember specifically what I could not get working with overlay — or that came across as far too complex for a simple conceptual change — but at least one of these things was either impossible or much harder than I think they ideally could be which is why I fell back to using jq and Bash because I knew how to make it work with them when faced with a looming deadline.

In summary I think each of these things are conceptually simple, something I can imagine many people doing codegen would want, and given their conceptual simplicity it would seem like they should be made easy to accomplish in overlay.

If you want me to elaborate on any of this, please do not hesitate to ask.

[lornajane]

Thanks @mikeschinkel , this is a great set of use cases and the jq is a great illustration of what you needed!

1. The rename could definitely be a stumbling block, because there's no real sense of a transform in an Overlay - you write or read and that's it - and identifying a node whose key you're changing needs something that I don't think we have yet.
2. Deleting things that either are named something, or are not named something, should be possible with a JSONPath expression, so I think this one should be achievable and if I can find some time then I'll try to turn it into a working example.
3. Stripping the endpoints is easy with Overlays - but that does leave the unused schemas, which if you know their names is easy to remove. Alternatively, make the endpoint changes with an Overlay, then clean up with something like a Redocly decorator could be a good approach.

[mikeschinkel]

Hi @lornajane — 

Thank you so much for your follow up.

FYI, this is from a prior project so I do not currently have the need to revisit this specific use-case thus my post is more of a "Hey, here are things that caused me to give up on overlay and switch to jq, assuming my input is helpful. #fwiw"

I do seem to remember that I could get one of those three done but don't remember which. At the time I decided it did not seem to make sense to have two processes — overlay and jq — so I just did it with jq as it was just more concise to do with one process like I did.

BTW, if deleting is possible w/JSONPath extension, yes I would like to see that. But also, if it turns out to be complex then I might suggest a more straightforward approach would be preferable, assuming those deciding believe in the rule of thumb "Make common things easy and uncommon things possible" (as I think this would be a common desire for code gen.)

But, as I said, I do not currently have this need anymore so I am just commenting in perchance you find my perspective helpful.