-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow scripting to support dynamic entity value matches #204
Comments
rozele
added a commit
to rozele/NLU.DevOps
that referenced
this issue
Oct 4, 2019
In order to support matching of dynamic entity values, this commit introduces to changes. First, it adds a `context` property to each predicted utterance from the NLU service, including a `timestamp` value for when the prediction request was made. Second, it adds a mechanism to include inline scripts, which are evaluated at runtime with Roslyn. Fixes microsoft#204
rozele
added a commit
to rozele/NLU.DevOps
that referenced
this issue
Oct 9, 2019
In order to support matching of dynamic entity values, this commit introduces to changes. First, it adds a `context` property to each predicted utterance from the NLU service, including a `timestamp` value for when the prediction request was made. Second, it adds a mechanism to include inline scripts, which are evaluated at runtime with Roslyn. Fixes microsoft#204
rozele
added a commit
to rozele/NLU.DevOps
that referenced
this issue
Oct 9, 2019
Adds an object model to NLU.DevOps.Core that includes a timestamp for each predicted utterance. This is useful context for asserting correct predictions for dynamic entity values like dates. Towards microsoft#204
rozele
added a commit
to rozele/NLU.DevOps
that referenced
this issue
Oct 10, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET. A side-effect of this PR is that each NLU provider now labels utterances with a timestamp. Towards microsoft#204
rozele
added a commit
to rozele/NLU.DevOps
that referenced
this issue
Oct 10, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET. A side-effect of this PR is that each NLU provider now labels utterances with a timestamp. Towards microsoft#204
rozele
added a commit
to rozele/NLU.DevOps
that referenced
this issue
Oct 10, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET. A side-effect of this PR is that each NLU provider now labels utterances with a timestamp. Towards microsoft#204
rozele
added a commit
that referenced
this issue
Oct 11, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET. A side-effect of this PR is that each NLU provider now labels utterances with a timestamp. Towards #204
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For example, if a test includes text: "show me my schedule a week from tomorrow", likely we'll want to assert that "a week from tomorrow" is matched as a date entity, but we may also want to check the entity value assigned to that date.
Given that this would vary depending on the time the tests were run, we could allow users to specify a script that would calculate the expected value. E.g.,
We could embed these as strings within the expected JSON, e.g.:
There's still a slight issue in that the time the tests are run may differ from the time the compare command is run, but it alleviates the issue somewhat. We could also consider adding globals that could be included as well, e.g., if we could grab the test run "timestamp" context. In that case, maybe we could add a context object to each test result:
And we could refer to these globals in the expected output:
The text was updated successfully, but these errors were encountered: