Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow scripting to support dynamic entity value matches #204

Open
rozele opened this issue Oct 4, 2019 · 0 comments
Open

Allow scripting to support dynamic entity value matches #204

rozele opened this issue Oct 4, 2019 · 0 comments

Comments

@rozele
Copy link
Contributor

rozele commented Oct 4, 2019

For example, if a test includes text: "show me my schedule a week from tomorrow", likely we'll want to assert that "a week from tomorrow" is matched as a date entity, but we may also want to check the entity value assigned to that date.

Given that this would vary depending on the time the tests were run, we could allow users to specify a script that would calculate the expected value. E.g.,

DateTimeOffset.Now.Date.AddDays(8)

We could embed these as strings within the expected JSON, e.g.:

{
  "text": "show me my schedule for a week from tomorrow",
  "intent": "Calendar",
  "entities": [
     {
        "entityType": "date",
        "matchText": "a week from tomorrow",
        "entityValue": "csharp(DateTimeOffset.Now.Date.AddDays(8))"
     }
  ]
}

There's still a slight issue in that the time the tests are run may differ from the time the compare command is run, but it alleviates the issue somewhat. We could also consider adding globals that could be included as well, e.g., if we could grab the test run "timestamp" context. In that case, maybe we could add a context object to each test result:

{
  "text": "show me my schedule for a week from tomorrow",
  "intent": "Calendar",
  "entities": [ ... ],
  "globals": {
    "timestamp": "1970-01-01T00:00:00Z"
  }

And we could refer to these globals in the expected output:

{
  "text": "show me my schedule for a week from tomorrow",
  "intent": "Calendar",
  "entities": [
     {
        "entityType": "date",
        "matchText": "a week from tomorrow",
        "entityValue": "csharp(DateTimeOffset.Parse(timestamp).AddDays(8))"
     }
  ]
}
rozele added a commit to rozele/NLU.DevOps that referenced this issue Oct 4, 2019
In order to support matching of dynamic entity values, this commit introduces to changes.

First, it adds a `context` property to each predicted utterance from the NLU service, including a `timestamp` value for when the prediction request was made.

Second, it adds a mechanism to include inline scripts, which are evaluated at runtime with Roslyn.

Fixes microsoft#204
rozele added a commit to rozele/NLU.DevOps that referenced this issue Oct 9, 2019
In order to support matching of dynamic entity values, this commit introduces to changes.

First, it adds a `context` property to each predicted utterance from the NLU service, including a `timestamp` value for when the prediction request was made.

Second, it adds a mechanism to include inline scripts, which are evaluated at runtime with Roslyn.

Fixes microsoft#204
rozele added a commit to rozele/NLU.DevOps that referenced this issue Oct 9, 2019
Adds an object model to NLU.DevOps.Core that includes a timestamp for each predicted utterance. This is useful context for asserting correct predictions for dynamic entity values like dates.

Towards microsoft#204
rozele added a commit to rozele/NLU.DevOps that referenced this issue Oct 10, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET.

A side-effect of this PR is that each NLU provider now labels utterances with a timestamp.

Towards microsoft#204
rozele added a commit to rozele/NLU.DevOps that referenced this issue Oct 10, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET.

A side-effect of this PR is that each NLU provider now labels utterances with a timestamp.

Towards microsoft#204
rozele added a commit to rozele/NLU.DevOps that referenced this issue Oct 10, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET.

A side-effect of this PR is that each NLU provider now labels utterances with a timestamp.

Towards microsoft#204
rozele added a commit that referenced this issue Oct 11, 2019
In order to support a number of additional properties, including timestamps, intent confidence scores, transcription confidence scores, and utterance IDs, this change removes all specializations of NLU.DevOps.Models.LabeledUtterance in favor of NLU.DevOps.Core.JsonLabeledUtterance, which simply stores any additional properties in a dictionary labeled with JsonExtensionData from Json.NET.

A side-effect of this PR is that each NLU provider now labels utterances with a timestamp.

Towards #204
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant