Adds experimental use of LUIS batch testing #4
Conversation
Adds a new interface for NLU providers capable of providing batch testing capabilities. Creates an implementation of the batch testing interface for LUIS. Fixes microsoft#334
| var delta = 1; | ||
| var retryAfter = DateTimeOffset.Now.AddSeconds(10).ToString("r", CultureInfo.InvariantCulture); | ||
| var delay = Retry.GetRetryAfterDelay(retryAfter, TimeSpan.Zero); | ||
| delay.TotalSeconds.Should().BeApproximately(10, delta); |
There was a problem hiding this comment.
Why do we use delta=1 here, but in previous test it is value 10e-6.
There was a problem hiding this comment.
I actually should not use BeApproximately here, because it's +/- delta. The problem with this test is that the implementation of GetRetryAfterDelay uses DateTimeOffset.Now, rather than some clock interface that can be easily mocked. So, this test is technically non-deterministic because we don't know how much of a diff will be between our estimated clock time (on L31) and the clock time in the GetRetryAfterDelay implementation, though given this is a synchronous operation without much compute work, we can be pretty confident that it will be < O(100ms). Thus I chose 1 second as a possible diff value.
It's a good catch though, as this tests 9 <= delay <= 11 (due to BeApproximately) when what we really want is 9 <= delay <= 10.
| var matchIndex = jsonObject.Value<int>("matchIndex"); | ||
|
|
||
| // Remove the generic entity properties | ||
| jsonObject.Remove("entityType"); |
There was a problem hiding this comment.
Should we modify in place or create a new object with fields that we need to have good understanding of what is required?
There was a problem hiding this comment.
We modify in place because the batch API may support additional properties that we don't care about. For example, if the batch API started allowing users to specify a role property on each entity, we should pass that through. The only thing we're trying to do is remove any of the generic utterance format properties (matchText, matchIndex and entityType) and replace them with LUIS batch testing format. It's safe to modify in place because we do a deep clone on L129.
@in4margaret - yes, this won't be ready for merge into the main repo for a while. |
- Fixes issue with Retry test. - Adds test for batch operator where item count modulo batch size is non-zero. - Adds documentation to note that speech and batch testing are incompatible. - Fixes typo in documentation.
rozele
force-pushed
the
issue334
branch
2 times, most recently
from
15f1ceb
to
00e5929
Compare
Adds a new interface for NLU providers capable of providing batch testing capabilities. Creates an implementation of the batch testing interface for LUIS.
Fixes microsoft#334