Skip to content
This repository has been archived by the owner on Nov 1, 2024. It is now read-only.

Give task owners the ability to choose default setting for getting random contexts #902

Open
maxbartolo opened this issue Mar 21, 2022 · 2 comments

Comments

@maxbartolo
Copy link
Contributor

The current default method for getRandomContext() is min, however, there is no way for task owners to change this currently.

Ideally this should be controlled from the task config (e.g. https://dynabench.org/task-owner-interface/qa#advanced) similar to aggregation_metric.

We would need to modify: https://github.com/facebookresearch/dynabench/blob/main/frontends/web/src/common/Annotation/CreateInterface.js#L95.

I suspect the steps required to implement are:

  1. Move the taskConfig loading slightly earlier
  2. Process the new taskConfig.ATTRIBUTE_NAME attribute accordingly with a default value of min (not sure if we need to make any other changes to make it editable)
  3. Change .getRandomContext(this.state.task.id, this.state.task.cur_round) to .getRandomContext(this.state.task.id, this.state.task.cur_round, [], taskConfig.ATTRIBUTE_NAME)
@TristanThrush
Copy link
Contributor

Might be easier to control it with a new place in the task owner settings (similar to how the validation threshold is selected). Do you have any objections to doing that instead?

@maxbartolo
Copy link
Contributor Author

No, none at all. That works great too!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants