WIP

site/closed-track.md

@@ -1,4 +1,4 @@
-# GermEval2024 GerMS - Closed Track Competitions
+# Closed Track Competitions
 
 There is a _Closed Track_ competition for each of the two subtasks. Please note the following:
 

site/download.md

@@ -1,13 +1,12 @@
-# GermEval2024 GerMS - Download                 
+# Downloads                
 
 On this page, the files for training and labeling can be downloaded 
 for each of the phases of the GermEval2024 GerMS competition.
 
+All files are made available under a [CC BY-NC-SA 4.0 License](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en).
 
-## Trial Phase
 
-* [Training Data](data/germeval-trial-train.jsonl)
-* [Testing Data](data/germeval-trial-test.jsonl)
+## Trial Phase
 
 
 ## Development Phase

site/guidelines.md

@@ -0,0 +1,4 @@
+# Annotation Guidelines
+
+* [German Language Original](GERMS-AT_Annotierrichtlinien.pdf)
+* [English Translation](GERMS-AT_AnnotationGuidelines_and_AnnotatorAgreement_English_version.pdf)

site/index.md

@@ -20,8 +20,7 @@ are forum moderators.
 The main aim of annotating the presence and strength of sexism/misogyny in 
 the corpus was to identify comments which make it less welcoming to 
 women to participate in the conversation. 
-The full annotator guidelines with examples are available in an 
-[English translation of the German original](guidelines.pdf)
+The full annotator guidelines with examples are [available](guidelines.html). 
 
 Since the sexism/misogyny present in this corpus is often present in 
 a subtle form that avoids outright offensiveness or curse words, 
@@ -66,8 +65,9 @@ become available.
 * **Competition phase**: June 7 - June 25, 2024
   * During this phase, the training data will consist of the training data of the previous phase plus the labeled test data of the previous phase 
     and a new unlabeled test set is made available. You can upload the labels for 
-    the test set to the competition site and your most last submission will be the one that will be used for ranking. During that phase, the leaderboard is not shown. 
-    The final leaderboard/ranking is shown after the end of the competition phase. 
+    the test set to the competition site and your most last submission will be the one that will be used for ranking. During that phase, the leaderboard is not shown. Please note that only submissions which adhere to all terms and rules are considered for the final ranking.
+  * A preliminary leaderboard/ranking is shown after the end of the competition phase. 
+  * The final leaderboard/ranking will be available once paper review and final submission is completed, on July 21th 2024.
 * **Paper submission due**: July 1, 2024
 * **Camera ready due**: July 20, 2024
 * **Shared Task @KONVENS**: 10 September, 2024

site/open-track.md

@@ -1,4 +1,4 @@
-# GermEval2024 GerMS - Open Track Competitions
+# Open Track Competitions
 
 
 There is an _Open Track_ competition for each of the two subtasks. Please note the following:

site/report.md

@@ -1,3 +1,4 @@
-# GermEval2024 GerMS - Report
+# Report
 
-TBD
+This page will contain the report about the shared task after completion 
+and publication.

site/subtask1.md

@@ -1,4 +1,4 @@
-# GermEval2024 GerMS - Subtask 1
+# Subtask 1
 
 IMPORTANT: please note that there is a [closed](closed-track.md) and an [open](open-track.md) track for this subtask!
 

site/subtask2.md

@@ -1,107 +1,107 @@
-# GermEval2024 GerMS - Subtask 2
-
-IMPORTANT: please note that there is a [closed](closed-track.md) and an [open](open-track.md) track for this subtask!
-
-**Only submissions to the closed track which follow the rules for the closed track qualify for a paper submission and only an accepted paper qualifies for the 
-inclusion of your results in the final competition ranking.**
-
-In subtask 2 the goal is to predict the distribution for each text in a dataset where the distribution is  derived from the original distribution of labels assigned by several human annotators. 
-
-The human annotators assigned (according to the [annotation guidelines](guidelines.md) ) 
-the strength of misogyny/sexism present in the given text via the following labels:
-
-* `0-Kein`: no sexism/misogyny present
-* `1-Gering`: mild sexism/misogyny
-* `2-Vorhanden`: sexism/misogyny present
-* `3-Stark`: strong sexism/misogyny
-* `4-Extrem`: extreme sexism/misogyny
-
-While the annotation guidelines define what kind of sexism/misogyny should get annotated, there has been made no attempt to give rules about how to decide on the strength. For this reason, if an annotator decided that sexism/misogyny is present in a text, the strength assigned is a matter of personal judgement.
-
-The distributions to predict in subtask 2 are
-* the binary distribution ('dist_bin'): two values are predicted, which add up to 1.
-   * `dist_bin_0`: refers to the portion of annotators labeling the text as 'not-sexist' (`0-Kein`)
-   * `dist_bin_1`: refers to the portion of annotators labeling the text as 'sexist' (`1-Gering`, `2-Vorhanden`, `3-Stark`, or `4-Extrem`).
-* the multi score distribution ('dist_multi'): five values are predicted, which add up to 1.
-   * `dist_multi_0`: predict the portion of annotators labeling the text as `0-Kein`.
-   * `dist_multi_1`: predict the portion of annotators labeling the text as `1-Gering`.
-   * `dist_multi_2`: predict the portion of annotators labeling the text as `2-Vorhanden`.
-   * `dist_multi_3`: predict the portion of annotators labeling the text as `3-Stark`.
-   * `dist_multi_4`: predict the portion of annotators labeling the text as `4-Extrem`.
-
-## Data
-
-For the *trial phase* of subtask 1, we provide a small dataset, containing
-* a small labeled dataset containing 'id', 'text', and 'annotations' (annotator ids and the label assigned by them)
-* a small unlabeled dataset containing 'id', 'text' and 'annotators' (annotator ids)
-
-For the *development phase* of subtask 1, we provide all participants with the following data:
-* the labeled training set containing 'id', 'text', and 'annotations' (annotator ids and the label assigned by them)
-* the unlabeled dev set containing 'id', 'text' and 'annotators' (annotator ids)
-
-For the *competition phase* of subtask 1, we provide
-* the unlabeled test set containing 'id', 'text' and 'annotators' (annotator ids)
-
-All of the five files are in JSONL format (one JSON-serialized object per line) where each object is a dictionary with the following 
-fields:
-
-* `id`: a hash that identifies the example
-* `text`: the text to classify. The text can contain arbitrary Unicode and new lines 
-* `annotations` (only in the labeled dataset): an array of dictionaries which contain the following key/value pairs:
-  * `user`: a string in the form "A003" which is an anonymized id for the annotator who assigned the label
-  * `label`: the label assigned by the annotator
-  * Note that the number of annotations and the specific annotators who assigned labels vary between examples
-* `annotators` (only in the unlabeled dataset): an array of annotator ids who labeled the example
-
-You can [download](download.md) the data for each phase as soon as the corresponding phase starts.
-
-## Submission
-
-Your submission must be a file in TSV (tab separated values) format which contains the following columns in any order:
-
-* `id`: the id of the example in the unlabeled dataset for which the predictions are submitted
-* `dist_bin_0`: prediction of one value between 0 and 1 (all `dist_bin` values need to add up to 1). 
-* `dist_bin_1`: prediction of one value between 0 and 1 (all `dist_bin` values need to add up to 1).
-* `dist_multi_0`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
-* `dist_multi_1`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
-* `dist_multi_2`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
-* `dist_multi_3`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
-* `dist_multi_4`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
-
-Note that the way how you derive those values is up to you (as long as the rules for the closed or open tracks are followed):
-
-* you can train several models or a single model to get the predicted distribution
-* you can derive the mode-specific training set in any way from the labeled training data
-* you can use the information of which annotator assigned the label or ignore that
-
-To submit your predictions to the competition:
-
-* the file MUST have the file name extension `.tsv` 
-* the TSV file must get compressed into a ZIP file with extension `.zip`
-* the ZIP file should then get uploaded as a submission to the correct competition
-* !! Please make sure you submit to the competition that corresponds to the correct subtask (1 or 2) and correct track (Open or Closed)!
-* under "My Submissions" make sure to fill out the form and:
-  * enter the name of your team which has been registered for the competition
-  * give a name to your method 
-  * confirm that you have checked that you are indeed submitting to the correct competition for the subtask and track desired
-
-**Submission errors and warnings**
-
-* Always make sure a phase is selected before trying to upload your submission.
-* A submission is successful, if it has the submission status 'finished'. 'Failed' submissions can be investigated for error sources by clicking at '?' next to 'failed' and looking at LOGS > scoring logs > stderr.
-* If you experience any issue such as a submission file stuck with a "scoring" status, please cancel the submission and try again. In case the problem persists you can contact us using the Forum.
-* Following a successful submission, you need to refresh the submission page in order to see your score and your result on the leaderboard.
-
-## Phases
-
-* For the *trial phase*, multiple submissions are allowed for getting to know the problem and the subtask.
-* For the *development phase*, multiple submissions are allowed and they serve the purpose of developing and improving the model(s).
-* For the *competition phase*, participants may only submit a limited number of times. Please note that only the latest valid submission determines the final task ranking.
-
-## Evaluation
-
-System performance on subtask 2 is evaluated using the Jensen-Shannon distance for both (i) the prediction of the binary distribution, and (ii) the prediction of the multi score distribution. We chose the Jensen-Shannon distance as it is a standard method for measuring the similarity between two probability distributions and it is a proper 
-distance metric which is between 0 and 1. It is the square root of the Jensen-Shannon divergence, which is based on the Kullback-Leibler divergence. 
-
-The overall score which is used for ranking the submissions is calculated as the unweighted average between the two JS-distances.
-
+# Subtask 2
+
+IMPORTANT: please note that there is a [closed](closed-track.md) and an [open](open-track.md) track for this subtask!
+
+**Only submissions to the closed track which follow the rules for the closed track qualify for a paper submission and only an accepted paper qualifies for the 
+inclusion of your results in the final competition ranking.**
+
+In subtask 2 the goal is to predict the distribution for each text in a dataset where the distribution is  derived from the original distribution of labels assigned by several human annotators. 
+
+The human annotators assigned (according to the [annotation guidelines](guidelines.md) ) 
+the strength of misogyny/sexism present in the given text via the following labels:
+
+* `0-Kein`: no sexism/misogyny present
+* `1-Gering`: mild sexism/misogyny
+* `2-Vorhanden`: sexism/misogyny present
+* `3-Stark`: strong sexism/misogyny
+* `4-Extrem`: extreme sexism/misogyny
+
+While the annotation guidelines define what kind of sexism/misogyny should get annotated, there has been made no attempt to give rules about how to decide on the strength. For this reason, if an annotator decided that sexism/misogyny is present in a text, the strength assigned is a matter of personal judgement.
+
+The distributions to predict in subtask 2 are
+* the binary distribution ('dist_bin'): two values are predicted, which add up to 1.
+   * `dist_bin_0`: refers to the portion of annotators labeling the text as 'not-sexist' (`0-Kein`)
+   * `dist_bin_1`: refers to the portion of annotators labeling the text as 'sexist' (`1-Gering`, `2-Vorhanden`, `3-Stark`, or `4-Extrem`).
+* the multi score distribution ('dist_multi'): five values are predicted, which add up to 1.
+   * `dist_multi_0`: predict the portion of annotators labeling the text as `0-Kein`.
+   * `dist_multi_1`: predict the portion of annotators labeling the text as `1-Gering`.
+   * `dist_multi_2`: predict the portion of annotators labeling the text as `2-Vorhanden`.
+   * `dist_multi_3`: predict the portion of annotators labeling the text as `3-Stark`.
+   * `dist_multi_4`: predict the portion of annotators labeling the text as `4-Extrem`.
+
+## Data
+
+For the *trial phase* of subtask 1, we provide a small dataset, containing
+* a small labeled dataset containing 'id', 'text', and 'annotations' (annotator ids and the label assigned by them)
+* a small unlabeled dataset containing 'id', 'text' and 'annotators' (annotator ids)
+
+For the *development phase* of subtask 1, we provide all participants with the following data:
+* the labeled training set containing 'id', 'text', and 'annotations' (annotator ids and the label assigned by them)
+* the unlabeled dev set containing 'id', 'text' and 'annotators' (annotator ids)
+
+For the *competition phase* of subtask 1, we provide
+* the unlabeled test set containing 'id', 'text' and 'annotators' (annotator ids)
+
+All of the five files are in JSONL format (one JSON-serialized object per line) where each object is a dictionary with the following 
+fields:
+
+* `id`: a hash that identifies the example
+* `text`: the text to classify. The text can contain arbitrary Unicode and new lines 
+* `annotations` (only in the labeled dataset): an array of dictionaries which contain the following key/value pairs:
+  * `user`: a string in the form "A003" which is an anonymized id for the annotator who assigned the label
+  * `label`: the label assigned by the annotator
+  * Note that the number of annotations and the specific annotators who assigned labels vary between examples
+* `annotators` (only in the unlabeled dataset): an array of annotator ids who labeled the example
+
+You can [download](download.md) the data for each phase as soon as the corresponding phase starts.
+
+## Submission
+
+Your submission must be a file in TSV (tab separated values) format which contains the following columns in any order:
+
+* `id`: the id of the example in the unlabeled dataset for which the predictions are submitted
+* `dist_bin_0`: prediction of one value between 0 and 1 (all `dist_bin` values need to add up to 1). 
+* `dist_bin_1`: prediction of one value between 0 and 1 (all `dist_bin` values need to add up to 1).
+* `dist_multi_0`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
+* `dist_multi_1`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
+* `dist_multi_2`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
+* `dist_multi_3`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
+* `dist_multi_4`: prediction of one value between 0 and 1 (all `dist_multi` values need to add up to 1).
+
+Note that the way how you derive those values is up to you (as long as the rules for the closed or open tracks are followed):
+
+* you can train several models or a single model to get the predicted distribution
+* you can derive the mode-specific training set in any way from the labeled training data
+* you can use the information of which annotator assigned the label or ignore that
+
+To submit your predictions to the competition:
+
+* the file MUST have the file name extension `.tsv` 
+* the TSV file must get compressed into a ZIP file with extension `.zip`
+* the ZIP file should then get uploaded as a submission to the correct competition
+* !! Please make sure you submit to the competition that corresponds to the correct subtask (1 or 2) and correct track (Open or Closed)!
+* under "My Submissions" make sure to fill out the form and:
+  * enter the name of your team which has been registered for the competition
+  * give a name to your method 
+  * confirm that you have checked that you are indeed submitting to the correct competition for the subtask and track desired
+
+**Submission errors and warnings**
+
+* Always make sure a phase is selected before trying to upload your submission.
+* A submission is successful, if it has the submission status 'finished'. 'Failed' submissions can be investigated for error sources by clicking at '?' next to 'failed' and looking at LOGS > scoring logs > stderr.
+* If you experience any issue such as a submission file stuck with a "scoring" status, please cancel the submission and try again. In case the problem persists you can contact us using the Forum.
+* Following a successful submission, you need to refresh the submission page in order to see your score and your result on the leaderboard.
+
+## Phases
+
+* For the *trial phase*, multiple submissions are allowed for getting to know the problem and the subtask.
+* For the *development phase*, multiple submissions are allowed and they serve the purpose of developing and improving the model(s).
+* For the *competition phase*, participants may only submit a limited number of times. Please note that only the latest valid submission determines the final task ranking.
+
+## Evaluation
+
+System performance on subtask 2 is evaluated using the Jensen-Shannon distance for both (i) the prediction of the binary distribution, and (ii) the prediction of the multi score distribution. We chose the Jensen-Shannon distance as it is a standard method for measuring the similarity between two probability distributions and it is a proper 
+distance metric which is between 0 and 1. It is the square root of the Jensen-Shannon divergence, which is based on the Kullback-Leibler divergence. 
+
+The overall score which is used for ranking the submissions is calculated as the unweighted average between the two JS-distances.
+