Merge pull request #549 from PAIR-code/dev

Merge dev to main for v0.4
PAIR-code · Nov 9, 2021 · 4d8a9ae · 4d8a9ae
2 parents 3eb824b + b254192
commit 4d8a9ae
Show file tree

Hide file tree

Showing 281 changed files with 52,748 additions and 4,899 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,59 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+# Config file for GitHub Actions for CI.
+
+name: CI
+
+# Run on pushes and pull requests on main and dev branches.
+on:
+  push:
+    branches: [ main, dev ]
+  pull_request:
+    branches: [ main, dev ]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  build-and-test:
+    name: Build and test (${{ matrix.python-version }})
+    strategy:
+      matrix:
+        python-version: ["3.7"]
+    defaults:
+      run:
+        shell: bash -l {0}
+    runs-on: ubuntu-latest
+
+    steps:
+      # Checkout the code for this CI operation.
+      - uses: actions/checkout@v2
+
+      # Setup environment for LIT and test python.
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: |
+          conda env create -f environment.yml
+          conda activate lit-nlp
+          pip install pytest lime
+          pytest -v
+
+      # Build front-end.
+      - uses: actions/setup-node@v2
+      - run: pushd lit_nlp && yarn && yarn build && popd
+
+      # TODO(jwexler): Test front-end
diff --git a/.travis.yml b/.travis.yml
diff --git a/README.md b/README.md
@@ -1,10 +1,11 @@
 # 🔥 Language Interpretability Tool (LIT)
 
-<!--* freshness: { owner: 'lit-dev' reviewed: '2020-08-04' } *-->
+<!--* freshness: { owner: 'lit-dev' reviewed: '2021-07-13' } *-->
 
 The Language Interpretability Tool (LIT) is a visual, interactive
-model-understanding tool for NLP models. It can be run as a standalone server,
-or inside of notebook environments such as Colab and Jupyter.
+model-understanding tool for ML models, focusing on NLP use-cases. It can be run
+as a standalone server, or inside of notebook environments such as Colab,
+Jupyter, and Google Cloud Vertex AI notebooks.
 
 LIT is built to answer questions such as:
 
@@ -39,13 +40,12 @@ Stay up to date on LIT by joining the
 [lit-announcements mailing list](https://groups.google.com/g/lit-annoucements).
 
 For a broader overview, check out [our paper](https://arxiv.org/abs/2008.05122) and the
-[user guide](documentation/user_guide.md).
+[user guide](https://github.com/PAIR-code/lit/wiki/ui_guide.md).
 
 ## Documentation
 
-*   [User Guide](documentation/user_guide.md)
-*   [Developer Guide](documentation/development.md)
-*   [FAQ](documentation/faq.md)
+*   [Documentation index](https://github.com/PAIR-code/lit/wiki/)
+*   [FAQ](https://github.com/PAIR-code/lit/wiki/faq.md)
 *   [Release notes](./RELEASE.md)
 
 ## Download and Installation
@@ -94,8 +94,8 @@ Explore a collection of hosted demos on the
 [LIT website demos page](https://pair-code.github.io/lit/demos).
 
 Colab notebooks showing the use of LIT inside of notebooks can be found at
-google3/third_party/py/lit_nlp/example/notebooks. A simple example can be viewed
-[here](https://colab.research.google.com/github/pair-code/lit/blob/main/examples/notebooks/LIT_sentiment_classifier.ipynb).
+google3/third_party/py/lit_nlp/examples/notebooks. A simple example can be viewed
+[here](https://colab.research.google.com/github/pair-code/lit/blob/main/lit_nlp/examples/notebooks/LIT_sentiment_classifier.ipynb).
 
 ### Quick-start: classification and regression
 
@@ -134,7 +134,7 @@ the notebook.
 
 ### Run LIT in a Docker container
 
-See [docker.md](documentation/docker.md) for instructions on running LIT as
+See [docker.md](https://github.com/PAIR-code/lit/wiki/docker.md) for instructions on running LIT as
 a containerized web app. This is the approach we take for our
 [website demos](https://pair-code.github.io/lit/demos/).
 
@@ -148,7 +148,7 @@ python -m lit_nlp.examples.<example_name> --port=5432 [optional --args]
 
 ## User Guide
 
-To learn about LIT's features, check out the [user guide](documentation/user_guide.md), or
+To learn about LIT's features, check out the [user guide](https://github.com/PAIR-code/lit/wiki/ui_guide.md), or
 watch this [video](https://www.youtube.com/watch?v=CuRI_VK83dU).
 
 ## Adding your own models or data
@@ -158,20 +158,25 @@ launcher, similar to those in [lit_nlp/examples](./lit_nlp/examples). The basic
 steps are:
 
 *   Write a data loader which follows the
-    [`Dataset` API](documentation/python_api.md#datasets)
-*   Write a model wrapper which follows the [`Model` API](documentation/python_api.md#models)
+    [`Dataset` API](https://github.com/PAIR-code/lit/wiki/api.md#datasets)
+*   Write a model wrapper which follows the [`Model` API](https://github.com/PAIR-code/lit/wiki/api.md#models)
 *   Pass models, datasets, and any additional
-    [components](documentation/python_api.md#interpretation-components) to the LIT server
+    [components](https://github.com/PAIR-code/lit/wiki/api.md#interpretation-components) to the LIT server
     class
 
 For a full walkthrough, see
-[adding models and data](documentation/python_api.md#adding-models-and-data).
+[adding models and data](https://github.com/PAIR-code/lit/wiki/api.md#adding-models-and-data).
 
 ## Extending LIT with new components
 
 LIT is easy to extend with new interpretability components, generators, and
-more, both on the frontend or the backend. See the
-[developer guide](documentation/development.md) to get started.
+more, both on the frontend or the backend. See our
+[documentation](https://github.com/PAIR-code/lit/wiki/index.md) to get started.
+
+## Pull Request Process
+
+To make code changes to LIT, please work off of the `dev` branch and create
+pull requests against that branch. The `main` branch is for stable releases, and it is expected that the `dev` branch will always be ahead of `main` in terms of commits.
 
 ## Citing LIT
 

diff --git a/RELEASE.md b/RELEASE.md
@@ -1,5 +1,25 @@
 # Language Interpretability Tool releases
 
+## Release 0.4
+
+This release adds a lot of new features. The website and documentation have
+been updated accordingly.
+
+The UI has been slightly revamped, bugs have been fixed, and new capabilities
+have been added. Noteable changes include:
+- Support for Google Cloud Vertex AI notebooks.
+- Preliminary support for tabular and image data, in addition to NLP models.
+- Addition of TCAV global interpretability method.
+- New counterfactual generators for ablating or flipping text tokens for
+  minimal changes to flip predictions.
+- New counterfactual generator for tabular data for minimal changes to flip
+  predictions.
+- Partial depdence plots for tabular input features.
+- Ability to set binary classification thresholds separately for different
+  facets of the dataset
+- Controls to find optimal thresholds across facets given different fairness
+  constraints, such as demographic parity or equal opportunity.
+
 ## Release 0.3
 
 This release adds the ability to use LIT directly in colab and jupyter

diff --git a/docs/assets/css/new.css b/docs/assets/css/new.css
@@ -639,3 +639,7 @@ img.status-emoji {
   height: 24px;
   padding-right: 4px;
 }
+
+.tutorial-caption {
+  padding-bottom: 16px;
+}
diff --git a/docs/assets/images/actor_to_actress.png b/docs/assets/images/actor_to_actress.png
diff --git a/docs/assets/images/actress_relative_cav.png b/docs/assets/images/actress_relative_cav.png
diff --git a/docs/assets/images/lit_data_table_annotated.png b/docs/assets/images/lit_data_table_annotated.png
diff --git a/docs/assets/images/lit_slice_editor_annotated.png b/docs/assets/images/lit_slice_editor_annotated.png
diff --git a/docs/assets/images/lit_tcav_screen_annotated.png b/docs/assets/images/lit_tcav_screen_annotated.png
diff --git a/docs/assets/images/lit_tcav_settings_annotated.png b/docs/assets/images/lit_tcav_settings_annotated.png
diff --git a/docs/assets/images/tcav_result.png b/docs/assets/images/tcav_result.png
diff --git a/docs/assets/images/tcav_results_2.png b/docs/assets/images/tcav_results_2.png
diff --git a/docs/demos/images.html b/docs/demos/images.html
@@ -0,0 +1,5 @@
+<html>
+  <script>
+    window.location.replace("http://104.155.142.190/");
+  </script>
+</html>
diff --git a/docs/demos/index.html b/docs/demos/index.html
@@ -97,15 +97,15 @@
   <div class="demo-card-cta-button"><a href="/lit/demos/glue.html"></a></div>
 </div>
 <div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
-  <div class="demo-card-title"><a href="https://colab.research.google.com/github/PAIR-code/lit/blob/main/lit_nlp/examples/notebooks/LIT_sentiment_classifier.ipynb" target="_blank">Notebook usage</a></div>
-  <div class="demo-card-tags"> <span class="demo-tag"> BERT </span>  <span class="demo-tag"> binary classification </span>  <span class="demo-tag"> notebooks </span>
+  <div class="demo-card-title"><a href="/lithttps://colab.research.google.com/github/PAIR-code/lit/blob/main/lit_nlp/examples/notebooks/LIT_sentiment_classifier.ipynb" target="_blank">Notebook usage</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> BERT </span>  <span class="demo-tag"> binary classification </span>  <span class="demo-tag"> notebooks </span> 
   </div>
   <div class="demo-card-data-source-title">DATA SOURCES</div>
   <div class="demo-card-data-source">
     Stanford Sentiment Treebank
   </div>
   <div class="demo-card-copy">Use LIT directly inside a Colab notebook. Explore binary classification for sentiment analysis using SST2 from the General Language Understanding Evaluation (GLUE) benchmark suite.</div>
-  <div class="demo-card-cta-button"><a href="https://colab.research.google.com/github/PAIR-code/lit/blob/main/lit_nlp/examples/notebooks/LIT_sentiment_classifier.ipynb"></a></div>
+  <div class="demo-card-cta-button"><a href="/lithttps://colab.research.google.com/github/PAIR-code/lit/blob/main/lit_nlp/examples/notebooks/LIT_sentiment_classifier.ipynb"></a></div>
 </div>
 <div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
   <div class="demo-card-title"><a href="/lit/demos/coref.html" target="_blank">Gender bias in coreference systems</a></div>
@@ -139,6 +139,28 @@
   </div>
   <div class="demo-card-copy">Use a T5 model to summarize text. For any example of interest, quickly find similar examples from the training set, using an approximate nearest-neighbors index.</div>
   <div class="demo-card-cta-button"><a href="/lit/demos/t5.html"></a></div>
+</div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/penguins.html" target="_blank">Tabular data</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> tabular </span>  <span class="demo-tag"> binary classification </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    Palmer Penguins
+  </div>
+  <div class="demo-card-copy">Analyze a tabular data model with LIT, including exploring partial dependence plots and automatically finding counterfactuals.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/penguins.html"></a></div>
+</div>
+<div class="demo-card mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--4-col-phone">
+  <div class="demo-card-title"><a href="/lit/demos/images.html" target="_blank">Image classification</a></div>
+  <div class="demo-card-tags"> <span class="demo-tag"> images </span>  <span class="demo-tag"> mutli-class classification </span> 
+  </div>
+  <div class="demo-card-data-source-title">DATA SOURCES</div>
+  <div class="demo-card-data-source">
+    Imagenette
+  </div>
+  <div class="demo-card-copy">Analyze an image classification model with LIT, including muliple image salience techniques.</div>
+  <div class="demo-card-cta-button"><a href="/lit/demos/images.html"></a></div>
 </div>
   </div>
 

diff --git a/docs/demos/penguins.html b/docs/demos/penguins.html
@@ -0,0 +1,5 @@
+<html>
+  <script>
+    window.location.replace("http://35.222.78.90/");
+  </script>
+</html>
diff --git a/docs/faqs/index.html b/docs/faqs/index.html
@@ -101,7 +101,7 @@ <h3 class="faq-title">How many datapoints can LIT handle?</h3>
   <h3 class="faq-title">What kinds of models can LIT handle?</h3>
   <div class="faq-copy">
 <p>LIT can handle a variety of models, regardless of ML framework used, or output modality of the model.
-It is not limited to a strict set of model types. For more information, see the <a href="https://github.com/PAIR-code/lit/blob/main/documentation/development.md">developers guide</a>.</div></p>
+It is not limited to a strict set of model types. For more information, see the <a href="https://github.com/PAIR-code/lit/wiki/">documentation</a>.</div></p>
 </div>
 <div class="faq-element">
   <h3 class="faq-title">I have proprietary data. Is LIT secure for my team to use?</h3>

diff --git a/docs/index.html b/docs/index.html
@@ -98,8 +98,8 @@
 <li>Does my model behave consistently if I change things like textual style, verb tense, or pronoun gender?</li>
 </ul>
 <p>LIT contains many built-in capabilities but is also customizable, with the ability to add custom interpretability techniques, metrics calculations, counterfactual generators, visualizations, and more.</p>
-<p>For a similar tool to explore general-purpose machine learning models, check out the <a href="https://whatif-tool.dev">What-If Tool</a>.</p>
-<p>LIT can be run as a standalone server, or inside of python notebook environments such as Colab and Jupyter.</p>
+<p>LIT also contains preliminary support for non-language models, working with tabular and image data. For a similar tool built to explore general-purpose machine learning models, check out the <a href="https://whatif-tool.dev">What-If Tool</a>.</p>
+<p>LIT can be run as a standalone server, or inside of python notebook environments such as Colab, Jupyter, and Google Cloud Vertex AI Notebooks.</p>
 </div>
 <div class="spacer" style="height:50px;"></div>
 <div class="display2">Flexible and powerful model probing</div>
@@ -113,6 +113,7 @@ <h3>Built-in capabilities</h3>
 <p>Counterfactual generation</p>
 <p>Model and datapoint comparison</p>
 <p>Embedding visualization</p>
+<p>Testing with Concept Activation Vectors</p>
 <p>And more...</p>
 </div>
 </div>
@@ -125,6 +126,7 @@ <h3>Supported task types</h3>
 <p>Masked language models</p>
 <p>Span labeling</p>
 <p>Multi-headed models</p>
+<p>Image and tabular data support</p>
 <p>And more...</p>
 </div>
 </div>
@@ -144,20 +146,20 @@ <h3>Framework agnostic</h3>
 <div class="spacer" style="height:50px;"></div>
 <h2>What’s the latest</h2>
 <div class="mdl-grid no-padding">
-<div class="mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--8-col-phone">
-  <img class="home-card-image" src="/lit/assets/images/LIT_Contribute.png"/>
-  <div class="home-card-action">CODE</div>
-  <h3 class="home-card-title">Contribute to LIT</h3>
-  <div class="home-card-desc">LIT is open to anyone who wants to help develop and improve it!</div>
-  <div class="home-card-cta-button"><a href="https://github.com/PAIR-code/lit/blob/main/documentation/development.md" target="_blank" >View developer guide</a><img class="external-arrow" src="/lit/assets/images/arrow-link-out.png"/></div>
-</div>
 <div class="mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--8-col-phone">
   <img class="home-card-image" src="/lit/assets/images/LIT_Updates.png"/>
   <div class="home-card-action">UPDATES</div>
-  <h3 class="home-card-title">Latest updates</h3>
+  <h3 class="home-card-title">Version 0.4</h3>
   <div class="home-card-desc">New features, updates, and improvements to LIT.</div>
   <div class="home-card-cta-button"><a href="https://github.com/PAIR-code/lit/blob/main/RELEASE.md" target="_blank" >See release notes</a><img class="external-arrow" src="/lit/assets/images/arrow-link-out.png"/></div>
 </div>
+<div class="mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--8-col-phone">
+  <img class="home-card-image" src="/lit/assets/images/LIT_Contribute.png"/>
+  <div class="home-card-action">CODE</div>
+  <h3 class="home-card-title">Contribute to LIT</h3>
+  <div class="home-card-desc">LIT is open to anyone who wants to help develop and improve it!</div>
+  <div class="home-card-cta-button"><a href="https://github.com/PAIR-code/lit/wiki" target="_blank" >View documentation</a><img class="external-arrow" src="/lit/assets/images/arrow-link-out.png"/></div>
+</div>
 <div class="mdl-cell mdl-cell--6-col mdl-cell--4-col-tablet mdl-cell--8-col-phone">
   <img class="home-card-image" src="/lit/assets/images/LIT_Paper.png"/>
   <div class="home-card-action">RESEARCH</div>