Merge pull request #1526 from PAIR-code/dev

Merge LIT v1.2 onto main for release

.github/workflows/ci.yml

@@ -33,7 +33,7 @@ jobs:
     strategy:
       matrix:
         node-version: [18]
-        python-version: ["3.10", "3.11"]
+        python-version: ["3.9", "3.10", "3.11"]
     defaults:
       run:
         shell: bash -l {0}
@@ -47,7 +47,9 @@ jobs:
         python-version: ${{ matrix.python-version }}
     - name: Install Python dependencies
       run: python -m pip install -r requirements.txt
-    - name: Test Python library
+    - name: Install LIT package
+      run: python -m pip install -e .
+    - name: Test LIT
       run: |
         python -m pip install pytest
         pytest -v

.gitignore

@@ -14,7 +14,6 @@ docs/documentation/.buildinfo
 docs/documentation/.doctrees/**
 
 **/.DS_Store
-.dalle-venv/
 .tydi-venv/
 .venv/
 .vscode/

MANIFEST.in

@@ -1,2 +1,2 @@
 recursive-include lit_nlp/client/build/default *.js* *.gif *.html *.png *.svg
-include lit_nlp/examples/datasets/prompt_examples.jsonl
+include lit_nlp/examples/prompt_debugging/prompt_examples.jsonl

README.md

@@ -1,6 +1,6 @@
 # 🔥 Learning Interpretability Tool (LIT)
 
-<!--* freshness: { owner: 'lit-dev' reviewed: '2023-09-22' } *-->
+<!--* freshness: { owner: 'lit-dev' reviewed: '2024-06-25' } *-->
 
 The Learning Interpretability Tool (🔥LIT, formerly known as the Language
 Interpretability Tool) is a visual, interactive ML model-understanding tool that
@@ -21,8 +21,8 @@ LIT is built to answer questions such as:
 LIT supports a variety of debugging workflows through a browser-based UI.
 Features include:
 
-*   **Local explanations** via salience maps, attention, and rich visualization
-    of model predictions.
+*   **Local explanations** via salience maps and rich visualization of model
+    predictions.
 *   **Aggregate analysis** including custom metrics, slicing and binning, and
     visualization of embedding spaces.
 *   **Counterfactual generation** via manual edits or generator plug-ins to
@@ -90,8 +90,8 @@ git clone https://github.com/PAIR-code/lit.git && cd lit
 ```
 
 
-Note: be sure you are running Python 3.10. If you have a different version on  
-your system, use the `conda` instructions below to set up a Python 3.10 
+Note: be sure you are running Python 3.9+. If you have a different version on
+your system, use the `conda` instructions below to set up a Python 3.9
 environment.
 
 Set up a Python environment with `venv`:
@@ -106,7 +106,7 @@ Or set up a Python environment using `conda`:
 ```sh
 conda create --name lit-nlp
 conda activate lit-nlp
-conda install python=3.10
+conda install python=3.9
 conda install pip
 ```
 
@@ -142,13 +142,7 @@ To explore classification and regression models tasks from the popular
 [GLUE benchmark](https://gluebenchmark.com/):
 
 ```sh
-python -m lit_nlp.examples.glue_demo --port=5432 --quickstart
-```
-
-Or, using `docker`:
-
-```sh
-docker run --rm -e DEMO_NAME=glue_demo -p 5432:5432 -t lit-nlp --quickstart
+python -m lit_nlp.examples.glue.demo --port=5432 --quickstart
 ```
 
 Navigate to http://localhost:5432 to access the LIT UI.
@@ -160,19 +154,6 @@ but you can switch to
 [STS-B](http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark) or
 [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) using the toolbar or the
 gear icon in the upper right.
-
-### Quick-start: language modeling
-
-To explore predictions from a pre-trained language model (BERT or GPT-2), run:
-
-```sh
-python -m lit_nlp.examples.lm_demo --models=bert-base-uncased --port=5432
-```
-
-Or, using `docker`:
-
-```sh
-docker run --rm -e DEMO_NAME=lm_demo -p 5432:5432 -t lit-nlp --models=bert-base-uncased
 ```
 
 And navigate to http://localhost:5432 for the UI.
@@ -192,7 +173,7 @@ See [lit_nlp/examples](./lit_nlp/examples). Most are run similarly to the
 quickstart example above:
 
 ```sh
-python -m lit_nlp.examples.<example_name> --port=5432 [optional --args]
+python -m lit_nlp.examples.<example_name>.demo --port=5432 [optional --args]
 ```
 
 ## User Guide

RELEASE.md

@@ -1,5 +1,68 @@
 # Learning Interpretability Tool Release Notes
 
+## Release 1.2
+
+This release covers clean-ups on various obsolete demos, as well as improved
+packaging and isolated dependencies on the GLUE, Penguin, Prompt Debugging with
+Sequence Salience and TyDi demos for easier launch.
+
+### New Stuff
+* Improved packaging and instructions for launching Prompt Debugging with
+Sequence Salience demo, as well as minor bug fixes -
+[08289df](https://github.com/PAIR-code/lit/commit/08289df0dd9927dee7147e5aad6e8b51bbe74f9e),
+[675ca2d](https://github.com/PAIR-code/lit/commit/675ca2de21b68dc62e4909c80a2cd57d8ee8b601),
+[15eccb1](https://github.com/PAIR-code/lit/commit/15eccb1197366c925a5beff310fb5d7d369bde0c),
+[e0e35c3](https://github.com/PAIR-code/lit/commit/e0e35c3ffcfd9ad5331d4154e7d33d0b1d0daf89),
+[c7970fb](https://github.com/PAIR-code/lit/commit/c7970fb8c51d2a8bd3647cc7eedd15cca285ac08),
+[cee3b58](https://github.com/PAIR-code/lit/commit/cee3b58baea2de27633109e6dd5b3e4211fa46ea)
+
+* Clean up of obsolete demos -
+[b16059f](https://github.com/PAIR-code/lit/commit/b16059fbd0320d411298009c0226489e1f548a69),
+[f4c0990](https://github.com/PAIR-code/lit/commit/f4c099082f0e89986aad162cc3cd0ac9bc2214c7),
+[6aa2eb6](https://github.com/PAIR-code/lit/commit/6aa2eb64eddb8ca154401bfd6a039762bc374d6d),
+[c2fb41b](https://github.com/PAIR-code/lit/commit/c2fb41b4945edb91fac973cf0ddbca48c6257511),
+[dd196e9](https://github.com/PAIR-code/lit/commit/dd196e941058a1d4246b3df3a3c37595f9791b18),
+[72fd772](https://github.com/PAIR-code/lit/commit/72fd772fa02c7445f27fb517e667987ea8ab34d7),
+[71d88fb](https://github.com/PAIR-code/lit/commit/71d88fb86eb88ffb80d665cf7571b21d7ae06bd2),
+[aa49340](https://github.com/PAIR-code/lit/commit/aa493409c454a2ed269fdedd15353404c14b4936),
+[fc7b0d0](https://github.com/PAIR-code/lit/commit/fc7b0d0624f6cc8e456ac0a1d75a4149927bef2f),
+[2475b3b](https://github.com/PAIR-code/lit/commit/2475b3bb677c8685ab9a291c490783ae2ccce5b8),
+[a59641c](https://github.com/PAIR-code/lit/commit/a59641c014b17409e8e5cfdac1cc1e6916d6da15),
+[1ed82d4](https://github.com/PAIR-code/lit/commit/1ed82d4e81ff6a6ff5146b6198e35444960d326b),
+[7d5ef58](https://github.com/PAIR-code/lit/commit/7d5ef5831427de71416c096a6dbcd46ea064457e),
+[992823b](https://github.com/PAIR-code/lit/commit/992823b027fca8c60edabe837248a508ac04da22),
+[3dad2b0](https://github.com/PAIR-code/lit/commit/3dad2b061b45cb44b1c3f9b9364660e907662069),
+[0656386](https://github.com/PAIR-code/lit/commit/0656386188d6e4b6c83dab58fb4e6569ebea217e),
+[27d7a84](https://github.com/PAIR-code/lit/commit/27d7a841cf6d514e67ebfb2af9f603398499f6e3),
+[8863019](https://github.com/PAIR-code/lit/commit/886301972ec1e7ed274040b46ec0e0c3f34c8ace),
+[71cbdba](https://github.com/PAIR-code/lit/commit/71cbdbaee0fee8e96f52cd4df7a269a0873b9259),
+[416d573](https://github.com/PAIR-code/lit/commit/416d573d79f84b9a6964d36e498b850a249ef452)
+
+* Python requirements update and isolated setup for individual demos -
+[bcc481e](https://github.com/PAIR-code/lit/commit/bcc481e44185d04268f5f8bb4ba762ec2cd35907),
+[bb29f43](https://github.com/PAIR-code/lit/commit/bb29f430ff7be55d74a82aec5dee1e54fa27bed0),
+[fbd8874](https://github.com/PAIR-code/lit/commit/fbd88746263fec0f72f2f01bcc382e88e902ab50),
+[b3c120b](https://github.com/PAIR-code/lit/commit/b3c120b22138fb03a712f11778197cf4966d0c3a),
+[5188c8c](https://github.com/PAIR-code/lit/commit/5188c8c835328efcc9dff5a0a4cf4cd79fabe099),
+[5639e3b](https://github.com/PAIR-code/lit/commit/5639e3b1b71b1c0ddf4a3c9e1bd25517fba18375)
+
+* Documentation cleanup and updates -
+[afd51fe](https://github.com/PAIR-code/lit/commit/afd51fe299c0070a19946a789984957f14a9b5bb),
+[7dda659](https://github.com/PAIR-code/lit/commit/7dda659bec4e933d187b0d7afc04d954ae262cc2),
+[79ada6e](https://github.com/PAIR-code/lit/commit/79ada6edf8b2e485ec6a6425d4c60720b4dab8d1),
+[1c8d6a0](https://github.com/PAIR-code/lit/commit/1c8d6a0269ce5637e05e79ae435f770e2a0da147),
+[2e9d267](https://github.com/PAIR-code/lit/commit/2e9d26738d9344cde0eebd66d49dfc14cd800e74)
+
+### Non-breaking Changes, Bug Fixes, and Enhancements
+* Refactor DataService reactions - [483082d](https://github.com/PAIR-code/lit/commit/483082dcb0beb39795c0fc093fe93036bb6a274c)
+* Add warm_start option to LitWidget - [a5265a4](https://github.com/PAIR-code/lit/commit/a5265a4feeb701b878986f79665d5fdf9ddc244c)
+* Pretty-printing of Model objects - [4fb3bde](https://github.com/PAIR-code/lit/commit/4fb3bde897c68fdeb3bd829f6e5a88223bc131a4)
+* Avoid equivalent shuffles in Scrambler - [0d8c0d9](https://github.com/PAIR-code/lit/commit/0d8c0d948480e0835fd3f451b95b7ec306b6409d)
+* Updated gunicorn config for demos running in Docker - [b14e3b1](https://github.com/PAIR-code/lit/commit/b14e3b1a81d7b6305063f778f46666a4d1326045)
+* Disable embeddings for TyDi - [7ff377f](https://github.com/PAIR-code/lit/commit/7ff377f92820748476e796994fd207e1b5dba1d9)
+* Cast embeddings to float32 before computing distances - [5456011](https://github.com/PAIR-code/lit/commit/5456011db8ead5d53db6f39bcdca3fc388802fbe)
+* Update colab examples to include installation of the lit-nlp package - [48b029c](https://github.com/PAIR-code/lit/commit/48b029c3a1a3f25d4d2611a9b0e94355d41078ef)
+
 ## Release 1.1.1
 
 This release covers various improvements for sequence salience, including new

docs/documentation/_sources/getting_started.md.txt

@@ -76,7 +76,7 @@ Most demos can be run with a single blaze command. To run the default one, you
 can do:
 
 ```sh
-python -m lit_nlp.examples.glue_demo \
+python -m lit_nlp.examples.glue.demo \
   --quickstart --port=4321 --alsologtostderr
 ```
 

lit_nlp/api/dataset_test.py

@@ -179,7 +179,6 @@ def test_init_from_examples_without_ids(self):
         examples=examples
     )
 
-    # TODO(b/266681945): Enabled zip(..., strict=true) once updated to Py3.10
     for indexed_example, example, original in zip(
         dataset.indexed_examples, dataset.examples, examples
     ):

lit_nlp/api/layout.py

@@ -29,43 +29,39 @@ class LitModuleName(dtypes.EnumSerializableAsValues, enum.Enum):
   Entries should map the TypeScript class name to the HTML element name,
   as declared in HTMLElementTagNameMap in the .ts file defining each LitModule.
   """
-  AnnotatedTextModule = 'annotated-text-module'
+  # keep-sorted start
   AnnotatedTextGoldModule = 'annotated-text-gold-module'
-  AttentionModule = 'attention-module'
+  AnnotatedTextModule = 'annotated-text-module'
   ClassificationModule = 'classification-module'
   ConfusionMatrixModule = 'confusion-matrix-module'
   CurvesModule = 'curves-module'
   DataTableModule = 'data-table-module'
-  SimpleDataTableModule = 'simple-data-table-module'
   DatapointEditorModule = 'datapoint-editor-module'
-  # Non-replicating version of Datapoint Editor
-  SingleDatapointEditorModule = 'single-datapoint-editor-module'
-  # Simplified, non-replicating version of Datapoint Editor
-  SimpleDatapointEditorModule = 'simple-datapoint-editor-module'
   DiveModule = 'dive-module'
   DocumentationModule = 'documentation-module'
   EmbeddingsModule = 'embeddings-module'
   FeatureAttributionModule = 'feature-attribution-module'
   GeneratedImageModule = 'generated-image-module'
   GeneratedTextModule = 'generated-text-module'
   GeneratorModule = 'generator-module'
-  LanguageModelPredictionModule = 'lm-prediction-module'
-  LMSalienceModule = 'lm-salience-module'
+  LegacySequenceSalienceModule = 'legacy-sequence-salience-module'
   MetricsModule = 'metrics-module'
   MultilabelModule = 'multilabel-module'
   PdpModule = 'pdp-module'
   RegressionModule = 'regression-module'
   SalienceClusteringModule = 'salience-clustering-module'
   SalienceMapModule = 'salience-map-module'
   ScalarModule = 'scalar-module'
-  LegacySequenceSalienceModule = 'legacy-sequence-salience-module'
-  SpanGraphGoldModule = 'span-graph-gold-module'
-  SpanGraphModule = 'span-graph-module'
-  SpanGraphGoldModuleVertical = 'span-graph-gold-module-vertical'
-  SpanGraphModuleVertical = 'span-graph-module-vertical'
+  SequenceSalienceModule = 'sequence-salience-module'
+  SimpleDataTableModule = 'simple-data-table-module'
+  # Simplified, non-replicating version of Datapoint Editor
+  SimpleDatapointEditorModule = 'simple-datapoint-editor-module'
+  # Non-replicating version of Datapoint Editor
+  SingleDatapointEditorModule = 'single-datapoint-editor-module'
   TCAVModule = 'tcav-module'
-  TrainingDataAttributionModule = 'tda-module'
   ThresholderModule = 'thresholder-module'
+  TrainingDataAttributionModule = 'tda-module'
+  # keep-sorted end
 
   def __call__(self, **kw):
     return ModuleConfig(self.value, **kw)
@@ -125,12 +121,9 @@ def to_json(self) -> dtypes.JsonDict:
 modules = LitModuleName  # pylint: disable=invalid-name
 
 MODEL_PREDS_MODULES = (
-    modules.SpanGraphGoldModuleVertical,
-    modules.SpanGraphModuleVertical,
     modules.ClassificationModule,
     modules.MultilabelModule,
     modules.RegressionModule,
-    modules.LanguageModelPredictionModule,
     modules.GeneratedTextModule,
     modules.AnnotatedTextGoldModule,
     modules.AnnotatedTextModule,
@@ -204,7 +197,6 @@ def to_json(self) -> dtypes.JsonDict:
         'Explanations': [
             modules.SalienceMapModule,
             modules.LegacySequenceSalienceModule,
-            modules.AttentionModule,
             modules.FeatureAttributionModule,
         ],
         'Clustering': [modules.SalienceClusteringModule],
@@ -238,7 +230,6 @@ def to_json(self) -> dtypes.JsonDict:
             *MODEL_PREDS_MODULES,
             modules.SalienceMapModule,
             modules.LegacySequenceSalienceModule,
-            modules.AttentionModule,
             modules.FeatureAttributionModule,
         ],
         'Salience Clustering': [modules.SalienceClusteringModule],

lit_nlp/api/model.py

@@ -89,6 +89,18 @@ def description(self) -> str:
     """
     return inspect.getdoc(self) or ''
 
+  def __str__(self) -> str:
+    classname = self.__class__.__module__ + '.' + self.__class__.__qualname__
+    indented_description = '  ' + self.description().replace('\n', '\n  ')
+    return f'{classname}(...):\n{indented_description}'
+
+  def _repr_pretty_(self, p, cycle):
+    """Pretty-printing for IPython environments, both notebooks and repl."""
+    if not cycle:
+      p.text(str(self))
+    else:
+      p.text('...')
+
   @classmethod
   def init_spec(cls) -> Optional[Spec]:
     """Attempts to infer a Spec describing a Model's constructor parameters.