Merge branch 'main' into pdl-95

IBM · Sep 3, 2024 · af14815 · af14815
2 parents 890154a + d3b362e
commit af14815
Show file tree

Hide file tree

Showing 36 changed files with 2,261 additions and 976 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -39,4 +39,12 @@ jobs:
     - name: pre-commit checks
       run: pre-commit run -a
     - name: run tests
+      env:
+        WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }}
+        WATSONX_KEY: ${{ secrets.WATSONX_KEY }}
+        WATSONX_API: ${{ secrets.WATSONX_API }}
+        GENAI_KEY: ${{ secrets.GENAI_KEY }}
+        GENAI_API: ${{ secrets.GENAI_API }}
       run: py.test -v --capture=tee-sys tests
+
+
diff --git a/.travis.yml b/.travis.yml
diff --git a/README.md b/README.md
@@ -21,8 +21,8 @@ The PDL interpreter (`pdl/pdl.py`) takes a PDL program as input and renders it i
 
 See below for installation notes, followed by an [overview](#overview) of the language. A more detailed description of the language features can be found in this [tutorial](https://ibm.github.io/prompt-declaration-language/tutorial).
 
-## Demo Video
-
+## Demo Video 
+ 
 https://github.com/user-attachments/assets/2629bf1e-bc54-4c45-b598-47914ab05a45
 
 
@@ -46,7 +46,7 @@ In order to run the examples that use foundation models hosted on IBM's [watsonx
 - `WATSONX_APIKEY`, the API key (see information on [key creation](https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui#create_user_key))
 - `WATSONX_PROJECT_ID`, the project hosting the resources (see information about [project creation](https://www.ibm.com/docs/en/watsonx/saas?topic=projects-creating-project) and [finding project ID](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-project-id.html?context=wx)).
 
-
+ 
 To run the interpreter:
 
 ```
@@ -62,21 +62,28 @@ The interpreter prints out a log by default in the file `log.txt`. This log cont
 To change the log filename, you can pass it to the interpreter as follows:
 
 ```
-python3 -m pdl.pdl --log <my-logfile> <my-example>
+python -m pdl.pdl --log <my-logfile> <my-example>
 ```
 
 We can also pass initial data to the interpreter to populate variables used in a PDL program, as follows:
 
 ```
-python3 -m pdl.pdl --data <JSON-or-YAML-data> <my-example>
+python -m pdl.pdl --data <JSON-or-YAML-data> <my-example>
 ```
 
 This can also be done by passing a JSON or YAML file:
 
 ```
-python3 -m pdl.pdl --data_file <JSON-or-YAML-file> <my-example>
+python -m pdl.pdl --data_file <JSON-or-YAML-file> <my-example>
 ```
 
+The interpreter can also output a trace file that is used by the Live Document visualization tool (see [Live Document](#live_document)):
+
+```
+python -m pdl.pdl --trace <json | yaml> <my-example>
+```
+
+
 ## Overview
 
 In PDL, we can write some YAML to create a prompt and call an LLM:
@@ -335,6 +342,20 @@ The data block takes various variables and combines their values into a JSON obj
 See [PDL Language Tutorial](https://ibm.github.io/prompt-declaration-language/tutorial)
 
 
+## Live Document Visualizer
+
+PDL has a Live Document visualizer to help in program understanding given an execution trace.
+To produce an execution trace consumable by the Live Document, you can run the interpreter with the `--trace` argument and set the value to either `json` or `yaml`:
+
+```
+python -m pdl.pdl --trace <json | yaml> <my-example>
+```
+
+This produces an additional file named `my-example_result.json` or `my-example_result.yaml` that can be uploaded to the [Live Document](https://ibm.github.io/prompt-declaration-language/viewer/) visualizer tool. Clicking on different parts of the Live Document will show the PDL code that produced that part 
+in the right pane. 
+
+This is similar to a spreadsheet for tabular data, where data is in the forefront and the user can inspect the formula that generates the data in each cell. In the Live Document, cells are not uniform but can take arbitrary extents. Clicking on them similarly reveals the part of the code that produced them.
+
 
 ## Additional Notes and Future Work
 TODO

diff --git a/docs/README.md b/docs/README.md
@@ -339,6 +339,19 @@ The data block takes various variables and combines their values into a JSON obj
 
 See [PDL Language Tutorial](https://ibm.github.io/prompt-declaration-language/tutorial)
 
+## Live Document Visualizer
+
+PDL has a Live Document visualizer to help in program understanding given an execution trace.
+To produce an execution trace consumable by the Live Document, you can run the interpreter with the `--trace` argument and set the value to either `json` or `yaml`:
+
+```
+python -m pdl.pdl --trace <json | yaml> <my-example>
+```
+
+This produces an additional file named `my-example_result.json` or `my-example_result.yaml` that can be uploaded to the [Live Document](https://ibm.github.io/prompt-declaration-language/viewer/) visualizer tool. Clicking on different parts of the Live Document will show the PDL code that produced that part 
+in the right pane. 
+
+This is similar to a spreadsheet for tabular data, where data is in the forefront and the user can inspect the formula that generates the data in each cell. In the Live Document, cells are not uniform but can take arbitrary extents. Clicking on them similarly reveals the part of the code that produced them.
 
 
 ## Additional Notes and Future Work

diff --git a/docs/tutorial.md b/docs/tutorial.md
@@ -1277,6 +1277,21 @@ result, document, _, _ = process_prog(state, empty_scope, data)
 where `result` is a JSON object, and `document` is a string, both corresponding to the result of the program.
 
 
+## Live Document Visualizer
+
+PDL has a Live Document visualizer to help in program understanding given an execution trace.
+To produce an execution trace consumable by the Live Document, you can run the interpreter with the `--trace` argument and set the value to either `json` or `yaml`:
+
+```
+python -m pdl.pdl --trace <json | yaml> <my-example>
+```
+
+This produces an additional file named `my-example_result.json` or `my-example_result.yaml` that can be uploaded to the [Live Document](https://ibm.github.io/prompt-declaration-language/viewer/) visualizer tool. Clicking on different parts of the Live Document will show the PDL code that produced that part 
+in the right pane. 
+
+This is similar to a spreadsheet for tabular data, where data is in the forefront and the user can inspect the formula that generates the data in each cell. In the Live Document, cells are not uniform but can take arbitrary extents. Clicking on them similarly reveals the part of the code that produced them.
+
+
 
 
 

diff --git a/examples/arith/Arith-litellm.pdl b/examples/arith/Arith-litellm.pdl
@@ -0,0 +1,68 @@
+description: Arithmetic Expressions
+document:
+- |
+  Question: Noah charges $60 for a large painting and $30 for a small painting.
+  Last month he sold eight large paintings and four small paintings.
+  If he sold twice as much this month, how much is his sales for this month?
+
+  Answer: Let's think step by step.
+  He sold 8 large paintings and 4 small paintings last month.
+  He sold twice as many this month.
+  8 large paintings x $60 = << 8*60= 480 >> 480
+  4 small paintings x $30 = << 4*30= 120 >> 120
+  So he sold << 480+120= 600 >> 600 paintings last month.
+  Therefore he sold << 600*2= 1200 >> this month.
+  The answer is $1200.
+
+  Question: Noah charges $30 for a large vases and $10 for a small vases.
+  Last month he sold five large vases and three small vases.
+  If he sold three times as much this month, how much is his sales for this month?
+
+  Answer: Let's think step by step.
+  He sold 5 large vases and 3 small vases last month.
+  He sold three times as many this month.
+  5 large vases x $30 = << 5*30= 150 >> 150
+  3 small vases x $10 = << 3*10= 30 >> 30
+  So he sold << 150+30= 180 >> 180 vases last month.
+  Therefore he sold << 180*3= 540 >> this month.
+  The answer is $540.
+
+- repeat:
+    document:
+    - "\nQuestion: "
+    - def: QUESTION
+      model: watsonx/ibm/granite-20b-code-instruct
+      parameters:
+        stop_sequences:
+        - Answer
+        include_stop_sequence: false
+    - "Answer: Let's think step by step.\n"
+    - repeat:
+        document:
+        - def: REASON_OR_CALC
+          model: watsonx/ibm/granite-20b-code-instruct
+          parameters:
+            stop:
+            - '<<'
+            include_stop_sequence: true
+        - if: '{{ REASON_OR_CALC.endswith("<<") }}'
+          then:
+            document:
+            - def: EXPR
+              model: watsonx/ibm/granite-20b-code-instruct
+              parameters:
+                stop:
+                - '='
+                - "\n"
+                include_stop_sequence: false
+            - '= '
+            - def: RESULT
+              lan: python
+              code: result = {{ EXPR }}
+            - ' >>'
+          else: ""
+      until: '{{ "The answer is" in REASON_OR_CALC }}'
+      as: document
+    - "\n\n"
+  num_iterations: 3
+  as: document
diff --git a/examples/arith/Arith.pdl b/examples/arith/Arith.pdl
@@ -32,7 +32,7 @@ document:
     - "\nQuestion: "
     - def: QUESTION
       model: ibm/granite-20b-code-instruct
-      parameters:
+      params:
         stop_sequences:
         - Answer
         include_stop_sequence: false
@@ -41,7 +41,7 @@ document:
         document:
         - def: REASON_OR_CALC
           model: ibm/granite-20b-code-instruct
-          parameters:
+          params:
             stop_sequences:
             - '<<'
             include_stop_sequence: true
@@ -50,7 +50,7 @@ document:
             document:
             - def: EXPR
               model: ibm/granite-20b-code-instruct
-              parameters:
+              params:
                 stop_sequences:
                 - '='
                 - "\n"

diff --git a/examples/arith/litellm.txt b/examples/arith/litellm.txt
@@ -0,0 +1,76 @@
+Question: Noah charges $60 for a large painting and $30 for a small painting.
+Last month he sold eight large paintings and four small paintings.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 8 large paintings and 4 small paintings last month.
+He sold twice as many this month.
+8 large paintings x $60 = << 8*60= 480 >> 480
+4 small paintings x $30 = << 4*30= 120 >> 120
+So he sold << 480+120= 600 >> 600 paintings last month.
+Therefore he sold << 600*2= 1200 >> this month.
+The answer is $1200.
+
+Question: Noah charges $30 for a large vases and $10 for a small vases.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as many this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10= 30 >> 30
+So he sold << 150+30= 180 >> 180 vases last month.
+Therefore he sold << 180*3= 540 >> this month.
+The answer is $540.
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold 50 small paintings last month.
+Therefore he sold 50 x 2 = << 50*2 = 100 >> 100 small paintings this month.
+The answer is 100
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+ Answer: Let's think step by step.
+ He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10 = 50 >>  50
+So he sold 50 small paintings last month.
+Therefore he sold 50 x 2 = << 50*2 = 100 >>  100 small paintings this month.
+The answer is 100
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+  Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10  = 50 >>    50
+So he sold 50 small paintings last month.
+Therefore he sold 50 x 2 = << 50*2   = 100 >>    100 small paintings this month.
+The answer is 100 
+
+Question:  
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+    Answer: Let's think step by step.
+    He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10     = 50 >>
diff --git a/examples/arith/watsonx.txt b/examples/arith/watsonx.txt
@@ -0,0 +1,71 @@
+Question: Noah charges $60 for a large painting and $30 for a small painting.
+Last month he sold eight large paintings and four small paintings.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 8 large paintings and 4 small paintings last month.
+He sold twice as many this month.
+8 large paintings x $60 = << 8*60= 480 >> 480
+4 small paintings x $30 = << 4*30= 120 >> 120
+So he sold << 480+120= 600 >> 600 paintings last month.
+Therefore he sold << 600*2= 1200 >> this month.
+The answer is $1200.
+
+Question: Noah charges $30 for a large vases and $10 for a small vases.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as many this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10= 30 >> 30
+So he sold << 150+30= 180 >> 180 vases last month.
+Therefore he sold << 180*3= 540 >> this month.
+The answer is $540.
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold << 50*2= 100 >> this month.
+The answer is $100.
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold three times as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold << 50*3= 150 >> this month.
+The answer is $150.
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold five times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold five times as many this month.
+5 small paintings x $10 = << 5*10= 50 >>
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold ten times as much this month, how much is his sales for this month?Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold ten times as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold << 50*10= 500 >> this month.
+The answer is $500.