Release/0.9.2 (#134)

* Sprk 361 - Merging run configured solver into run solvers (#123) * Updating run solvers to start a job per solver instead of per instance/run combination * Merging run_configured_solver into run_solvers * updating file names to be more clear, adding place holder test for run solvers * Test fix * Fixing specific configuration running * Update changelog * Bugfixes for selection example * Removing Validation directory from Platform * Reference_List moving into Settings directory * Bugfixes for snapshots * Adding sparkle citation * CLI execution rights fix * Release/0.9.1.1 (#125) * Pathing typos * Updating changelog * Adding init to core module * Release/0.9.1.2 (#126) (#127) * Sprk 361 - Merging run configured solver into run solvers (#123) * Updating run solvers to start a job per solver instead of per instance/run combination * Merging run_configured_solver into run_solvers * updating file names to be more clear, adding place holder test for run solvers * Test fix * Fixing specific configuration running * Update changelog * Bugfixes for selection example * Removing Validation directory from Platform * Reference_List moving into Settings directory * Bugfixes for snapshots * Adding sparkle citation * CLI execution rights fix * Release/0.9.1.1 (#125) * Pathing typos * Updating changelog * Adding init to core module * Patchfix * Fixed command specification * Adding images to platform documentation * Sprk 364 (#128) * Fixing Feature data for SMAC2 configuration * Updating SMAC3 to take features from FDF * Adding feature data to IRACE * Bugfixes smac3 tests * SMAC2 test update * Irace fixes * Changelog update * Add prefix for cli options (#129) * Sprk 367 - Tutorial tips bug fixes (#130) * Fixing warning messages * Adding python3 infront of python script calls to avoid execution right issues * Changing command log dir naming * Updating Sparkle wait to give better information * Fixing table to change layout once finished * Fixing platform bug where changing dir does not happen on time when executing commands in subdir * Various fixes based on tutorial * Adding documentation for venv * updating logging pytest * Updating Venv installation manual * Adding CLI cores option to SMAC2 * flake8 * Bugfix settings * Bugfixes regarding overscheduling of jobs, adding better names to configuration jobs * Enabling terminal autocompletion of commands (#71) * environment yml fix * Adding in autocompletion to the CLI functionalities * Adding readme * Changelog * Better names * Renamiong file * updating install msg * Script bugfixes * Updating solution verifier argument docs * Changelog fix * flake8 * bugfixes * various cli bugfixes for autocomplete * Adding sparkle autocomplete to bash profile for installation * Updating documentation * RunRunner upgrade * SPRK 343: Merging wait and cancel command into jobs command (#132) * Updating sparkle cancel command to have all functionalities from sparkle wait * Adding comments * Minor bug fixes and performance updates * Renaming cancel command to jobs, adapting CLI args accordingly * Deleting wait command as it is now handled by the jobs command, updating docs accordingly * Changelog mistake * Bugfixes for interactive table * Changelog update, version change, shell script updates * New release tarball * Attempted bug fixes for Github pipeline --------- Co-authored-by: Hadar Shavit <[email protected]>
ADA-research · Jan 7, 2025 · 2499beb · 2499beb
1 parent 399ef27
commit 2499beb
Show file tree

Hide file tree

Showing 71 changed files with 2,623 additions and 580 deletions.
diff --git a/.github/actions/setup_slurm_action/action.yml b/.github/actions/setup_slurm_action/action.yml
@@ -44,7 +44,7 @@ runs:
                     SlurmdSpoolDir: /tmp/slurmd # the default /var/lib/slurm/slurmd does not work because of noexec mounting in github actions
                     StateSaveLocation: /var/lib/slurm/slurmctld
                     AccountingStorageType: accounting_storage/slurmdbd
-                    SelectType: select/cons_res
+                    SelectType: select/cons_tres
                 slurmdbd_config:
                     StorageType: accounting_storage/mysql
                     PidFile: /run/slurmdbd.pid
@@ -86,7 +86,7 @@ runs:
       shell: bash -e {0}
       run: |
         sudo retry --until=success -- sacctmgr -i create account "Name=runner"
-        sudo sacctmgr -i create user "Name=runner" "Account=runner"
+        sudo retry --until=success -- sacctmgr -i create user "Name=runner" "Account=runner"
 
     - name: Test srun submission
       shell: bash -e {0}

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,23 @@
 
 Notable changes to Sparkle will be documented in this file.
 
+
+## [0.9.2] - 07/01/2025
+
+### Fixed
+- Bugs regarding using Configuration with instance features [SPRK-364]
+- Allow user to easily recompile runsolver with initialise command for Venv support [SPRK-367]
+- Added performance data argument to clean up command [SPRK-367]
+- Updated wait command to yield more information when jobs are finished [SPRK-367]
+
+### Added
+- Added documentation on how to use Venv instead of Conda for Sparkle [SPRK-367]
+- Added autocompletion for Sparkle CLI [SPRK-332]
+- Cancelling a job through the interactive table now has a confirmation dialog [SPRK-343]
+
+### Changed
+- The wait command and cancel command have been merged into the `jobs` command [SPRK-343]
+
 ## [0.9.1.2] - 2025/12/12
 
 Patchfix

diff --git a/Documentation/requirements.txt b/Documentation/requirements.txt
@@ -7,4 +7,4 @@ sphinx-copybutton
 sphinx_rtd_theme==2.0.0
 pandas # Also in environment.yml but needed for sphinx
 filelock==3.15.1
-RunRunner==0.1.8
+RunRunner==0.1.9.1
diff --git a/Documentation/source/commandlist.md b/Documentation/source/commandlist.md
@@ -2,24 +2,21 @@
 - {ref}`cmd-add-feature-extractor`
 - {ref}`cmd-add-instances`
 - {ref}`cmd-add-solver`
-- {ref}`cmd-cancel`
 - {ref}`cmd-cleanup`
 - {ref}`cmd-compute-features`
 - {ref}`cmd-compute-marginal-contribution`
 - {ref}`cmd-configure-solver`
 - {ref}`cmd-construct-portfolio-selector`
 - {ref}`cmd-generate-report`
 - {ref}`cmd-initialise`
+- {ref}`cmd-jobs`
 - {ref}`cmd-load-snapshot`
 - {ref}`cmd-remove-feature-extractor`
 - {ref}`cmd-remove-instances`
 - {ref}`cmd-remove-solver`
 - {ref}`cmd-run-ablation`
-- {ref}`cmd-run-configured-solver`
 - {ref}`cmd-run-parallel-portfolio`
 - {ref}`cmd-run-portfolio-selector`
 - {ref}`cmd-run-solvers`
 - {ref}`cmd-save-snapshot`
 - {ref}`cmd-status`
-- {ref}`cmd-validate-configured-vs-default`
-- {ref}`cmd-wait`
diff --git a/Documentation/source/commands.md b/Documentation/source/commands.md
@@ -20,7 +20,7 @@ srun -N1 -n1 -c1 path/to/package/sparkle/CLI/initialise.py
 ```
 
 In the `Examples/` directory a number of common command sequences are given. For instance, for configuration with specified training and
-testing sets see e.g. `Examples/configuration_runtime.sh` for an example of a sequence of commands to execute. Note that some command run in the background and need time to complete before the next command is executed. To see whether a command is still running the `wait` command can be used.
+testing sets see e.g. `Examples/configuration_runtime.sh` for an example of a sequence of commands to execute. Note that some command run in the background and need time to complete before the next command is executed. To see whether a command is still running the `jobs` command can be used.
 
 In the `Output/` directory paths to generated scripts and logs are gathered per executed command.
 

diff --git a/Documentation/source/commandsautoprogram.md b/Documentation/source/commandsautoprogram.md
@@ -30,14 +30,6 @@
 
 ```
 
-(cmd-cancel)=
-
-```{eval-rst}
-.. autoprogram:: sparkle.CLI.cancel:parser_function()
-   :prog: cancel
-
-```
-
 (cmd-cleanup)=
 
 ```{eval-rst}
@@ -94,6 +86,14 @@
 
 ```
 
+(cmd-jobs)=
+
+```{eval-rst}
+.. autoprogram:: sparkle.CLI.jobs:parser_function()
+   :prog: jobs
+
+```
+
 (cmd-load-snapshot)=
 
 ```{eval-rst}
@@ -173,11 +173,3 @@
    :prog: status
 
 ```
-
-(cmd-wait)=
-
-```{eval-rst}
-.. autoprogram:: sparkle.CLI.wait:parser_function()
-   :prog: wait
-
-```
diff --git a/Documentation/source/conf.py b/Documentation/source/conf.py
@@ -23,7 +23,7 @@
 from pygments.formatters.terminal import TerminalFormatter
 
 import sparkle.CLI
-from sparkle.CLI.cli import commands
+from sparkle.CLI._cli_ import commands
 
 # Add path to sparkle root
 sys.path.append(os.path.abspath("../../sparkle"))

diff --git a/Documentation/source/images/sparkle_platform_base.drawio b/Documentation/source/images/sparkle_platform_base.drawio
@@ -0,0 +1,76 @@
+<mxfile host="app.diagrams.net" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Safari/605.1.15" version="25.0.3">
+  <diagram name="Page-1" id="vNwlJehTaBxg6yKafy_n">
+    <mxGraphModel dx="908" dy="731" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
+      <root>
+        <mxCell id="0" />
+        <mxCell id="1" parent="0" />
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-1" value=".../my_directory" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
+          <mxGeometry x="280" y="520" width="160" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-3" value="Extractors" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;dashed=1;" vertex="1" parent="1">
+          <mxGeometry y="380" width="120" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-2" value="Solver&lt;span style=&quot;white-space: pre;&quot;&gt;s&lt;/span&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;dashed=1;" vertex="1" parent="1">
+          <mxGeometry y="440" width="120" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-5" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-1" target="dhtb9G5XnYohRwvSJTVj-3">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="420" y="480" as="sourcePoint" />
+            <mxPoint x="470" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-4" value="Instances" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;dashed=1;" vertex="1" parent="1">
+          <mxGeometry x="160" y="440" width="120" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-6" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-1" target="dhtb9G5XnYohRwvSJTVj-4">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="420" y="480" as="sourcePoint" />
+            <mxPoint x="470" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-7" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-1" target="dhtb9G5XnYohRwvSJTVj-2">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="420" y="480" as="sourcePoint" />
+            <mxPoint x="470" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-8" value="Settings" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" vertex="1" parent="1">
+          <mxGeometry x="300" y="440" width="120" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-10" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-1" target="dhtb9G5XnYohRwvSJTVj-8">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="360" y="480" as="sourcePoint" />
+            <mxPoint x="410" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-11" value="&lt;b&gt;Output&lt;/b&gt;" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;strokeWidth=2;" vertex="1" parent="1">
+          <mxGeometry x="440" y="440" width="120" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-13" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-1" target="dhtb9G5XnYohRwvSJTVj-11">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="360" y="480" as="sourcePoint" />
+            <mxPoint x="410" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-14" value="&lt;i&gt;sparkle_settings.ini&lt;/i&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;fontColor=#333333;strokeColor=#666666;dashed=1;dashPattern=8 8;" vertex="1" parent="1">
+          <mxGeometry x="300" y="389" width="120" height="30" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-16" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-8" target="dhtb9G5XnYohRwvSJTVj-14">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="360" y="480" as="sourcePoint" />
+            <mxPoint x="410" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-17" value="Snapshots" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;opacity=50;" vertex="1" parent="1">
+          <mxGeometry x="600" y="440" width="120" height="40" as="geometry" />
+        </mxCell>
+        <mxCell id="dhtb9G5XnYohRwvSJTVj-18" value="" style="endArrow=none;html=1;rounded=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;opacity=50;" edge="1" parent="1" source="dhtb9G5XnYohRwvSJTVj-1" target="dhtb9G5XnYohRwvSJTVj-17">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="360" y="480" as="sourcePoint" />
+            <mxPoint x="410" y="430" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+      </root>
+    </mxGraphModel>
+  </diagram>
+</mxfile>
diff --git a/Documentation/source/images/sparkle_platform_base.png b/Documentation/source/images/sparkle_platform_base.png