This repository reports the code and the benchmarks for the paper "SpectAcle: Fault Localisation of AI-Enabled CPS by Exploiting Sequences of DNN Controller Inferences".
- Operating system: Linux / MacOS / Windows
- Matlab (Simulink/Stateflow) version: >= 2020a (Matlab license needed)
- Python version: >= 3.3
- MATLAB toolboxes dependency:
- Install Breach
- start matlab, set up a C/C++ compiler using the command
mex -setup
. (Refer to here for more details.) - navigate to
breach/
in Matlab command line, and runInstallBreach
- start matlab, set up a C/C++ compiler using the command
- Install Git Large File Storage
- for linux:
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash sudo apt-get install git-lfs git lfs install
- for mac:
brew install git-lfs git lfs install
- for linux:
- Retrieve the dataset stored in the form of a large file by the following command
git lfs pull
- Navigate to the directory
test/
. Run the commandpython train.py config/[benchmark]/train
. - Now the executable scripts have been generated under the directory
test/scripts/
. - Navigate to the root directory
Spectacle/
and run the commandmake
. The automatically generated NN controller files will be stored under the root directory.
- Navigate to the directory
test/
. Run the commandpython mutation.py config/[benchmark]/mutation[number]
. - Now the executable scripts have been generated under the directory
test/scripts/
. - Navigate to the root directory
Spectacle/
and run the commandmake
. The neccessary simulation logs of mutants for fault localisation will be stored under the root directory, including the diagnostic information of current test suitecur_diagInfo_suite
, the safety rate of current mutant on the given test suitecur_safety_rate
, the Boolean state of current mutant for each test casecur_state_set
(either 1 (satisfied) or 0 (violated)), and the mutator's informationmut_info
.
To reproduce the experimental results, users should follow the steps below:
- The user-specified configuration files are stored under the directory
test/config/
. Replace the paths ofSpectacle
in user-specified file under the lineaddpath 1
with their own path. Users can also specify other configurations, such as topk and tops. - Users need to edit the executable scripts permission using the command
chmod -R 777 *
. - The corresponding log will be stored under directory
output/
.
- Navigate to the directory
test/
. Run the commandpython spectacle.py config/[benchmark]/fl[number]
. - Now the executable scripts have been generated under the directory
test/scripts/
. - Navigate to the root directory
Spectacle/
and run the commandmake
. The fault localisation results will be stored under the root directory, including the mutator's informationmut_info
, different hyperparameter combinations for fault localisationhyper_config
, the execution spectra under different hyperparameter combinationscur_hyper_exe_spectrum
, the suspicious weights under different hyperparameter combinationscur_hyper_all_sps_weight
, and the suspicious scores of all weights under different hyperparameter combinations cur_hyper_all_sps_score.
Based on the simulations logs and fault localisation results, all the tables and figures can be obtained through the following scripts:
- DR Plots: RQ1_DR_plot.m
-
$AUC_{DR}$ : RQ1_AUC_DR.m -
$AUC_{FDR}$ : RQ1_AUC_FDR.m
- Statistical Comparison in terms of
$AUC_{DR}$ : RQ1_Cohens_d_DR_hyperparm.m - Statistical Comparison in terms of
$AUC_{FDR}$ : RQ1_Cohens_d_FDR_hyperparm.m
- Comparison between
$SpectAcle_{uw}$ and$SpectAcle_{w}$ : RQ3.m
RQ4: How does the selection of the suspiciousness metric SusMetr influence the effectiveness of SpectAcle?
- Statistical Comparison in terms of
$AUC_{DR}$ : RQ1_Cohens_d_DR_metric.m - Statistical Comparison in terms of
$AUC_{FDR}$ : RQ1_Cohens_d_FDR_metric.m
- Navigate to the directory
test/
. Run the commandpython contrRep4SpectAcle.py config/[benchmark]/repair[number]
. - Now the executable scripts have been generated under the directory
test/scripts/
. - Navigate to the root directory
Spectacle/
and run the commandmake
. The repair results will be stored under the root directory.
- The time cost of SpectAcle: RQ6.m