🔬 dnn inference

dnn_inference is a Python module for hypothesis testing based on black-box models, including deep neural networks.

• GitHub repo: https://github.com/statmlben/dnn-inference
• Documentation: https://dnn-inference.readthedocs.io
• PyPi: https://pypi.org/project/dnn-inference
• Open Source: MIT license
• Paper: arXiv:2103.04985

🎯 What We Can Do

dnn-inference is able to provide an asymptotically valid p-value to examine if $S$ is discriminative features to predict $Y$. Specifically, the proposed testing is: $$H_0: R(f^*) - R_{S}(g^*) = 0, H_a: R(f^*) - R_{S}(g^*) < 0,$$ where $X_S$ is a collection of hypothesized features, $R$ and $R_S$ are risk functions with/without the hypothesized features $X_S$, and $f^*$ and $g^*$ are population minimizers on $R$ and $R_S$ respectively. The proposed test just considers the difference between the best predictive scores with/without hypothesized features. Please check more details in our paper arXiv:2103.04985.

• When log-likelihood is used as a loss function, then the test is equivalent to a conditional independence test: $Y indep X_{S} | X_{S^c}$.
• Only a small number of fitting on neural networks is required, and the number can be as small as 1.
• Asymptotically Type I error control and power consistency.

Installation

Dependencies

dnn-inference requires: Python>=3.8 + requirements.txt

pip install -r requirements.txt

User installation

Install dnn-inference using pip

pip install dnn_inference
pip install git+https://github.com/statmlben/dnn-inference.git

Reference

If you use this code please star the repository and cite the following paper:

@article{dai2022significance,
   title={Significance Tests of Feature Relevance for a Black-Box Learner},
   author={Dai, Ben and Shen, Xiaotong and Pan, Wei},
   journal={IEEE Transactions on Neural Networks and Learning Systems},
   year={2022},
   publisher={IEEE}
}

Notebook

• MNIST dataset: Notebook1
• Boston house prices dataset: Notebook2