Skip to content

This is the official clone for the implementation of the NIPS18 paper Multi-Layered Gradient Boosting Decision Trees (mGBDT) .

Notifications You must be signed in to change notification settings

kingfengji/mGBDT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Multi-Layered Gradient Boosting Decision Trees

This is the official clone for the implementation of mGBDT.

Package Official Website: http://lamda.nju.edu.cn/code_mGBDT.ashx

This package is provided "AS IS" and free for academic usage. You can run it at your own risk. For other purposes, please contact Prof. Zhi-Hua Zhou ([email protected]).

Description: A python implementation of mGBDT proposed in [1]. A demo implementation of mGBDT library as well as some demo client scripts to demostrate how to use the code. The implementation is flexible enough for modifying the model or fit your own datasets.

Reference: [1] J. Feng, Y. Yu, and Z.-H. Zhou. Multi-Layered Gradient Boosting Decision Trees. In:Advances in Neural Information Processing Systems 31 (NIPS'18), Montreal, Canada, 2018.

ATTN: This package was developed and maintained by Mr.Ji Feng(http://lamda.nju.edu.cn/fengj/) .For any problem concerning the codes, please feel free to contact Mr.Feng.([email protected]) or open some issues here.

Environments

  • The code is developed under Python 3.5, so create a Python 3.5 environment using anaconda at first
conda create -n mgbdt python=3.5 anaconda
  • Install the dependent packages
source activate mgbdt
conda install pytorch=0.1.12 cuda80 -c pytorch
pip install -r requirements.txt

Demo Code

from sklearn import datasets
from sklearn.model_selection import train_test_split

# For using the mgbdt library, you have to include the library directory into your python path.
# If you are in this repository's root directory, you can do it by using the following lines
import sys
sys.path.insert(0, "lib")

from mgbdt import MGBDT, MultiXGBModel

# make a sythetic circle dataset using sklearn
n_samples = 15000
x_all, y_all = datasets.make_circles(n_samples=n_samples, factor=.5, noise=.04, random_state=0)
x_train, x_test, y_train, y_test = train_test_split(x_all, y_all, test_size=0.3, random_state=0, stratify=y_all)

# Create a multi-layerd GBDTs
net = MGBDT(loss="CrossEntropyLoss", target_lr=1.0, epsilon=0.1)

# add several target-propogation layers
# F, G represent the forward mapping and inverse mapping (in this paper, we use gradient boosting decision tree)
net.add_layer("tp_layer",
    F=MultiXGBModel(input_size=2, output_size=5, learning_rate=0.1, max_depth=5, num_boost_round=5),
    G=None)
net.add_layer("tp_layer",
    F=MultiXGBModel(input_size=5, output_size=3, learning_rate=0.1, max_depth=5, num_boost_round=5),
    G=MultiXGBModel(input_size=3, output_size=5, learning_rate=0.1, max_depth=5, num_boost_round=5))
net.add_layer("tp_layer",
    F=MultiXGBModel(input_size=3, output_size=2, learning_rate=0.1, max_depth=5, num_boost_round=5),
    G=MultiXGBModel(input_size=2, output_size=3, learning_rate=0.1, max_depth=5, num_boost_round=5))

# init the forward mapping
net.init(x_train, n_rounds=5)

# fit the dataset
net.fit(x_train, y_train, n_epochs=50, eval_sets=[(x_test, y_test)], eval_metric="accuracy")

# prediction
y_pred = net.forward(x_test)

# get the hidden outputs
# hiddens[0] represent the input data
# hiddens[1] represent the output of the first layer
# hiddens[2] represent the output of the second layer
# hiddens[3] represent the output of the final layer (same as y_pred)
hiddens = net.get_hiddens(x_test)

Expriments

circle dataset

By running the following scripts

  • It will train a multi-layered GBDTs with structure (input - 5 - 3 - output) on the sythetic circle dataset
  • The visualization of the input (which is 2D) will be saved in outputs/circle/input.jpg (as show below)
  • The visualization of the second layer's output (which is 3D) will be saved in outputs/circle/pred2.jpg (as show below)
python exp/circle.py
Input Transformed

scurve dataset

By running the following scripts

  • It will train an autoencoder using multi-layered GBDTs with structure (input - 5 - output) on the sythetic scurve dataset
  • The visualization of the input (which is 3D) will be saved in outputs/circle/input.jpg (as show below)
  • The visualization of the resonstructed result (which is 3D) will be saved in outputs/circle/pred2.jpg (as show below)
python exp/scurve.py
Input Reconstructed
  • The visualization of the encoding will also be saved, since the 5D encodings are impossible to visualize directly, here we visualize every pairs of the 5 dimentions in 2D space.
  • The visualization of the $i'th and $j'th dimension will be saved in outputs/scurve/pred1.$i_$j.jpg (as show below)
Dimension 1 and 2 Dimension 1 and 5

At first, you need to download the dataset by running the following command:

cd dataset/uci_adult
sh get_data.sh

Then, by running the following scripts

  • It will train a multi-layered GBDTs with structure (input - 128 - 128 - output)
  • the accuracy will be logged for each epochs
python exp/uci_adult.py

By running the following scripts

  • It will train a multi-layered GBDTs with structure (input - 16 - 16 - output)
  • 10-fold cross-validation is used
  • the accuracy will be logged for each epochs and each folds
python exp/uci_yeast.py

Happy hacking.

About

This is the official clone for the implementation of the NIPS18 paper Multi-Layered Gradient Boosting Decision Trees (mGBDT) .

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published