91 lines
3.2 KiB
Plaintext
91 lines
3.2 KiB
Plaintext
|
# EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks
|
||
|
|
|
||
|
|
This project implements [this paper](https://arxiv.org/abs/1709.07634) in [PyTorch](pytorch.org). The implementation refers to [ResNeXt-DenseNet](https://github.com/D-X-Y/ResNeXt-DenseNet)
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
All the model definations are located in the directory `models`.
|
||
|
|
|
||
|
|
All the training scripts are located in the directory `scripts` and `Xscripts`.
|
||
|
|
|
||
|
|
To train the ResNet-110 with EraseReLU on CIFAR-10:
|
||
|
|
```bash
|
||
|
|
sh scripts/warmup_train_2gpu.sh resnet110_erase cifar10
|
||
|
|
```
|
||
|
|
|
||
|
|
To train the original ResNet-110 on CIFAR-10:
|
||
|
|
```bash
|
||
|
|
sh scripts/warmup_train_2gpu.sh resnet110 cifar10
|
||
|
|
```
|
||
|
|
|
||
|
|
### MiniImageNet for PatchShuffle
|
||
|
|
```
|
||
|
|
sh scripts-shuffle/train_resnet_00000.sh ResNet18
|
||
|
|
sh scripts-shuffle/train_resnet_10000.sh ResNet18
|
||
|
|
sh scripts-shuffle/train_resnet_11000.sh ResNet18
|
||
|
|
```
|
||
|
|
|
||
|
|
```
|
||
|
|
sh scripts-shuffle/train_pmd_00000.sh PMDNet18_300
|
||
|
|
sh scripts-shuffle/train_pmd_00000.sh PMDNet34_300
|
||
|
|
sh scripts-shuffle/train_pmd_00000.sh PMDNet50_300
|
||
|
|
|
||
|
|
sh scripts-shuffle/train_pmd_11000.sh PMDNet18_300
|
||
|
|
sh scripts-shuffle/train_pmd_11000.sh PMDNet34_300
|
||
|
|
sh scripts-shuffle/train_pmd_11000.sh PMDNet50_300
|
||
|
|
```
|
||
|
|
|
||
|
|
### ImageNet
|
||
|
|
- Use the scripts `train_imagenet.sh` to train models in PyTorch.
|
||
|
|
- Or you can use the codes in `extra_torch` to train models in Torch.
|
||
|
|
|
||
|
|
#### Group Noramlization
|
||
|
|
```
|
||
|
|
sh Xscripts/train_vgg_gn.sh 0,1,2,3,4,5,6,7 vgg16_gn 256
|
||
|
|
sh Xscripts/train_vgg_gn.sh 0,1,2,3,4,5,6,7 vgg16_gn 64
|
||
|
|
sh Xscripts/train_vgg_gn.sh 0,1,2,3,4,5,6,7 vgg16_gn 16
|
||
|
|
sh Xscripts/train_res_gn.sh 0,1,2,3,4,5,6,7 resnext50_32_4_gn 16
|
||
|
|
```
|
||
|
|
|
||
|
|
| Model | Batch Size | Top-1 Error | Top-5 Errpr |
|
||
|
|
|:--------------:|:----------:|:-----------:|:-----------:|
|
||
|
|
| VGG16-GN | 256 | 28.82 | 9.64 |
|
||
|
|
|
||
|
|
|
||
|
|
## Results
|
||
|
|
|
||
|
|
| Model | Error on CIFAR-10 | Error on CIFAR-100|
|
||
|
|
|:--------------:|:-----------------:|:-----------------:|
|
||
|
|
| ResNet-56 | 6.97 | 30.60 |
|
||
|
|
| ResNet-56 (ER) | 6.23 | 28.56 |
|
||
|
|
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
If you find this project helos your research, please consider cite the paper:
|
||
|
|
```
|
||
|
|
@article{dong2017eraserelu,
|
||
|
|
title={EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks},
|
||
|
|
author={Dong, Xuanyi and Kang, Guoliang and Zhan, Kun and Yang, Yi},
|
||
|
|
journal={arXiv preprint arXiv:1709.07634},
|
||
|
|
year={2017}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Download the ImageNet dataset
|
||
|
|
The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset has 1000 categories and 1.2 million images. The images do not need to be preprocessed or packaged in any database, but the validation images need to be moved into appropriate subfolders.
|
||
|
|
|
||
|
|
1. Download the images from http://image-net.org/download-images
|
||
|
|
|
||
|
|
2. Extract the training data:
|
||
|
|
```bash
|
||
|
|
mkdir train && mv ILSVRC2012_img_train.tar train/ && cd train
|
||
|
|
tar -xvf ILSVRC2012_img_train.tar && rm -f ILSVRC2012_img_train.tar
|
||
|
|
find . -name "*.tar" | while read NAME ; do mkdir -p "${NAME%.tar}"; tar -xvf "${NAME}" -C "${NAME%.tar}"; rm -f "${NAME}"; done
|
||
|
|
cd ..
|
||
|
|
```
|
||
|
|
|
||
|
|
3. Extract the validation data and move images to subfolders:
|
||
|
|
```bash
|
||
|
|
mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar
|
||
|
|
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash
|
||
|
|
```
|