Title
Experiments on CIFAR & MNIST datasets with gradient encryption.

• 0.1. Introduction
• 0.2. Requirements
  • 0.2.1. software
  • 0.2.2. Hardware
• 0.3. Usage
  • 0.3.1. Clone this repository
  • 0.3.2. Edit syft_main.py relevant parameters of model
  • 0.3.3. Train
  • 0.3.4. Extension (For fun)

0.1. Introduction

The model is built for safe deep learning gradient encryption aggregation experiments on the cifar100 & mnist dataset. This realizes to use differential dataset training on the independent models then all parameters will be encrypted aggregation into specifical parameters container. Finally, the models upload parameters after gradients of each model be encrypted and aggregate together.

This structure could be built by using PySyft framework, so i also wrote Extension (For fun) to realizes it to test this solution, but not working very well. Therefore, the main scheme is to manually implement gradient encryption through the homomorphic encryption algorithm. Algorithm in:

utils/vhe.py

• Hint: this algorithm consumes extremely huge memory space lead to memory overflow even in simply CNN model. vhe.py Must be improved to manage to optimize memory space, for example, point instead of variables.

• Brach sql_ store variables into SQL database to save the memory.

Reimplement state-of-the-art CNN models in Cifar100 dataset with Pysyft, now including:

• For cifar100 dataset:

  1.ResNet

  2.PreActResNet

  3.WideResNet

  4.ResNeXt

  5.DenseNet

  6.Alexnet

• For Mnist dataset:

  7.Lenet

Other results will be added later.

0.2. Requirements

0.2.1. software

Requirements for PyTorch

And others package must be included in your environment specified in requirements.txt

0.2.2. Hardware

For now only can run on CPU.

0.3. Usage

0.3.1. Clone this repository

git clone https://github.com/Gaopeng-Bai/Gradient-Secure.git

• In this project, the network structure is defined in the "models" folder, the script secure_gradient.py is running on two separates models, gradient aggregation though Syft virtual machine. Then upload new parameters after aggregation to each model.

• In secure_gradient.py, Mnist dataset can train on lenet5, simply_cnn, simply_cnn2. but AlexNet model testing with Tiny ImageNet or MNIST could not be done due to their smaller feature sizes (images do not fit the input size 227 x 227). Mnist on lenet5 model tested on learning rate 0.01 and epoch 15 reached 99% accuracy, and on simply_cnn spent more time than lenet5.

• cifar100 dataset can train on resnet20, resnet32, resnet44, resnet110, preact_resnet110, resnet164, resnet1001, preact_resnet164, preact_resnet1001,wide_resnet, resneXt, densenet.

0.3.2. Edit syft_main.py relevant parameters of model

• In the secure_gradient.py, you can specify the model you want to train(for example):

  

  Then, you need specify some parameter for training process, like epoche....

0.3.3. Train

• Two local worker with gradient aggregation

  python secure_gradient.py
  

• Normal Model

  python Cifar100_main.py
  
  python mnist_main.py
  

0.3.4. Extension (For fun)

• In the syft_test.py, it is a cifar100 secure gradient aggregation model modified completely according to Syft tutorials 10. However, it's not working. Because when the data send to virtual machine (bob and alice).

  

• In the training, the model point the data position, input data to the current model point suddenly, it broken here.

  

  Don't know the reason why this happen yet. It seem like some wrong in PySyft framework.