Full Demo: Training an Encrypted Neural Network across Workers using Crypten (with jails)

[youben11]

Description

This PR add an experimental notebook showing how to train an encrypted neural network using crypten, it's based on Tutorial7 of CrypTen.

Two issues were also solved to implement the demo correctly. First an issue regarding how the crypten computation is initiated in the syft context, and a second regarding how virtual workers are used.

First issue

CrypTen computation is started from a master worker that will be responsible of running the master party which needs to bind and listen to a port for synchronizing between distributed parties, our initial design start a party (rank 0) on the local worker to be the master that will do the previous job, this introduced some complication with the API as users will list workers in the run_multiworker decorator and doesn't expect an additional party to be running on the local worker. We solved this by removing the need to start a local party, but the ip address and port referred on the decorator would be the one used by the first worker in the list (more work on this need to be done).

Second issue

Using virtual workers doesn't parallelize the computation as calling send will just call recv and the execution of the commands sent to the VirtualWorker will just run sequentially on the main thread. Using threads to call operations on VirtualWorker (which was our case) doesn't solve the issue as well as most computation is cpu-bound, and python doesn't support threads as expected because of the Global Interpreter Lock. So we switched to using processes instead, this introduces some extra cost for creating these processes, but this shouldn't be a big deal if the memory is shared using copy-on-write, a deeper analysis of this may be required only if this start popping up issues.

Results

With the previous changes, we were able to reproduce the same training as in the CrypTen tutorial with little changes, the running time is almost the same even for VirtualWorker due to the parallelism using processes.

Checklist:

• My changes are covered by tests.
• I have run the pre-commit hooks to format and check my code for style issues.
• I have commented my code following Google style.

(See the the contribution guidelines for additional tips.)

[review-notebook-app]

Check out this pull request on 

Review Jupyter notebook visual diffs & provide feedback on notebooks.

---

Powered by ReviewNB

[gmuraru]

Q: It would be a good idea to also add a title and an author (you :P)?

[gmuraru]

Maybe also add some "comments" (I think you can do this in a follow up PR) regarding what is happening in each cell?

[gmuraru]

LGTM!

[gmuraru]

Tried to run the notebook but got this error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-b5df2d527789> in <module>
      6 from time import time
      7 from syft import WebsocketClientWorker
----> 8 from syft.frameworks.crypten.context import run_multiworkers
      9 
     10 

ModuleNotFoundError: No module named 'syft.frameworks.crypten.context'

Is there something special that I need to change?

[gmuraru]

Got it!

[%] Starting computation
[+] run_encrypted_training() took 32s
Epoch 0 in progress:
	Batch 1 of 10 Loss 0.4638
	Batch 2 of 10 Loss 0.4666
	Batch 3 of 10 Loss 0.4065
	Batch 4 of 10 Loss 0.3488
	Batch 5 of 10 Loss 0.3314
	Batch 6 of 10 Loss 0.2796
	Batch 7 of 10 Loss 0.2767
	Batch 8 of 10 Loss 0.2432
	Batch 9 of 10 Loss 0.2456
	Batch 10 of 10 Loss 0.2003
Epoch 1 in progress:
	Batch 1 of 10 Loss 0.1625
	Batch 2 of 10 Loss 0.1515
	Batch 3 of 10 Loss 0.1549
	Batch 4 of 10 Loss 0.1922
	Batch 5 of 10 Loss 0.1319
	Batch 6 of 10 Loss 0.1635
	Batch 7 of 10 Loss 0.2243
	Batch 8 of 10 Loss 0.1453
	Batch 9 of 10 Loss 0.1717
	Batch 10 of 10 Loss 0.1335

[codecov]

Codecov Report

Merging #3565 into crypten will decrease coverage by 0.12%.
The diff coverage is 93.23%.

@@             Coverage Diff             @@
##           crypten    #3565      +/-   ##
===========================================
- Coverage    94.79%   94.67%   -0.13%     
===========================================
  Files          159      169      +10     
  Lines        17189    17863     +674     
===========================================
+ Hits         16294    16911     +617     
- Misses         895      952      +57     

Impacted Files	Coverage Δ
syft/execution/placeholder.py	94.97% <ø> (-0.19%)	⬇️
test/execution/test_placeholder.py	100.00% <ø> (ø)
syft/frameworks/crypten/worker_support.py	43.63% <43.63%> (ø)
syft/messaging/message.py	93.65% <92.30%> (-0.20%)	⬇️
syft/frameworks/crypten/utils.py	92.45% <92.45%> (ø)
syft/frameworks/crypten/hook/hook.py	95.37% <95.37%> (ø)
syft/frameworks/crypten/jail.py	96.20% <96.20%> (ø)
syft/frameworks/crypten/context.py	97.10% <97.10%> (ø)
syft/dependency_check.py	88.88% <100.00%> (+1.38%)	⬆️
syft/execution/plan.py	93.81% <100.00%> (-1.59%)	⬇️
... and 24 more