Secure and Private Machine Learning

Subhankar Mishra

School of Computer Sciences
NISER

Big Data Era

• A lot fo data is collected.
  • By companies like Amazon, Facebook, Apple, Google etc.
  • By government
• Data collected is highly personal and sensitive
  • Browsing history, purchase history, speech, geolocation, health etc.

Risks of exposure?

• Credentials (credit cards, passwords)
  • steal personal property
• Identification (name, biometric data)
  • Identity Theft
• information about you (medical status)
  • discrimination

Privacy and Utility

• Increased privacy affects utility
• Take measures not to remove utility
• Find better trade-offs between privacy and utility

Privacy vs Security

• Privacy - How is your data handled? Your right wrt your personal information.
• Security - Protection against unauthorized access

Examples

• Your bank data is shared with an ad agency by the bank. - Privacy/Security breach?
• Entire bank data is leaked on web - Privacy/Security breach?

Example

If this data is leaked, assume it will cause havoc :)
We want to know how many people had chocolate?

What do we do?

Strategy 1

$y_i = \begin{cases} x_i & \text{with probability 1}\\ 1 - x_i & \text{with probability 0} \end{cases}$

Perfectly accurate, no privacy.

What do we do?

Strategy 2

$y_i = \begin{cases} x_i & \text{with probability 1/2}\\ 1 - x_i & \text{with probability 1/2} \end{cases}$

Perfectly private, no accuracy.

What do we do?

Strategy 3

$y_i = \begin{cases} x_i & \text{with probability 1/2 + } \gamma\\ 1 - x_i & \text{with probability 1/2 - } \gamma \end{cases}$

$\gamma \in [0,1/2]$

Also called Randomized Response

What do we do?

Strategy 3 - Randomized Response

Secure each row

• Flip a coin
• If heads
  • Flip a second coin
    • Yes if heads
    • No if tails
• If tails, dont change the value

Example

Now, data is 50% real and 50% random. We do not know which row is real or random.

This gives the individuals "plausible deniability".

Analysis?

• We want to know how many people had chocolate?
• Assume p to be ground truth of the fraction of people who had chocolate.
• number of yes answers =

Differential Privacy

Formal definition

A randomized computation M satisfies $\epsilon$-differential privacy if for any adjacent data sets $x$ and $x'$, and any subset C of possible outcomes Range(M)

$Pr[M(x) \in C] \leq exp(\epsilon) \times Pr[M(x') \in C]$

• As we remove rows, the answer distribution remains same with a factor of $e^\epsilon$
• Smaller the $\epsilon$, better the utility. But more data is also leaked.

Dwork, Cynthia, and Aaron Roth. "The algorithmic foundations of differential privacy." Foundations and Trends in Theoretical Computer Science 9.3-4 (2014): 211-407.

Idea behind Differential Privacy

• Lets say, there are two databases D1 and D2 are neighbouring if they agree except for a single entry.

• The mechanism behaves nearly identically for D1 and D2, then an attacker can’t tell whether D1 or D2 was used (and hence can’t learn much about the individual).

Alternative to differential privacy?

• Withhold sensitive information?
  • Then we can figure out if sensitive information is present or not!
• k-Anonymity
  • Does not work for high dimensional data
  • Prone to linkage attacks
  • Destroys utility

Narayanan, Arvind, and Vitaly Shmatikov. "How to break anonymity of the netflix prize dataset." arXiv preprint cs/0610105 (2006).

Differential Privacy in the world

• US Census Bureau
  Abowd, John M. "The US Census Bureau adopts differential privacy." Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018.
• Apple
  Cormode, Graham, et al. "Privacy at scale: Local differential privacy in practice." Proceedings of the 2018 International Conference on Management of Data. 2018.
• Microsoft
  Ding, Bolin, Janardhan Kulkarni, and Sergey Yekhanin. "Collecting telemetry data privately." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017

Let's design a ML system with sensitive user data

Sensitive Data

Anything that must be protected against unauthorised access.

Privacy Nightmare

• Build an app that collects user data
• Use data as we want
• But the data is personal.

However, we need the data!

• Perfomance of the model depends on the data

Best source of data?

Devices that we use everyday

So what if the data never leaves your device?

How would we train our ML model then?

Federated Learning (FL)!

• Train a centralised model
• On decentralised data

Federated Learning. https://federated.withgoogle.com. Accessed on 1st November, 2020

FL - Data stays on device

• Data stays on the device
• Design a smart way to train the central model

FL - Bring the training to the device

• Idea is to bring the training to the device rather than
• Bringing data from user to the central server

But Training is resource heavy

• Battery
• Computing Power

Select the devices that are

• on charge
• on Wi-fi
• idle

Steps

Step 1 - Select a subset of eligible devices

Step 2 - Each device receives a training model

Step 3 - Train on the device

• Because the data on the device itself is small.
• Training time is low

Step 4 - Send the trained model back

• Once the model is trained
• Model is sent back to the server
• Not the data

At Center - FedAvg

$f(w) = \sum_{k=1}^K \frac{n_k}{n}F_k(w)$
where

• $F_k(w) = \frac{1}{n_k} \sum_{i \in P_k} f_i(w)$
• $f_i(w) = l(x_i, y_i, w)$ loss of the prediction on example $(x_i, y_i)$
• $K$ clients, with $P_k$ data points on client $k$, with $n_k = |P_k|$

H. B. McMahan, E. Moore, D. Ramage, S. Hampson and B. A. y Arcas, ”Communication-Efficient Learning of Deep Networks from Decentralized Data”, arXiv preprint arXiv:1602.05629 , 2017.

So we good then?

Not exactly

Problem 1: Model inversion attacks!

• There are attachts that reconstructs the data from the model itself
• Goal would be to encrypt the model itself

Chen, Si, Ruoxi Jia, and Guo-Jun Qi. "Improved Techniques for Model Inversion Attack." arXiv preprint arXiv:2010.04092 (2020).

Secure Aggregation

• Model is encrypted
• With a key that the server does not have

Zhang, Yuheng, et al. "The secret revealer: generative model-inversion attacks against deep neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Secure Aggregation

• Every device applies the secure aggregration protocol
• Adds zero-sum masks to scramble the trained model
• Adding all the training results
  • all the masks cancel out.

Bonawitz, Keith, et al. "Practical secure aggregation for privacy-preserving machine learning." Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017.

Problem 2: What if a model memorizes data?

• What if there is rare data?
• Then model memorizes the data

Differential Privacy

• Adding noise to the data (Differential Privacy)
• Even if the model remembers, it remembers noisy data only.

Carlini, Nicholas, et al. "The secret sharer: Evaluating and testing unintended memorization in neural networks." 28th {USENIX} Security Symposium ({USENIX} Security 19). 2019.

https://secml.github.io/

Problem 3: How to test?

• Now that the model is trained, how do we test?
• There is no data at the central server
• All the data resides on the users' devices

Test at the devices !

• We can just test at the users' devices
• Some devices can be training
• Some devices can be testing

Summary

• Learn from all individual
• Without learning about any individual

Work from our lab

1. Sudipta Paul, Poushali Sengupta, and Subhankar Mishra. Flaps: Federated learning and privately scaling.arXivpreprint arXiv:2009.06005, 2020
2. Poushali Sengupta, Sudipta Paul, and Subhankar Mishra. Buds: Balancing utility and differential privacy byshuffling.11th ICCCNT, 2020. (Accepted)
3. Poushali Sengupta, Sudipta Paul, and Subhankar Mishra. Learning with differential privacy.Handbook of Researchon Cyber Crime and Information Privacy, 2021
4. Sudipta Paul and Subhankar Mishra. Ara: Aggregated rappor and analysis for centralized differential privacy. SN Computer Science, 1(1):22, 2020

Thank you