docs/ml/advanced_examples/EncryptedPandas.ipynb

Encrypted Data-frames

The following notebook shows how to encrypt Pandas data-frames and run a left join on them using Fully Homomorphic Encryption (FHE) in a client-server setting using Concrete ML. This example is separated into three main sections :

1. Two independent clients load their own csv file using Pandas, encrypt their data and send them to a server
2. The server runs a left join in FHE
3. One of the client receives the encrypted output data-frame and decrypts it

In such a setting, several parties are thus able to merge private databases without ever disclosing any of their sensitive data. Additionally, Concrete ML provides a user-friendly API meant to be as close as possible to Pandas.

In [1]:

import shutil
import time
from pathlib import Path
from tempfile import TemporaryDirectory

import numpy
import pandas

from concrete.ml.pandas import ClientEngine, load_encrypted_dataframe
from concrete.ml.pytest.utils import pandas_dataframe_are_equal

numpy.random.seed(0)

DATA_PATH = Path("data/encrypted_pandas")

# pylint: disable=pointless-statement, consider-using-with

Clients

User 1

On the first user's side, load the private data using Pandas. This example uses the Tips dataset. It was split into two csv files so that:

• all columns are different, except for column "index", representing the initial data-frame's index
• some indexes are common, some others are not

In [2]:

CLIENT_1_DIR = DATA_PATH / "client_1"

df_left = pandas.read_csv(CLIENT_1_DIR / "df_left.csv")

df_left

Out [2]:

	index	total_bill	tip	sex	smoker
0	client_1	12.54	2.50	Male	No
1	client_2	11.17	1.50	Female	No
2	client_3	20.29	2.75	Female	No
3	client_4	14.07	2.50	Male	No
4	client_5	15.69	3.00	Male	Yes
5	client_6	18.29	3.00	Male	No
6	client_7	16.93	3.07	Female	No
7	client_8	24.27	2.03	Male	Yes
8	client_9	8.77	2.00	Male	No

In order to be encrypted, string values first need to be mapped to integers (see section below about get_schema). By default, this mapping is done automatically. However, for example, the column won't be able to be selected when merging encrypted data-frames. This is because such an operator requires the data-frames' string mapping to match, else values will be mixed up.

This is exactly the case here, as the index column only contains string values, thus the mapping must be defined by the application developer. This mapping will then be shared to the second client (see below) in order to make sure both matches. Other non-integer columns do not require any pre-computed mapping if they are not expected to be selected for merging. All mappings are grouped per column as a dictionary, called "schema".

Therefore, let's define our schema:

In [3]:

schema = {"index": {index_value: i + 1 for i, index_value in enumerate(df_left["index"].values)}}

A ClientEngine instance is then initialized, which is used for managing keys (encryption, decryption).

In [4]:

client_1_temp_dir = TemporaryDirectory(dir=str(CLIENT_1_DIR))
client_1_temp_path = Path(client_1_temp_dir.name)

# Define the directory where to store the serialized keys
client_1_keys_path = client_1_temp_path / "keys"

client_1 = ClientEngine(keys_path=client_1_keys_path)

Using the ClientEngine instance, the user is now able to encrypt the Pandas data-frame, building a new EncryptedDataFrame instance. The schema, which includes the string mapping for column index, is provided as well.

In [5]:

df_left_enc = client_1.encrypt_from_pandas(df_left, schema=schema)

EncryptedDataFrame objects are able to handle multiple data-types: integers, floating points and strings. Under the hood, the data needs to be quantized under a certain precision before encryption (more info on why: Key Concepts and Quantization ). More specifically:

• integers: the values are kept as they are but an error is raised if they are not within the range currently allowed
• floating points: the values are quantized under a certain precision, and quantization parameters (scale, zero-point) are sent to the server
• strings: the values are mapped to integers using a dict, which is sent to the server as well

More generally, the quantized values must be within the range currently allowed. This notably means that the number of rows allowed in a data-frame are also limited, as keys on which to merge are expected to be unique.

Once the inputs are quantized and encrypted, the user can print the encrypted data-frame's schema. A schema represents the data-frame's columns as well as their dtype and associated quantization parameters or mappings.

In [6]:

df_left_enc.get_schema()

Out [6]:

	index	total_bill	tip	sex	smoker
dtype	object	float64	float64	object	object
str_to_int	{'client_1': 1, 'client_2': 2, 'client_3': 3, ...	NaN	NaN	{'Male': 1, 'Female': 2}	{'No': 1, 'Yes': 2}
scale	NaN	0.903226	8.917197	NaN	NaN
zero_point	NaN	6.92129	12.375796	NaN	NaN

The encrypted data-frame can be serialized and saved using the save method.

In [7]:

df_left_enc_path = client_1_temp_path / "df_left_enc"
df_left_enc.save(df_left_enc_path)

User 2

The second user's steps are very similar to the first one. It is important to note that both users are expected not to share any of their data-base with each other.

In [8]:

CLIENT_2_DIR = DATA_PATH / "client_2"

df_right = pandas.read_csv(CLIENT_2_DIR / "df_right.csv")

df_right

Out [8]:

	index	day	time	size
0	client_2	Thur	Lunch	2
1	client_5	Sat	Dinner	3
2	client_9	Sun	Dinner	2

Currently, the users need to share the private keys in order to be able to run an encrypted merge. Future works will provide new techniques that would avoid this.

In [9]:

client_2_temp_dir = TemporaryDirectory(dir=str(CLIENT_2_DIR))
client_2_temp_path = Path(client_2_temp_dir.name)

# Define the directory where to store the serialized keys
client_2_keys_path = client_2_temp_path / "keys"

# Copy the first user's keys
shutil.copy2(client_1_keys_path, client_2_keys_path)

client_2 = ClientEngine(keys_path=client_2_keys_path)

Encrypt the second user's data-frame. Here, the same schema used for client 1 is needed in order to make sure that custom mappings are matching.

It is possible to get the encrypted data-frame's representation by simply returning the variable.

In [10]:

df_right_enc = client_2.encrypt_from_pandas(df_right, schema=schema)

df_right_enc

Out [10]:

index	day	time	size
..3a5cea3d89..	..5e1c826a69..	..a79208da47..	..4a46bbeabe..
..c959c81aa2..	..a992e555a0..	..ad73b4157e..	..947b705b07..
..ac944db8d7..	..76d5b0360e..	..c0544afe88..	..fa7f74cea1..

Save the second user's encrypted data-frame

In [11]:

df_right_enc_path = client_2_temp_path / "df_right_enc"
df_right_enc.save(df_right_enc_path)

Server

The server only receives serialized encrypted data-frames. Once it has them, anyone is able to decide which operation to run on which data-frames, but only the parties that encrypted them will be able to decrypt the result.

First, the server can deserialize the data-frames using Concrete ML's load_encrypted_dataframe function.

In [12]:

df_left_enc = load_encrypted_dataframe(df_left_enc_path)
df_right_enc = load_encrypted_dataframe(df_right_enc_path)

The server can now run a left join on the encrypted data-frames' common column "index" using FHE. This step can take several seconds.

In [13]:

start = time.time()
df_joined_enc_server = df_left_enc.merge(df_right_enc, how="left", on="index")
end = time.time() - start

print(f"Total execution time: {end:.2f}s")

Total execution time: 6.84s

The encrypted output data-frame is then serialized.

In [14]:

df_joined_enc_server_path = client_1_temp_path / "df_joined_enc"

df_joined_enc_server.save(df_joined_enc_server_path)

Clients

Both user 1 and 2 are able to decrypt the server's encrypted output data-frame, but it first needs to be deserialized.

In [15]:

df_joined_enc = load_encrypted_dataframe(df_joined_enc_server_path)

The user can now decrypt it and recover the joined data-frame as a Pandas DataFrame object.

In [16]:

df_joined_cml = client_1.decrypt_to_pandas(df_joined_enc)

df_joined_cml

Out [16]:

	index	total_bill	tip	sex	smoker	day	time	size
0	client_1	12.091429	2.509286	Male	No	NaN	NaN	NaN
1	client_2	10.984286	1.500000	Female	No	Thur	Lunch	2.0
2	client_3	19.841429	2.733571	Female	No	NaN	NaN	NaN
3	client_4	14.305714	2.509286	Male	No	NaN	NaN	NaN
4	client_5	15.412857	2.957857	Male	Yes	Sat	Dinner	3.0
5	client_6	18.734286	2.957857	Male	No	NaN	NaN	NaN
6	client_7	16.520000	3.070000	Female	No	NaN	NaN	NaN
7	client_8	24.270000	2.060714	Male	Yes	NaN	NaN	NaN
8	client_9	8.770000	1.948571	Male	No	Sun	Dinner	2.0

Concrete ML vs Pandas comparison

For this demo, expected output from Pandas (in a non-private setting) can be computed and compared to the result above.

In [17]:

df_joined_pandas = pandas.merge(df_left, df_right, how="left", on="index")

df_joined_pandas

Out [17]:

	index	total_bill	tip	sex	smoker	day	time	size
0	client_1	12.54	2.50	Male	No	NaN	NaN	NaN
1	client_2	11.17	1.50	Female	No	Thur	Lunch	2.0
2	client_3	20.29	2.75	Female	No	NaN	NaN	NaN
3	client_4	14.07	2.50	Male	No	NaN	NaN	NaN
4	client_5	15.69	3.00	Male	Yes	Sat	Dinner	3.0
5	client_6	18.29	3.00	Male	No	NaN	NaN	NaN
6	client_7	16.93	3.07	Female	No	NaN	NaN	NaN
7	client_8	24.27	2.03	Male	Yes	NaN	NaN	NaN
8	client_9	8.77	2.00	Male	No	Sun	Dinner	2.0

Slight differences cab be observed between Pandas and Concrete ML with floating points values. This is only due to quantization artifacts, as currently only 4 bits of precision are supported. Still, both data-frames are equal under a small float relative tolerance.

In [18]:

df_are_equal = pandas_dataframe_are_equal(
    df_joined_pandas, df_joined_cml, float_rtol=0.1, equal_nan=True
)

print("Concrete ML data-frame is equal to Pandas data-frame:", df_are_equal, "\n")

Concrete ML data-frame is equal to Pandas data-frame: True 

In [19]:

# Clean the temporary directories and their content
client_1_temp_dir.cleanup()
client_2_temp_dir.cleanup()

Conclusion

Concrete ML provides a way for multiple parties to run Pandas operations on their data-frames without ever disclosing any sensitive data. This is done through a Pandas-like API that enables users to encrypt the data-frames and a server to run the operations in a private and secure manner using Fully Homomorphic Encryption (FHE). The users are then able to decrypt the output and obtain a result similar to what Pandas would have provided in a non-private setting.

Future Work

In the near future, bigger precisions will be allowed, which would make encrypted data-frames able to handle larger integers, floating points with better precisions and more unique strings values, as well as provide more rows. Support for more encrypted operations on data-frames will also be added. While users need to share private keys with the current version of the API, threshold decryption, a multi party key generation protocol, could allow them to compute on joint data without revealing it to each other.