In my article on end-to-end encryption, I mentioned the story of a nameless character who needs privacy guarantees to use an application to its fullest. From his friend Lily, who works at the company, to the company’s CEO Caroline, and an external global threat represented by someone named Mario, all of them take part in the story that explains why privacy is important in applications.
AI works on top of data, and with encrypted data our capacity to provide AI features is reduced. We can do it on the edge, but that comes with application and performance limitations.
We also had a look at FHEs, where data can be processed while encrypted. However, this introduces noise and increases the processing capacity needed at each operation. With complex models such as LLMs, this is currently a no-go.
So what are we left with?
Well, instead of working on encrypted data, we can work inside encrypted, closed-box environments called enclaves. That raises some questions though…
- How do we know the remote box is really an opaque box where no one can look inside?
- How can we be proven that what runs there doesn’t leak our information, and that only we get the output?
- How do we know our data gets there privately?
Opaque box
Generally, this is mostly not possible—but some solutions are appearing where this can be assured to a satisfactory point. What do I mean by “satisfactory point”? Later on that…
Imagine your friend Mario has a really beefy GPU setup at home. He says you can rent it to host a language model there and access it. You get suspicious. He says, “Look man, I’ll set up a VM for you. You change your root access; we can do file system encryption so that I cannot peek at your data.”
You still go “meh”, but with a reason for it. He owns the host, and by peeking at the memory contents he can always see, to some extent, the data you’re processing.
That is why companies like AMD, Intel, and Nvidia launched technology at the level of their processing units to create closed enclaves. Solutions vary in scope, but one approach is to do it at a virtual machine level—so that the host cannot read or tweak guest memory.
But how do you know you are using one of these machines? Each processing chip has an internal key that is used to give you a signature you can validate with certificates published by the chip manufacturer. You do have to trust the chip does what the manufacturer says it does, and that nobody else gets a copy of that key.
Are you willing to trust them?
This is what I meant by “satisfactory point”. You have to trust the chip manufacturer, but you still reduce the threat considerably. I consider it satisfactory for two reasons:
- The infrastructure provider that holds your data (like Mario, or your cloud provider) is different from the entity you trust.
- You are trusting processes more than people. You go from being vulnerable to a single person in a team of sysadmins with root access, to being vulnerable to complex schemes that span across different companies and require more than one person to execute.
Knowing what is inside
So the box is opaque—or at least we trust the chip manufacturer’s word on that. How do we know the software inside is not leaking our data out? By knowing what that software is, and being proved it is running in the enclave. Let’s take the example of a virtual machine.
On boot, the VM collects measurements of system components such as firmware, BIOS, and the software and data running. The specific measurements depend on the chip used and system setup, but the common thing is: (a) they are collected, (b) they are relevant for what you need to have proven—with proof of freshness as well—and (c) they are signed with a chip-specific key you can validate.
To know what is running, one idea is: you have access to a container image configuration (open source if not yours) with a deterministic build process, and you have a way of verifying the build process matches what you get from the measurements.
If from the image you can see no paths to someone else alter the inside after boot and no code exporting the data you input, then you are safe. Assuming you really know how to look for everything, which might not always be the case.
Still, so is the source of client code of apps like Signal and Proton. You trust, because smart people looked at it and gave the OK. Or because it is out in the open and nobody said anything… Questionable, but I guess not the worst heuristic.
Private transport
Well, this has been solved for a long time and we all use it (looking at you, HTTPS). But I wanted to talk about this for two reasons.
One is that it is important to remember it and to close the loop: our data still needs to travel to the box privately.
The other is because we need to be guaranteed our private tunnel really ends in the box and not in some proxy midway. HTTPS certificates are mostly associated with the domain you are accessing: they prove who owns the domain, not where your tunnel ends.
Your options are then to use TLS-based attestation (e.g. RA-TLS) or encrypt your requests with a key you know only the enclave can decrypt. If you followed the lesson on [one of my previous blog posts], you probably know what that looks like.
Performance
Opposite to FHE, some sources point out there is practically no relevant performance degradation when using these systems, both in terms of processing times and model accuracy.
On NVIDIA H100 GPUs, the measured overhead for LLM inference stays below 5% for typical queries and approaches zero for large models and long sequences — the cost is mostly the encrypted CPU–GPU transfers, not the computation (benchmark). CPU-only TEEs pay more, around 20% latency overhead for LLM pipelines (study).
Trust model and limitations
Despite this good performance, keep in mind that in FHE you get a mathematical proof no one peeks at your data; here, you trust the chip provider’s word on it.
For me, this trust model is closer to pure encryption than the alternative of using compute as we always have—but opinions may vary, and you should stay true to yours.
One pain is that there is a lot of knowledge required to prove your data is safe. The attestation process is not trivial and often requires reading code and configurations to make sure nothing can be tampered with or gets out.
Another thing commonly mentioned is the existence of side-channel attacks. Timing, memory-access patterns, and even how encrypted memory changes can be watched by the host or inferred by VMs on the machine to gain insights on what happens inside.
Wrap up and final thoughts
So there it is: a way to have private communication that allows us to use large and complex models such as LLMs (assuming trust in the chip manufacturer). This is great!
But if you have a company and you want to provide private AI services, something is still missing. In one of my previous posts I mentioned a story about a journaling application that aims to be secure. At its helm, CEO Caroline wants to ship AI features—but that’s difficult when data is encrypted. Now there is a solution. For simpler and larger models!
However, clients need to be sure the code on the enclaves is not exfiltrating data. And with the base setup, doing this requires you to share what’s on the server: private code, model weights… all the things that cost time developing should not be given on a silver platter to competitors.
On the next one, I will talk about how this can be tackled, with some juicy code and a Terraform setup to go along. Stay tuned!