How End-to-End Encryption Actually Works in Remote Desktop

中文版本： 《远程桌面的端到端加密原理》

"End-to-end encrypted" is one of the most misused phrases in security marketing. This post explains what it actually means in remote desktop, how it's implemented under the hood, and how to tell real E2EE from marketing E2EE.

What "End-to-End" Actually Means

The two "ends" are the endpoint devices — your laptop and the remote machine. Encryption is "end-to-end" if only those two endpoints can decrypt the data. Specifically:

The vendor's signaling server can't read it.
The vendor's relay (if traffic falls back to one) can't read it.
The vendor's employees can't read it.
A subpoena to the vendor can't reveal session content (only metadata).

If any of those is false, it's not E2EE. It might still be encrypted in transit (TLS to the relay, TLS from the relay), but that's "transport encryption," not E2EE.

How It's Implemented

Modern P2P remote desktop tools typically use the WebRTC stack:

Application data
   ↓
SRTP (Secure Real-time Transport Protocol)
   ↓
DTLS (Datagram TLS — handshake & key exchange)
   ↓
UDP (or TCP fallback)

The DTLS handshake happens directly between the two endpoints. The shared secret derived during that handshake is used to encrypt SRTP frames. Neither the signaling server nor any relay sees the DTLS handshake's secrets — they're exchanged between the endpoints, with the signaling layer relaying only their public components (like in a Diffie-Hellman exchange).

This means:

If a TURN relay is used, it sees encrypted SRTP packets, not plaintext.
If the signaling server logs everything, it logs the negotiation, not the data.

The "End-to-End" Tests

To verify a vendor's E2EE claim, ask these questions:

1. Can the vendor decrypt a session even if they want to?

If yes, it's not E2EE. They have the keys somewhere.

2. Are session keys derived between endpoints, or issued by the vendor?

E2EE requires endpoint-to-endpoint key exchange. If the vendor's server hands out keys, it's not E2EE.

3. What happens during TURN relay fallback?

Many "E2EE" tools quietly downgrade to "encrypted to the relay" when P2P fails. Real E2EE preserves endpoint-to-endpoint encryption even through a TURN relay (the relay sees opaque bytes).

4. Is there session recording? If so, who has the recording keys?

Recording can be E2EE-compatible (encrypted with a key only the recording owner has) or not (encrypted with the vendor's key, decryptable by the vendor).

5. Is the implementation auditable?

Open source, third-party audited, or both. Closed-source "trust us" isn't verifiable.

Common Marketing Tricks

How vendors sound E2EE without being it:

"AES-256 encryption." Says nothing about who has the keys.
"TLS 1.2 in transit." That's transport encryption, not E2EE.
"Zero-trust architecture." A buzzword unrelated to key management.
"Encrypted at rest and in transit." Standard practice, doesn't imply E2EE.
"Military-grade encryption." Meaningless. AES is AES.

What E2EE-claiming products should say:

"DTLS-SRTP between endpoints" (or equivalent specifics).
"Vendor cannot decrypt sessions."
"TURN relays see only encrypted traffic."
"Session keys never leave endpoint devices."

Why It Matters

Compliance: HIPAA, GDPR, and SOC 2 are easier to defend with E2EE. The vendor is no longer a "could-see-the-data" risk.

Threat model: a vendor breach (employee compromise, infrastructure hack, government request) doesn't leak your session content with E2EE.

Predictability: your trust boundary is your own devices, not "everyone the vendor trusts and every law in the vendor's jurisdiction."

What About Session Recording?

This is the trickiest case. To record an E2EE session, you have to:

Have one endpoint produce the recording (and keep the keys to it).
Or use a TPM / key-escrow scheme that the recording owner controls.

If the vendor produces and stores recordings on their servers in plaintext, it's no longer E2EE for those sessions. This is a common compliance gotcha.