Why Messaging Apps Don’t Truly Protect Your Chat Privacy—and How to Do Encryption Right

Encryption in Messaging Apps

What prompted me to write this article was the study Obstacles to the Adoption of Secure Communication Tools (PDF). The authors found that “the vast majority of survey participants do not understand the core concept of end-to-end encryption.” In short, people tend to pick a messenger with their hearts, not their heads.

To begin with, each messenger handles E2EE a bit differently. In Signal it’s almost exemplary. In WhatsApp it’s nominally the same as Signal, except for one crucial detail: if a contact’s identity key changes, WhatsApp doesn’t block you from sending messages. At best you can turn on a largely useless notification (which is off by default). In Viber, end-to-end encryption isn’t enabled by default and only appeared starting with version 6. In Telegram, E2EE is also limited to Secret Chats, and their implementation is rather odd.

The Roskomnadzor–Telegram standoff ended up being great publicity for Telegram. Many everyday users now view Durov’s creation as a real thorn in the side of the security services (or somewhere even more sensitive), who can’t seem to do anything about this “bulletproof,” innovative platform. Telegram fans compare it to Signal and insist Telegram comes out on top.

However, there are no miracles in cryptography—especially in the applied kind. Many mathematically elegant ideas end up hopelessly compromised in implementation, where convenience and control are put ahead of security and privacy (as is the case almost every time).

Early messenger protocols used OTR (Off-the-Record). It employs AES in CTR mode for symmetric encryption, the DH key exchange, and the SHA-1 hash function. AES-CTR provides so‑called “deniable” encryption—the ability to plausibly deny authorship of a message if it’s intercepted. You can always claim that whoever intercepted the traffic altered the ciphertext so it decrypts to a different, same-length message. For example, instead of “go buy bread,” it could turn into “poison the queen”—technically possible, and this malleability is an intentional property of the scheme.

The Off-the-Record (OTR) protocol authenticates the parties and encrypts their conversations. It remains secure as long as participants regularly verify each other’s public key fingerprints and defend against other attack vectors, including social engineering.

OTR’s main drawback is that after you send a new key, you have to wait for the other party to confirm it. If they’re offline, communication is temporarily impossible. One solution was the Double Ratchet (DR) algorithm, developed by Trevor Perrin and Moxie Marlinspike at Open Whisper Systems. Today, DR is used in Signal, WhatsApp, Viber, and many other messengers that offer end-to-end encryption by default or as an optional “secret chat” mode.

End-to-End Encryption

E2EE uses a mix of asymmetric (public/private key) and symmetric cryptography. The high-level idea is simple, but the details are quite intricate. It relies on a large set of interrelated keys, some of which inevitably reside on the server and, in fact, must be uploaded before a conversation starts so messaging can begin at any time. Let’s take a closer look.

You probably know how the scheme begins, since it’s standard for all public‑key systems: a key pair is generated. That’s necessary because single-key (symmetric) ciphers like AES are too cumbersome to use directly in messaging. You’d have to set up a secure channel to exchange the key (for example, meet in person), and then do it again every time you changed it.

It works just like classic PGP: there are two parties—Alice and Bob—each generating their own key pair. They exchange public keys while keeping their corresponding private keys secret. Public keys are sent over an open channel (that’s why they’re public; interception is fine) and serve two purposes: letting others encrypt messages to you and verify your signatures. The private keys, in turn, are used to decrypt and to create (sign) messages.

This kind of cryptosystem works reasonably well for email, since it’s a service for delivering individual encrypted messages of arbitrary length. With email, the parties don’t have to be online at the same time. Messages are stored on the server and fetched on demand after the user successfully authenticates. Decryption happens locally using secret keys that are never transmitted. PGP‑encrypted email is popular, but it’s far from perfect. Why? See the article “Alice and Bob in the Land of PGP.”

Unfortunately, asymmetric encryption on its own isn’t a good fit for messaging apps. These services are built for high‑volume, real‑time chats composed of short messages that need to appear in a strict order, while the other party can go offline at any moment and disrupt the flow of the conversation.

Encrypting lots of short messages with a single key is a bad idea. Over just one day of chatting, you’ll generate hundreds—if not thousands—of them. Many messages produce a tiny, predictable amount of ciphertext (an emoji, a sticker). They also include standard headers, which makes cryptanalysis easier.

One peculiarity of messenger traffic is that the standardized metadata lets an attacker quickly capture a large volume of predictable ciphertext. Most of it will correspond to known plaintext. If it’s all encrypted under a single key, then a successful attack would compromise every message sent so far—and even those the participants will send in the future.

To prevent this, messaging apps implement properties like forward and backward secrecy. In practice, that means having only the current encryption key won’t let you decrypt messages that were sent in the past or ones that will be sent in the future. This is achieved with layered encryption that transitions from asymmetric to symmetric cryptography, plus additional keys with different lifetimes.

Diffie, Hellman—make it a triple!

According to the open documentation at https://core.telegram.org/api/end-to-end, Telegram uses the classic Diffie–Hellman (DH) protocol for authenticated key exchange. DH is a standard way to derive a shared symmetric session key using asymmetric operations, effectively bridging the asymmetric (e.g., RSA) and symmetric (e.g., AES) worlds. This allows two or more participants to start an encrypted conversation by exchanging only public keys over an open channel. The protocol produces session keys—a shared secret, also called a shared ephemeral key—computed from one party’s private key and the other party’s public key. These ephemeral keys are then authenticated with long‑term public keys.

Accordingly, Signal from Open Whisper Systems uses the X3DH (Extended Triple Diffie–Hellman) key agreement protocol X3DH with Curve25519 (Bernstein’s elliptic curve optimized for fast Diffie–Hellman) or X448. X3DH also uses HMAC‑SHA‑256 and AES‑256 as cryptographic primitives.

The Extended Triple Diffie-Hellman (X3DH) protocol establishes a shared secret between two parties that mutually authenticate each other using public keys. It also performs key confirmation immediately after the session is established and before any messages are sent. This greatly reduces the risk of man-in-the-middle attacks, making them highly impractical.

X3DH uses four types of keys, three of which are regularly rotated:

• IK (identity key). A long-term secret key generated once and used as the root for deriving all other keys.
• EK (ephemeral key). A short-lived key used to authenticate the peer without exposing their permanent identity.
• SPk (signed prekey). Essentially a short-lived public key signed with the identity key. In messengers it typically rotates every day to a week; sometimes rotation is triggered after a set number of messages.
• OPK (one-time prekey). A single-use ephemeral key. It’s generated by the sender before session setup and deleted immediately after a successful handshake.

Enigma’s Legacy

The rotation of auxiliary keys in X3DH is handled by the Double Ratchet algorithm. It replaced OTR and introduced the notion of a key chain, or key pool. To account for a peer being offline, an OPK pool is prepared: several one-time ephemeral keys are preloaded to the server in advance and consumed as the conversation progresses. This allows the server to accept encrypted messages and authenticate their sender via a fresh key pair even when the recipient is offline. If the OPK pool is exhausted, the server uses a fallback EK.

The name “Double Ratchet” alludes to the Enigma cipher machine’s ratcheted gearwheels, which prevented backward movement and the reuse of prior states. The digital analogue is that the Double Ratchet derives fresh ephemeral keys to encrypt the next message (or a small batch of messages). These ephemeral keys are guaranteed to differ, never repeat prior values, and are computationally infeasible to predict within any realistic attack window.

The TextSecure protocol, later renamed Signal, is built on X3DH. In its original or slightly modified form, the Signal protocol is used in the Signal app, as well as in WhatsApp, Viber, and others. Developers may brand their protocols differently, but under the hood it’s still X3DH with varying choices of hash functions, PRNGs, and other cryptographic primitives.

The Problem with Group Chats

The design of group chats in messaging apps is examined in detail in the recent paper “More is Less: On the End-to-End Security of Group Chats in Signal, WhatsApp, and Threema” (PDF). Here are its main takeaways.

Our systematic analysis found that communication integrity (understood as the integrity of all messages) and group membership (i.e., letting members control membership changes) lack end-to-end protection. Moreover, we show that backward secrecy—a key security property—does not hold when the Signal protocol is used for group chats.

Note: The cornerstone of end-to-end encryption is authenticated key exchange using a classic or augmented Diffie–Hellman protocol. It only works for two parties deriving a shared secret. Predictably, messengers don’t use DH in group chats, and the message exchange structure there lacks key cryptographic guarantees.

The authors demonstrate what kinds of manipulations an attacker-controlled server can carry out in group chats due to the lack of end-to-end encryption (E2EE). They studied this using Signal and WhatsApp as case studies, but it’s unlikely that other messaging apps have an elegant solution to this problem either.

Telegram’s quirks

Everything about Telegram is shrouded in secrecy. Public information on the MTProto 2.0 protocol is only partial. It hasn’t undergone an independent external audit, and Telegram’s “open-source model” is used in a heavily distorted form and purely for marketing purposes. Below I’ll explain why I think so.

According to the official description, any undelivered messages are (hopefully temporarily) stored on Telegram’s servers, which are distributed worldwide and tied together in a virtual cloud. These servers synchronize with each other to sequence and deliver messages to one or more recipients (in group chats) as soon as they come online. As a result, encryption is applied in two stages: client-to-server and server-to-server. That’s a standard architecture, but what’s unusual is that a direct peer-to-peer connection between clients is never used.

In Telegram, traffic is routed through servers even when you open a Secret Chat—where a P2P connection would be the more logical choice. The obvious conclusion is that without constantly relying on Telegram’s servers, communication in this messaging app doesn’t work at all. Other messengers may use their servers only at the outset—to match the participants’ current IP addresses and establish a direct connection between them. Telegram can’t do that, and it looks suspiciously like MITM by design.

For some reason, most discussions about MTProto 2.0’s security boil down to the idea that the DH algorithm reliably prevents interception. That’s not the case. The Diffie–Hellman algorithm is actually vulnerable to man-in-the-middle attacks. Moreover, in Telegram’s case it is likely further weakened at the pseudo-random number generator (PRNG) level.

The problem is that the Telegram client uses a very unclear notion of entropy. Instead of generating pseudorandom numbers locally and selecting high‑quality primes, the client requests them from the server. Which PRNG the server uses, how good the primes it produces are, and whether the server can selectively send primes with particular properties to specific clients are all unanswered questions. The client only verifies the server-supplied candidate, and even that verification is simplified, because a thorough primality test within a reasonable time frame would be too resource-intensive for a smartphone.

Another common argument for Telegram’s security is its “open source” status. But the server-side code isn’t public, and the client-side code is usually outdated. Telegram’s repositories are updated with a long lag (the stripped-down web version is the only one that’s more or less alive), and they always contain older versions. There isn’t even a way to verify that the current binaries are actually built from the published sources.

That’s why talking about auditing the messenger is basically pointless. By the time experts spend a few months digging through old code, a dozen new versions of Telegram will have shipped, with large chunks rewritten. And to make the entire encryption system vulnerable, a single-byte change is enough—for example, flipping one conditional branch.

Hackathons organized by Durov won’t substitute for a proper audit, because they don’t prove anything. Their challenges set up an artificial scenario where the attacker has only a single encrypted message. In the real world, there’s no such constraint, and there are many other ways to attack a messenger.

A Thousand and One Vulnerabilities

Signal is one of the few messaging apps whose protocol has undergone an external audit (PDF). The audit report is quite extensive, so I’ll quote the key findings in my own translation.

Our analysis shows that the Signal protocol satisfies standard cryptographic assumptions and security properties. We did not find any major weaknesses in its design, which is very encouraging. In real-world use of Signal, however, uncertainties remain. Therefore, it’s impossible to say whether the Signal app always achieves its stated goals.

You need to understand that analyzing the messaging protocol is an important part of an audit, but it’s far from the whole security picture. Any messenger runs in a real, highly vulnerable environment. In practice, it often runs on a not‑so‑up‑to‑date version of Android, alongside a hundred shady apps—some of which are almost certainly abusing permissions or even contain trojan backdoors—and the account itself is tied to a mobile phone number.

The huge gap is that confirmation codes come via SMS. They can be intercepted through a known vulnerability in the SS7 cellular signaling protocol. That way an attacker can get into your chats without knowing any encryption keys and without even trying to break Signal/Proteus/MTProto (or any other “secure” protocol). The messenger’s server will rotate the key and obligingly decrypt the recent conversation (at least any undelivered messages). It’ll even restore your sticker packs. Convenience first, right?

Another gaping hole in the security model is push notifications. Without them, you won’t know you’ve got a message until you manually open the messenger. With them, you effectively turn the push server into a sanctioned man-in-the-middle. For example, to make iMessage notifications work, it sends encryption keys to Apple’s servers. Apple then handles user authentication and, at a minimum, decrypts message headers. Restore your Apple account on another device and you’ll get everything back as it was — right down to your chats and saved Wi‑Fi passwords. Google and Microsoft servers work in almost the same way. Or do you still believe in end-to-end encryption that’s tied to your phone number and your primary smartphone account?

Insecure key management and an overly broad attack surface are problems that affect virtually every messaging app. WhatsApp, Viber, and many others allow chat backups (including cloud backups) and don’t encrypt metadata—and sometimes the fact that a conversation occurred matters more than its contents. Signal does a bit better, but I still don’t consider it an ideal messenger for a number of reasons:

• First, Signal also relies on Google’s push notification service. So on a phone without Google services (e.g., many China-market models that ship without GApps) it simply won’t work.
• Second, Signal uses the proprietary RedPhone server for voice calls.
• Third, like many other messengers, Signal lets you open a parallel session on another device just by scanning a QR code.
• Fourth, at HITBSecConf2017 they discussed a number of conceptual issues in Signal and demonstrated a successful attack against it (PDF: https://conference.hitb.org/hitbsecconf2017ams/materials/D2T1%20-%20Markus%20Vervier%20-%20Hunting%20for%20Vulnerabilities%20in%20Signal.pdf).

XMPP

As you can see, it’s hard to trust third‑party—let alone proprietary—messengers, even if they’re endorsed by Snowden, Assange, or the EFF. That’s why some people run their own messaging stack with open source software and plugins. For simple one‑to‑one chats, an OTR plugin is fine, but it doesn’t support group conversations. Related protocols like mpOTR and GOTR add group chat support.

Either way, for group communication it’s more convenient to use the open XMPP (Extensible Messaging and Presence Protocol), previously known as Jabber. That’s short for “Extensible Messaging and Presence Protocol,” which is pretty descriptive. Because it’s an open standard with widely available source code, you can run your own XMPP server, stay independent, and pay nothing. There are also plenty of ready-made servers and clients to choose from—for example, Pidgin on the desktop and Xabber for Android.

Extensibility means you can transmit not only text but other data types, and also add features and encryption schemes. For example, XMPP makes it easy to send voice messages, video, and files, optionally encrypting them with TLS or PGP. More recently, an XMPP extension protocol called OMEMO was introduced; it uses the same Double Ratchet from Open Whisper Systems as Signal and WhatsApp, but without their other drawbacks.

Conclusions

Modern messengers claim to support end-to-end encryption, but in practice it’s often implemented with odd quirks. On top of that, their code tends to have plenty of other holes—left in either by accident or on purpose. The latter seems more likely, given how much money and professional effort went into building them.

I try to strike a balance between convenience and my usual paranoid streak. I use whatever messengers my contacts prefer, but I never have truly private conversations on them. There are plenty of open-source alternatives for that. Beyond Xabber, Android users should consider Conversations—a free and open-source XMPP client with support for OTR, OMEMO, OpenPGP, and SOCKS5.