Protocol encryption (PE), message stream encryption (MSE) or protocol header encrypt (PHE) are related features of some peer-to-peer file-sharing clients, including BitTorrent clients. They attempt to enhance privacy and confidentiality. In addition, they attempt to make traffic harder to identify by third parties including Internet service providers (ISPs). However, encryption will not protect one from DMCA notices from sharing illegal content, as one is still uploading material and the monitoring firms can merely connect to the swarm.

MSE/PE is implemented in BitComet, BitTornado, Deluge, Flashget, KTorrent, libtorrent (used by various BitTorrent clients, including qBittorrent), Mainline, μTorrent, qBittorrent, rTorrent, Transmission, Tixati and Vuze. PHE was implemented in old versions of BitComet. Similar protocol obfuscation is supported in up-to-date versions of some other (non-BitTorrent) systems including eMule.

As of January 2005, BitTorrent traffic made up more than a third of total residential internet traffic, although this dropped to less than 20% as of 2009. Some ISPs deal with this traffic by increasing their capacity whilst others use specialised systems to slow peer-to-peer traffic to cut costs. Obfuscation and encryption make traffic harder to detect and therefore harder to throttle. These systems were designed initially to provide anonymity or confidentiality, but became required in countries where ISPs were granted the power to throttle BitTorrent users and even ban those they believed were guilty of illegal file sharing.

Protocol header encryption (PHE) was conceived by RnySmile and first implemented in BitComet version 0.60 on 8 September 2005. Some software like IPP2P claims BitComet traffic is detectable even with PHE. PHE is detectable because only part of the stream is encrypted. Since there are no open specifications to this protocol implementation, the only possibility to support it in other clients would have been via reverse engineering.

In late January 2006 the developers of Vuze (then known as Azureus) decided to design and simultaneously implement a new, open protocol obfuscation method, called message stream encryption (MSE). It was included in Azureus CVS snapshot 2307-B29 on 19 January 2006.

This first draft was heavily criticized since it lacked several key features. After negotiations between different BitTorrent developers, a new proposal was written and then implemented into the Azureus and μTorrent betas within days. In μTorrent, the new protocol was called protocol encryption (PE).

The BitComet PHE method used in versions 0.60 to 0.62 is neither published, nor is it compatible with MSE/PE.

MSE/PE uses key exchange combined with the infohash of the torrent to establish an RC4 encryption key. The key exchange helps to minimize the risk of passive listeners, and the infohash helps avoid man-in-the-middle attacks. RC4 is chosen for its speed. The first kibibyte (1024 bytes) of the output is discarded to prevent the Fluhrer, Mantin and Shamir attack.