Programmable Mode-based Memory Isolation

The principle of least privilege has shown to be a valuable building block for secure
computer systems. However, software does not always strictly adhere to the principle of
least privilege: The Heartbleed vulnerability specifically and buffer overflow vulnerabilities
and their impact in general show that the implementation of the principle of least privilege
in current software is still lacking. This can be partly attributed to the protection
mechanisms, such as paging, modern general-purpose processors offer. Paging, for
example, only offers coarse-grained protection on the level of pages and comes with
expensive protection domain switches due to the operating system’s involvement. Novel
protection mechanisms that offer fine-grained protection together with fast protection
domain switches could lead to better adoption of the principle of least privilege. Several
novel protection mechanisms have been developed to alleviate this problem, but while
most of them achieve fine-grained memory protection, they often lack fast protection
domain switch operations. Such fast protection domain switches are invaluable for a
fine-grained and, thus, more secure adoption of the memory protection mechanism.
In this thesis, I develop a protection mechanism that allows users to create a fully
programmable tree-shaped hierarchy of protection domains that allows users to split
up their programs into fine-grained components, making them adhere more closely to
the principle of least privilege. The protection domains are called modes. The modes
are supplied with a segmentation-based approach for the fine-grained encoding of the
mode’s memory permissions and fast, explicit mode switch instructions. Using the
Gem5 simulator, the mode system design is implemented in the RISC-V architecture,
replacing the traditional protection-ring-based security mechanism. The performance and
functionality of this mode-based protection domain are then demonstrated by porting
the xv6 teaching operating system to it and using a set of microbenchmarks as well as
an application benchmark that securely sandboxes two libraries. Those benchmarks not
only show the functionality of the mode system bul also demonstrate a clear performance
advantage of the mode system over a protection approach that solely uses paging.

(Master Defense)