Dottorato in INGEGNERIA INFORMATICA

Titolo della tesi: Taming Complex Bugs in Secure Systems

Modern systems rely on several layers of abstraction to implement their expected functionality. Securing complex systems is extremely difficult, as any abstraction layer can be a possible target. While defense-in-depth techniques help, attackers resort to chaining multiple exploits to break the different protection layers. Thus, malicious actors increase their attack complexity to exploit even the most secure systems. Vulnerabilities may hide in each layer of a modern system. Software applications are often the initial attack target, as they are usually exposed to the external world. Once a malicious actor can exploit a software application, it usually moves to increase their privilege, targetting other running applications or the operating system to exfiltrate data or achieve persistence. From a compromised operating system, an attacker may even target hypervisors and CPU enclaves aimed at protection against privileged attackers. Several techniques attempt to slow down attackers, make bugs hard to find, or, better, find and fix bugs in the first place. Obfuscation techniques complicate the software under analysis, slowing down program understanding from external actors. Automatic software testing aims to find bugs during development to avoid possible vulnerabilities in production systems. Enclaves protect software from a compromised operating system but require CPUs to support them correctly. However, given enough time, attackers can often break each protection layer by combining several complex vulnerabilities to take control of an entire secure system. Thus, a comprehensive approach needs to tackle vulnerabilities across several system layers to increase the attack complexity required to break modern systems. This thesis examines and enhances the security of multiple layers of modern systems. We design, implement and evaluate techniques to find and mitigate complex vulnerabilities across most of the layers of secure systems. Throughout the thesis, we build solutions by sequentially breaking assumptions on attacker capabilities. To slow down attackers, we propose techniques to obfuscate software against reverse engineering efficiently. To secure software and operating systems, we improve existing dynamic testing techniques to find vulnerabilities at a scale and introduce new strategies to find novel bug patterns. We design compilation frameworks to protect software from subtle bugs, from type confusion to CPU side-channel bugs. To deepen the understanding of the threats of privileged attackers, we systematize hardware bug classes, comparing architectural and microarchitectural vulnerabilities to software ones. In doing so, we discover the first architectural CPU bug that leaks confidential data without side channels. Finally, we propose the first CPU framework to inspect and customize modern CPU microcode, offering an unprecedented view of the inner working of current systems. As a byproduct of our research, we find, report, and mitigate over one hundred vulnerabilities across most modern system layers, including application software, operating systems, and CPUs. All the prototypes proposed in this thesis are open-sourced to foster future security research.