Kies de Nederlandse taal
Course module: 202100073
Empirical Security Analysis and Engineering
Course info
Course module202100073
Credits (ECTS)5
Course typeCourse
Language of instructionEnglish
Contact persondr. R. Holz
Lecturer A. Continella
dr. R. Holz
Contactperson for the course
dr. R. Holz
Academic year2022
Starting block
Application procedureYou apply via OSIRIS Student
Registration using OSIRISYes
After completion of the course, the students will be able to:
  • Understand and explain relevant methods (measurement and analysis) to determine the security of an Internet or mobile technology empirically and at scale
  • Understand recent literature on empirical security analysis and apply the findings in the context of similar systems and technologies
  • Analyze the security and principal building blocks of an Internet technology using appropriate tools for the task
  • Compare, evaluate, and apply the principles that the security community learned, and continues to learn, in empirical studies to the design and engineering process of a system
  • Create and design a measurement framework to empirically determine the deployment and use of a security system or technology on the Internet
Dan Geer once famously said: “Any security technology whose effectiveness can’t be empirically determined is indistinguishable from blind luck.” Or indeed, we may add, from placebo. The foundation of sound engineering is a deep understanding of the problem space, the technological state of the art, and the human element in both. In the past two decades, the security community has learned many important and sometimes painful lessons about what it means to design secure systems. Very frequently, these lessons were the result of large-scale study of a particular technology involving empirical methods. On too many occasions, the results that the analysts obtained proved previous assumptions wrong, and sometimes dangerously so. Successful security engineers need to know how they can approach and solve a new security problem by using empirical methods that will yield reliable results. This course will teach students the path from understanding a problem via measurement and analysis to deriving a successful design that achieves the appropriate level of security. 

This course will present a selection of the most important lessons that the security community learned through the application of empirical methods of measurement and analysis. We introduce results from recent research and from case studies of practice to bring students the skills to assess and improve the security of deployed systems. A particular focus is on data-driven approaches to collect operational data about a system's security. We explore deployment issues at local and global scale and also take human factors explicitly into account. Examples are network security, Web security, mobile security and privacy, and the application of machine learning in security. As a result, students will learn to put building blocks of security together in a sound way, to arrive at engineering solutions that are empirically verifiable, functional, and secure against realistic threats.

Closed book written exam, 50%. Two assignments, each 25%. Exercises in practicals and tutorials require sign-off to document sufficient invested effort (80% of exercises must have sign-off).

  • Introduction to measurement of security: techniques for data collection and analysis methods
  • Each of the following topics is covered in four steps: (1) introduction to the problem, (2) techniques and examples of real-world empirical analysis, (3) key results, (4) lessons learned for future engineering.
    • Real-world cryptography: when “secure” algorithms break in practice and the problems of the deployment chain
    • Usability in security: the importance of user-focused design
    • Network security: TLS and X.509 as an example of an Internet-scale problem
    • Transparency mechanisms: Certificate Transparency and the applications of transparency for public auditing at Internet scale
    • Machine Learning in security: the promise and the pitfalls
    • Web security
    • Security in the mobile ecosystem: the Android use case
    • Privacy in the mobile ecosystem: identifying and analyzing data flows and data leaks
Assumed previous knowledge
Security and Cryptography (201500027).

Students need to have previous experience working with Python and Java and be willing to become familiar with further programming and scripting languages as required.

Students may find the course Internet Measurement (202001579) to offer helpful knowledge concerning tools and an introduction to empirical research.

Students need to have taken a course in computer networks or hold equivalent knowledge about the Internet protocol stack.
Participating study
Master Internet Science and Technology
Participating study
Master Computer Science
Required materials
Provided dossier of course texts: academic publications and industry whitepapers
Chapters 1-3 in Security and Usability, L. F. Cranor, S. Garfinkel, O’Reilly Media, 2005. 978-0596008277
On Android (reverse) engineering: Chapters 1-5 in Android hacker’s handbook, J. J. Drake, Z. Lanier, C. Mulliner, P. O. Fora, S. A. Ridley, G. Wichersky, Wiley, 2014. 978-1118608647
Recommended materials
Security Engineering, R. Anderson, 3rd ed, Wiley, 2020. 978-1119642787
Collection of engineering lessons: Engineering Security, P. Gutmann, 2014. Available free online, URL on demand
Instructional modes
Presence dutyYes



Presence dutyYes

Self study without assistance

Presence dutyYes

Exam, Assignments, Practicals

Kies de Nederlandse taal