Research within the domain of human factors concerning security and privacy is fundamentally dependent on empirical studies engaging directly with participants. These investigations utilize a diverse array of study designs and methodologies for data collection. Integrating multiple study designs and data modalities is essential for deriving reliable conclusions regarding specific phenomena, such as the usability and efficacy of authentication methods for user identification. Nevertheless, not every study design and data type is universally applicable to all research questions. It is imperative, therefore, to select the study design and data collection methods that are most suitable for the specific context of the research inquiry.

Overview of Methodology at SECUSO

In the following, various types of study designs and data will be introduced and illustrated with examples from the research group SECUSO.

• Types of Study Designs
  • Surveys – could be both online or in person. Studies where people are asked to answer questions e.g. about their attitudes.
    • E.g. Cookie disclaimers: Dark patterns and lack of transparency
      Berens, B. M.; Bohlender, M.; Dietmann, H.; Krisam, C.; Kulyk, O.; Volkamer, M. 2024. Computers & Security, 136, Art.-Nr.: 103507.
  • Lab Study – studies where people are invited to a lab and participate in a study in an controlled environment.
    • E.g. PassGlobe: Ein Shoulder-Surfing resistentes Authentifizierungsverfahren für Virtual Reality Head-Mounted Displays
      Länge, T.; Matheis, P.; Düzgün, R.; Mayer, P.; Volkamer, M. 2022. Mensch und Computer 2022 - Workshopband. Ed.: K. Marky, Gesellschaft für Informatik (GI).
  • Field Study – studies conducted in people’s.
    • E.g. Design and Field Evaluation of PassSec: Raising and Sustaining Web Surfer Risk Awareness
      Volkamer, M.; Renaud, K.; Braun, K.; Canova, G.; Reinheimer, B. 2015. 8th International Conference on Trust and Trustworthy Computing, TRUST 2015, Heraklion, Greece, 24 August 2015 through 26 August 2015. Hrsg.: Mauro Conti, Matthias Schunter, Ioannis Askoxylakis, 104–122, Springer.
  • Observation – studies where the researcher does not intervene in any way and simply observes behavior.
    • E.g. Exploring Phishing Threats through QR Codes in Naturalistic Settings
      Sharevski, F.; Mossano, M.; Veit, M. F.; Schiefer, G.; Volkamer, M. 2024. Symposium on Usable Security and Privacy (USEC) 2024.
  • Interviews – studies where people are asked questions with an interviewer and the interviewer can directly asked further questions.
    • E.g. Sharing Information with Web Services – A Mental Model Approach in the Context of Optional Information
      Kulyk, O.; Reinheimer, B. M.; Volkamer, M. 2017. 5th International Conference on Human Aspects of Information Security, Privacy, and Trust (HAS) - Part of HCI International 2017, Vancouver, BC, Canada, July 9-14, 2017, 675–690, Springer.
  • Focus Groups – studies where multiple people at the same time are asked questions or work together on a subject.
    • E.g. Beware of website hackers: Developing an awareness video to warn for website hacking
      Hennig, A.; Schmidt-Enke, L.; Mutter, M.; Mayer, P. 2023, August. 19th Symposium on Usable Privacy and Security. Co-located with USENIX Security ’23 (SOUPS 2023), Anaheim, CA, USA, 6.–8. August 2023
• Types of Data retrieved from studies.
  • Qualitative (studies conducted by SECUSO) – is characterized by their collection through less standardized methods. Initially, these data require transformation—for example, from audio recordings to textual transcriptions or from raw text to coded data—to facilitate analysis. Commonly, this type of data is gathered using research designs such as focus groups or structured interviews.
    • E.g. A user-centred approach to facilitate locating company security policies : Nutzerzentrierter Ansatz zur Vereinfachung des Auffindens von Security Policies
      Aldag, L.; Ballreich, F.; Berens, B.; Volkamer, M. 2023. MuC ’23: Proceedings of Mensch und Computer. Ed.: M. Stolze, 173–185, Association for Computing Machinery (ACM).
  • Quantitative (studies conducted by SECUSO) – is characterized by its collection through more standardized methods. This data can be directly utilized for analysis or converted into other quantitative forms. Typically, such data is acquired using study designs such as surveys or laboratory studies.
    • E.g. An investigation of phishing awareness and education over time: When and how to best remind users
      Reinheimer, B. M.; Aldag, L.; Mayer, P.; Mossano, M.; Düzgün, R.; Lofthouse, B.; Landesberger, T. von; Volkamer, M. 2020. Proceedings of the Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020) : August 10-11, 2020, 259–284, Advanced Computing Systems Association (USENIX)