Simulating AI-Driven Social Engineering Attacks in Ethical Hacking Using Microsoft 365 Defender

Abstract

Phishing remains one of the most prevalent and effective social engineering attacks, primarily targeting human psychology rather than technical loopholes [1]. The recent development of artificial intelligence (AI), especially with advanced language models like OpenAi’s GPT-4, has provided cybercriminals as well as cybersecurity experts with powerful tools to create exceptionally realistic phishing messages [2]. The present research presents a comparative evaluation of AI generated phishing emails compared to conventional phishing email simulations in the context of an ethical hacking exercise, using Microsoft 365 Defender for Office 365 [3]. The customized and dynamic phishing emails were prepared using GPT-4, while the traditional static templates used were representative of common attack tactics. The experimental setup entailed a pilot group of 300 participants and measured several metrics, such as email opening rate, click-through rate (CTR), credential submission, realism perception, and improvement in awareness. The results show that AI-generated phishing emails outperformed conventional attacks across all the criteria measured, with a 48% CTR and a 34% increase in post-training awareness. The results highlight the increased realism and potency of AI-based phishing simulations [4], while also emphasizing improved, experiential security training in organizations. Future research projects will look to increase the number of participants, include statistical confirmation, and explore other modalities like voice and SMS phishing (vishing and smishing).