3 Ways Attack-Based Phish Testing is Failing Us

Attack-based phish testing creates fear, anxiety, and doubt in end users.  It does not create awareness, harms IT, and misinforms management.

A recent large-scale IEEE study, conducted from Oct. 2019 through Sept. 2020 of 14773 participants found that attack-based phish testing “can have unexpected side effects that can make employees even more susceptible to phishing“. 

Attack-based phish testing fails End Users, fails IT Departments, and fails Company Leadership seeking cybersecurity resilience.  This article will summarize how and why these failures occur.  It concludes with an alternative phishing test with positive outcomes for all three constituents mentioned above.  Traditional phish testing does not improve your cybersecurity resilience or protection. In fact, it’s likely to cause harm in many cases.

Large-Scale IEEE Study shows Phish Testing is Counterproductive

A research paper titled “Phishing in Organizations: Findings from a Large-Scale and Long-Term Study” was published in the IEEE Security & Privacy journal in 2020. The paper presents the findings of a long-term study of phishing attacks in organizations.

This research paper reports attack-based phishing tests were not an accurate indicator of an organization’s vulnerability to phishing attacks. In fact, it reports these tests could increase employee susceptibility to these attacks.

To combat these testing failures, the study suggests organizations should focus on developing a more comprehensive approach to phishing defense. Defenses should include a combination of technical controls, employee training exercises, and enhancing company culture. Also be sure to factor in evolving tactics used by attackers and the human factors that contribute to successful phishing attacks.

Overall, the study highlights the importance of taking a holistic approach to phishing defenses in organizations. Do not relying on traditional attack-based phish testing. 

Let’s turn our focus now on three specific failure points in attack-based phishing today:  Employee Failures, IT Failures, and Reporting/Metrics Failures.

Fundamental Flaws in Attack-Based Phish Testing

Employee failures

Employees experience phishing attacks as harmful.  They rarely create “awareness” on how to spot and avoid phishing attacks, and lead to punishing outcomes for failure.  At least one study found that employees repeatedly phish tested with attack emails perform more poorly when facing real-world attacks!  Why is this?

Employees are surprised with deceptive emails crafted by IT insiders to play on the fears, insecurities, and emotional state of employees.  This is meant to mirror what hackers “might do” while attacking employees.  However, in case after case, overzealous IT administrators have too often crossed the line and offended employees with their deception-based attack emails.  For example:

Chicago Tribune: Phishing Test goes Terribly Wrong

$650 Bonus for GoDaddy Christmas Gift a Phishing Test 

Not Again! Another Phishing Test goes Awry at West Midlands Railway

These are examples of attack-based phishing tests going horribly wrong.  Employees logged complaints with HR stating that instead of protecting them from harm, IT was causing harm.  Did these campaigns improve the preparedness of employees?  Did they enhance the cybersecurity resilience of the companies that practiced the measures?  It is highly doubtful.  There was very little education happening in these activities.  Instead, these measures lead to more fear, anxiety, and insecurity in the employees tested.

IT Failures

IT Teams are overworked, understaffed, and pressured to perform ever more devious and diabolical attack-based phishing to “show results”.  These attack-emails turn IT teams from protectors, to causes of harm.  This in turn leads to a loss of trust, compassion, and effectiveness in an organizations cybersecurity program and less resilience to attack.  Why is this?

IT teams are asked by leadership to doing everything possible to protect our company and its employees from breach and compromise. Phish testing is one protective measure conducted to “identify employees who are prone to clicking“.  When results don’t show any failures, management asks IT to perform more “difficult” tests.  “Your tests have been too easy!” “Make this a hard phishing test to spot, just like hackers might do to breach our company and its networks!” “I want to see valid results!“

Given these imperatives, it’s not hard to see why IT can take things so far. In fact, all too often they make the news cycle as shown above.  The unfortunate reality here is that employees are not learning anything from these tests, and the consequences for failure rarely address the root cause of failure; a lack of knowledge on how to identify any phishing attack and delete it.

So IT Teams fail to implement meaningful training measures to educates staff outside of “yet another phish training video“.  Employees rarely watch such videos. Nor do they learn from them.  The attack-based phish tests is broken and its many failures are reflected in the news headlines detailing companies breached and data stolen and lost.

Reporting Failures

Testing employees with phishing simulations is meant to achieve two purposes.  Report on an organizations susceptibility to future attacks, and to remind employees to stay vigilant in reviewing their email.  Neither of these metrics was shown in the IEEE study to be tied to effective attack-based phish testing.  Essentially phish testing failure rates underreport employee likelihood of clicking on future attacks.  Secondly, the study showed a negative impact from such tests on future user behaviors.

Poor Metrics that are flawed

Attack-based phish testing yields 5 to 10% failure rates in many organizations.  When Security professionals are asked “What happened to the other 90 to 95% of users?” they are met with blank stares.  The reality is that modern attack-based phish testing does not report on false negative rates of employees who did not see, read, or process the phishing test itself.  Overtaxed inboxes of management are selecting all messages and deleting them after scanning subject lines to cope with the influx of email.  “If its important, they will email me again” has been often quoted.  Consequently, these employees are not being tested and this is impossible to report on with a metric.