The Emperor’s New Clothes: Why Compulsory CBTs and Phishing Tests Keep Failing
What looks like assurance is often theatre. And that costs more than time — it also increases risk.
Over recent years I have irritated people immensely by insisting that most phishing training – and indeed most compulsory computer-based training (CBT) modules – are largely ineffective in reducing incidents, and are therefore a waste of time and resources.
Finally we have the information we need to challenge this, and find a better path that may actually reduce the frequency and impact of cyber incidents.
Compulsory training is often highlighted as good practice, but does it really have any substance?
The failure of compulsory training is a problem I know from personal experience.
When I rolled out a cyber CBT to 40,000 employees with just 8 questions to answer, one employee took 73 times to pass and one 86 times, such was their desire to comply whilst avoiding reading the material. The message was clear: you can force people to tick boxes, but you can’t force people to learn.
At another company, I asked a room of 100 employees who had just done training how many of them knew where to report a cyber incident. Nearly everyone raised their hand. But when I asked who was sure, all hands went down apart from two. Allowing for development and delivery costs, and assuming each employee spent half an hour on the CBT, each of those hands cost over $7,000.
Phishing testing performance had similar results, with failure rates depending largely on how hard the phish was. There was no real improvement over time, and the same proportion of people continued to click on similar retests.
There is of course a role for these on occasion. Onboarding of employees might be an example, when people are highly receptive and not distracted by day-to-day pressures. However it has been clear for some time that most time and money spent on these goes down a bottomless hole.
On the other hand, when we visited other companies and learned from their ideas, we found our own creative and fun ways to engage people. At one point our awareness campaign had a 92% engagement rate, versus a 4% corporate average. That campaign cost less than $1 per person for the same result.
What I didn’t have, however, was clear empirical evidence. Now we do.
In a newly released peer-reviewed study, Ho et al. (UC San Diego, 2025) studied 19,500 employees and found this training to be largely futile. Their peer-reviewed research, Understanding the Efficacy of Phishing Training in Practice, showed that phishing click rates did not improve over time and varied only by phish difficulty. Many participants spent less than a minute on the training, and a third didn’t open it at all.
They are not the only researchers to reach this conclusion - Rozema & Davis (Purdue University, 2025) rapidly followed up, reaching very similar conclusions.
Some defenders of CBTs argue they serve a purpose — especially during onboarding, when staff are more receptive. Others point to regulatory obligations or legal defensibility: it’s easier to prove a policy was taught than a behaviour was learned. Even flawed training, they argue, is better than none.
But these arguments don’t hold up to scrutiny. If the goal is learning, not just liability shielding, the outcomes matter. And here the outcomes are dire: low retention, high cost, and little evidence of impact.
As organisations need to show compliance, they then improvise to counter employee tendencies to avoid training they know is largely futile. These attempts to drive compliance are generally designed for force, rather than facilitate, completion. Companies randomise quiz questions, move answer boxes around the screen, and lock the quiz until video content has been finished. They insert trick questions, monitor completion times, and punish people for finishing too quickly. Everyone knows it’s theatre, but no one wants to break the spell.
If everyone knows it doesn’t work, why does nobody say so?
Occasionally people do raise this. But quite simply, nobody has an incentive to address the problem and most have an incentive to let the status quo continue. For example:
If you are an employee, telling your boss or HR that you are not doing the compulsory training properly will get you into trouble, and probably just result in disciplinary action or being reassigned the training again.
If you are a line manager, complaining about the cost or time spent on corporate mandated processes or activity is usually ineffectual, so managers won’t waste limited political capital on a lost cause.
If you are in Risk or Compliance, this can demonstrate everyone has had the information and the opportunity to learn what is expected, even if you know it doesn’t mean they’ve listened. You at least can use the fact that they should have understood to hold employees and managers accountable.
If you are in Legal and you need to show the organisation has passed on its obligations to employees, this allows you to show that the company exercised your responsibilities in a way that reaches everyone, and therefore to defend against many claims. Showing the training didn’t work is much harder than showing it was done.
If you are on a Board, or maybe a customer, CBTs and similar exercises provide simple and objective metrics to hold executive management to account and allow management to demonstrate to you that controls are in place. If it looks effective and management think it is the right thing to be doing, can you really invest precious time in unpicking this?
Why do we need to think differently?
There are two good reasons to care about this.
The first issue is cost. To work out how much your organisation is spending on CBTs and similar training, this is the basic calculation:
Total cost of CBT = (Average fully loaded cost per employee x time spent x number of employees) + (cost of technology + costs of licensing + cost of development + cost of deployment + cost of monitoring & reporting).For example, if your CBT costs £10,000 to license and run, and you have 1,000 employees who cost £50 per hour and each spends an hour on the CBT, your cost is £70,000 - not £10,000. Yet often only the software cost is measured, because that’s the only bit that’s visible in a budget.
If you then do 20 hours of CBTs per employee per year across all disciplines (which is quite normal when you consider HR, health and safety, and similar topics), you’re actually spending £1 million of staff time on this stuff. Do you get £1m of value? I doubt it.
At one point a function I ran was spending more than £6m a year of company time requiring everyone to do a compulsory CBT. We cut it from 60 minutes to 12 minutes, because we couldn’t point to a single benefit (other than avoiding a conversation with a regulator). Yet cutting that cost didn’t change my departmental budget, and I don’t think the company really noticed because nobody else was measuring this. It’s important to make these costs visible to quantify the impact of a control.
The second issue is the negative impact.
CBTs irritate, annoy, and reduce employee morale. They take hours to build, maintain and deliver. Phishing tests require IT configuration to bypass the very controls you ought to be testing. CBTs require tech or management solutions so they can be accessed by everyone, even if they are on the shop floor or outdoors. The constant need for completion monitoring also teaches employees that what they will really be measured on is compliance with corporate diktats, rather than trying to make the best possible decisions they can. CBTs can actually reduce performance and increase risk: they disenfranchise, disincentivise, and disconnect.
It’s a big burden to overcome before the claimed risk reduction outweighs the costs, and yet as UC San Diego found, that claimed risk reduction is often not forthcoming at all.
Ineffective controls are not harmless. Bad controls that look like good ones can be dangerous: metrics plus mis-stated impact can result in false assurance. That false assurance is then what we pass to boards, markets and regulators. No wonder it causes trouble later when the behaviours we have tried to force out by compulsion still occur.
Is there a better way?
The good news is that it does not have to be this way. It is possible to reduce costs, improve efficiency, build positive staff engagement, and reduce risk from undesirable behaviours. That doesn’t have to come at the price of compliance, but it does mean understanding that some boxes are just not worth ticking. Instead, the focus should be on performance and risk.
Firstly, don’t fight employees. Systems and processes should be designed to work with employees, not against them. If you don’t want employees to click links in emails, don’t send them links in emails. Instead of telling employees off for clicking, use technology to have new links open in a secure sandbox. If you want to educate, do it with small prompts and nudges in line with work activity: did you really want to send this sensitive attachment to your personal email? Now you ask, maybe not.
Secondly, match incentives and outcomes. If your employees’ incentives don’t match the desired outcomes, you will always get the wrong result. Don’t tell employees off for not doing a CBT, as that just drives fear and a tick-box culture. Instead, highlight and reward positive behaviour. For example, instead of monitoring compliance with a phishing test, monitor the rate at which real phish are reported and run a leaderboard for the best departments.
Finally, help employees have fun. The most effective awareness campaign I ran was called Harry the Hacker. Harry was a cartoon character and a shameless stereotype of a hacker, who got into all sorts of trouble including involvement with organised crime groups. A team member came up with the concept and we hired an ex-Marvel cartoonist to bring him to life. The hero was never the CISO or the CEO: it was always an employee who did the right thing. It was low tech, with a simple monthly email. He was such a hit that one day a member of staff dressed up as Harry, and sent us a photo of himself dressed as Harry with his computer open on our fake malicious website. Alongside this we cut the cyber CBT from 1 hour to 12 minutes. The results were excellent.
Eventually this was shut down by Corporate Communications who first relegated Harry from an employee email to a dusty corner of the intranet, then wanted us to make everything purple. Our engagement rate plummeted: sometimes you can’t win everything. However, my experience was that trying to engage people with the message in creative ways was more than worth the time and money. It cost more to develop than a standard CBT, but saved employees time and stress, whilst saving the company a fortune and reducing risk.
It’s true there will be a few that don’t engage - but they are also the few who would do a CBT 86 times to avoid reading the content. Sometimes we have to accept that there will always be some human risk, and whilst technology controls can help, having a CBT completion stat that denies the residual risk would hardly be beneficial.
Most employees want to do the right thing. All we need to do is help them.
What then should you do with those CBT and phishing stats?
Instead of punishing the employee, use them to measure the performance of the functions that set them. Hold the commissioning team to account for the business impact. This is where cyber risk quantification techniques can be valuable, but the simple cost calculation above will usually suffice to make the point clearly.
Report the real costs of the delivery, and ask them to justify the investment with reference to real business impact. For example, by measuring voluntary engagement rates, or frequency of desired behaviours vs undesired. If the ROI on the activity is too low, treat it as you would any other under-performing activity. If your marketing didn’t work, how long would you keep doing the same thing? Not long.
The emperor’s new clothes?
In the cautionary folk tale The Emperor’s New Clothes, the emperor and his courtiers are told by charlatan weavers that the Emperor’s clothes are visible unless you are incompetent or stupid. As a result nobody says anything for fear of looking stupid, and so the emperor walks naked through the streets, to general public shock and humiliation. It is a warning not against following, but against following blindly.
We’ve all been told that these tools work, and of course relying on them does not indicate incompetence or stupidity. At one point, this was best practice. I also am not advocating dropping them without careful consideration and a clear plan.
But if we now know they don’t work well, yet we continue to do them?
One day we might find ourselves wearing that security incident a little too visibly for comfort.
There are few things more harmful to an organisation than having maintained that everything is fine, only for it to turn out after an incident that it wasn’t, and that management should have known this all along.
The knowledge is now there to show that whilst compulsory CBTs and phishing exercises are often effective at ticking boxes, they are rarely good at reducing risk - and it’s clear that they cost a fortune whilst making organisations worse places to work.
Yet this is all avoidable, if we try a better approach.
The risk isn’t just in what these tools fail to teach — it’s in what we’ve convinced ourselves they have already taught.
Let’s do better.
___
About the author
Matt Palmer is an award winning cyber security leader. He currently runs national cyber defence for a small island state. He can be found on linkedin or on bluesky.
Original article
First published on 5/10/2025 at https://incidentally.mattpalmer.net/p/the-emperors-new-clothes-why-compulsory
References
Ho, G., Mirian, A., Luo, E., Tong, K., Lee, E., Liu, L., Longhurst, C. A., Dameff, C., Savage, S. and Voelker, G. (2025). Understanding the Efficacy of Phishing Training in Practice. In Proceedings of IEEE Symposium on Security and Privacy. Available at: https://www.computer.org/csdl/proceedings-article/sp/2025/223600a076/21B7RjYyG9q
Rozema, A.T. and Davis, J.C. (2025) Anti-Phishing Training (Still) Does Not Work: A Large-Scale Reproduction of Phishing Training Inefficacy Grounded in the NIST Phish Scale. arXiv preprint. Available at: https://arxiv.org/abs/2506.19899
