Below is a list of resources I am aware of exploring the intersection of behavioral economics and information security. If you are aware of others, please leave a comment.
Below is a list of resources I am aware of exploring the intersection of behavioral economics and information security. If you are aware of others, please leave a comment.
Snapchat recently disclosed that it was the victim of an increasingly common attack where someone in the HR department is tricked into providing personal details of employees to someone purporting to be the company’s CEO.
In response, the normal calls for “security awareness training!” and “phishing simulations!” is making the rounds. As I have said, I am in favor of security awareness training and phishing simulation exercises, but I am wary of people or organizations that believe this is a security “control”.
When organizations, information security people and management begin viewing awareness training and phishing simulations as a control, incidents like happened at Snapchat are viewed as a control failure. Management may ask “did this employee not take the training, or was he just incompetent?” I understand that your gut reaction may be to think such a reaction would not happen, but let me assure you that it does. And people get fired for falling for a good phish. Maybe not everywhere. Investment in training is often viewed the same as investment in other controls. When the controls fail, management wants to know who is responsible.
If you ask any phishing education company or read any of their reports, you will notice that there are times of day and days of the week where phishing simulations get more clicks than others, with everything else held constant. The reason is that people are human. Despite the best training in the world, factors like stress, impending deadlines, lack of sleep, time awake, hunger, impending vacations and many other factors will increase or decrease the likelihood of someone falling for a phishing email. Phishing awareness training needs to be considered for what it is: a method to reduce the frequency, in aggregate, of employees falling for phishing attacks.
So, I do think that heads of HR departments everywhere should be having a discussion with their employees on this particular attack. But, when a story like Snapchat makes news, we should be thinking about prevention strategies beyond just awareness training. And that is hard because it involves some difficult trade offs that many organizations don’t want to think about. Not thinking about them, however, is keeping our head in the sand.
I recently finished listening to episode 398 of the Risky Business podcast where Patrick interviews Professor Lawrence Gordon. The discussion is great, as all of Patrick’s shows are, but something caught my attention. Prof Gordon describes a model he developed many years ago for determining the right level of IT security investment; something that I am acutely interested in. Professor points out that a key aspect of determining the proper level of investment is the probability of an attack, and he points out that the probability needs to be estimated by the people who know the company in question best: the company’s leadership.
That got me thinking: how do company leaders estimate that probability? I am sure there are as many ways to do it as there are people doing it, however the discussion reminded me of a key topic in Daniel Kahneman’s book “Thinking Fast and Slow” regarding base rates. Base rates are more or less an average quantity measured against a population for a given concept. For example, the probability of dying in a car crash is about 1 in 470. That’s the base rate. If I wanted to estimate my likelihood of dying in a car crash, I should start with the base rate and make adjustments I believe are necessary given unique factors to me, such as that I don’t drive to work every day, I don’t drink while driving and so on. So, maybe I end up with my estimate being 1 in 60o.
If i didn’t use a base rate, how would I estimate my likelihood of dying in a car crash? Maybe I would do something like this:
Probability of Jerry dying in a car crash <
1/(28 years driving x 365 x 2 driving trips per day)
This tells me I have driven about 20,000 times without dying. So, I pin my likelihood of dying in a car crash at less than 1 in 20,000.
But that’s not how it works. The previous 20,000 times I drove don’t have a lot to do with the likelihood of me dying in a car tomorrow, except that I have experience that makes it somewhat less likely I’ll die. This is why considering base rates are key. If something hasn’t happened to me, or happens really rarely, I’ll assign it a low likelihood. But, if you ask me how likely it is for my house to get robbed right after it got robbed, I am going to overstate the likelihood.
This tells me that things like the Verizon DBIR or the VERIS database are very valuable in helping us define our IT security risk by providing a base rate we can tweak.
I would love to know if anyone is doing this. I have to believe this is already a common practice.
I am currently reading Richard Thayler’s new book “Misbehaving: The Making of Behavioral Economics”. I trust I don’t need to explain what the book is about. Early in the book, Thayler describes the work leading up to his thesis, “The Value of Saving a Life”, and points out something most of us can relate to: we value a specific person more than we value the nebulous thought of many unnamed people. Let me give an example: a girl is very sick and needs an expensive treatment that costs $5 million which her family cannot afford and is not covered by insurance. We have seen similar cases, where the family receives a flood of donations to pay for the treatment. Now consider a different situation: the hospital in the same city as the girl needs $5 million to make improvements which will save an average of two lives per year by reducing the risk of certain infections that are common in hospitals. There is no outpouring of support to provide $5 million to the hospital. The person in the first case is specific – an identified life, while we have no idea who the 2 people per year that would be saved are in the second case – statistical lives. Identified lives vs. statistical lives. If we were “rational” in the economic sense of the word, we should be far more willing to contribute money to the hospital’s improvement program since it will save many more people than just the lone sick girl. But we are not rational.
There seems to be a powerful implication for information security in this thought: we have trouble with valuing things that are abstract, like the theft of some unknown amount of our data belonging to people who may not even be customers of ours yet. After a breach, we care very deeply about the data and the victims, and not just because we are in the news, may face lawsuits and other penalties, but because the victims are now “real”. We only move from “statistical” data-subjects to “identified” data-subjects after a breach. Post breach, we generally care more about and invest more in security to avoid a repeat because the impacts are much more real to us.
One of the fundamental tenants of behavioral economics is that we humans often do not act in an economically rational way – this gave rise to calling the species of people who act according to standard economic theory “econs”. It occurs to me that, in the realm of IT security, we would do well to try to behave more like econs. Of course, it helps to understand the ways in which econs and humans think differently.
I’ve been reading “Thinking Fast and Slow” for the 3rd time now… Technically, I am listening to the audio book, and keep picking up new insights.
The most recent insight is related to intuition. To net out the topic, intuition, Kahneman believes, is actually only familiarity. A host of heuristics can influence our perception of how familiar something seems, however we the intuitions of people are often not very good, almost always fairing less well than a basic algorithm. Therefore, we should be wary when we use intuition to guide an important decision. Having said that, some people do develop intuition for some things, but two criteria must be met:
Kahneman recommends asking questions about whether an intuitive judgement is a related to a process and whether the person exhibiting the judgement has the requisite experience to have developed the intuition.
This is an interesting thought in the context of information security.
By the way, if you have not yet read “Thinking Fast and Slow”, I highly recommend it. The audio version is excellent, too, even though it is nearly 20 hours long.
A message came through the Security Metrics mailing list yesterday that got me thinking about our perception of statistics. The post is regarding a paper on the security of an electronic voting system.
I’ll quote the two paragraphs I find most interesting:
To create a completely unhackable system, Smartmatic combined the following ideas: security fragmentation, security layering, encryption, device identity assurance, multi-key combinations and opposing-party auditing. Explaining all of them is beyond the scope of this article.
The important thing is that, when all of these methods are combined, it becomes possible to calculate with mathematical precision the probability of the system being hacked in the available time, because an election usually happens in a few hours or at the most over a few days. (For example, for one of our average customers, the probability was 1 × 10−19. That is a point followed by 19 zeros and then 1). The probability is lower than that of a meteor hitting the earth and wiping us all out in the next few years—approximately 1 × 10−7 (Chemical Industry Education Centre, Risk-Ed n.d.)—hence it seems reasonable to use the term ‘unhackable’, to the chagrin of the purists and to my pleasure.
The claim here appears to be that the number of robust security controls included in the system, all of which have a small chance of being bypassed taken together, along with the limited time that an election runs yields a probability of 1×10^-19 of being hacked, which is effectively a probability of zero.
A brief bit of statistical theory: the process for calculating the probability of two or more events happening at the same time depends on whether the events are independent from each other. Take, for example, winning the lottery. Winning the lottery a second time is in no way related to winning the lottery a first time… You don’t “get better” at winning the lottery. Winning the lottery is an independent event. If the odds of winning a particular lottery are one in a million, or 1/1000000, the probability of winning the lottery twice is 1/1000000 x 1/1000000, which is 1/1000000000000 or 1×10^-12. However, many events are not actually independent from each other. For example, I manage a server and the probability of the server being compromised through a weak password might be 1/1000000. Since I am clever, getting shell on my server does not get you access to my data. To get at my data, you must also compromise the application running on the server through a software vulnerability and the probability of that might also be 1/1000000. Does this mean that the probability of someone stealing my data is 1×10^-12? These events are very likely not independent. The mechanism of dependence may not be readily apparent to us, and so we may be apt to treat them as independent and decide against the cyber insurance policy, given the remarkably low odds. Upon close inspection, there is a nearly endless list of ways in which the two events (getting a shell, then compromising the application) might not be independent, such as:
When we see the probability of something happening stated as being exceedingly low as with 1×10^-19, but then see the event actually happen, we are right to question the fundamental assumptions that went into the calculation.
A practical example of this comes from the book “The Black Swan” in which Taleb points out the Nobel Prize winning Modern Portfolio Theory calculated the odds of the 1987 stock market crash to be 5.51×10^-89.
My experience is that these kinds of calculations happen often in security, even if only mentally. However, we make these calculations without a comprehensive understanding of the relationships between systems, events and risks.
Outside of gambling, be skeptical of such extraordinary statements of low probabilities, particularly for very important decisions.
If your organization is like most, tough problems are addressed by assembling a group of SMEs into a meeting and hashing out a solution. Risk assessments are often performed in the same way: bring “experts” into a room, brain storm on the threats and hash out an agreed-upon set of vulnerability and impacts for each. I will leave the fundamental problems with scoring risks based on vulnerability and impact ratings for another post.
“None of us is as smart as all of us” is a common mantra. Certainly, we should arrive at better conclusions through the collective work of a number of smart people. We aren’t. Many people have heard the phrase “the wisdom of crowds” and implicitly understood that this reinforces the value of the collaborative effort of SMEs. It doesn’t.
The “wisdom of crowds” concept describes the phenomenon where a group of people are each biased in random directions when estimating some quantity. When we average out the estimates of the “crowd”, the resulting average is often very close to the actual quantity. This works with the estimates are given independently of one another. If the “crowd” collaborates or compares ideas when estimating the quantity, this effect isn’t present. People are heavily influenced by each other and the previously present array biases are tamped down, resulting in a estimates that reflect the group consensus and not the actual quantity being analyzed.
The oft cited example is the county fair contest where the crowd writes down his or her guess for the weight of a cow or giant pumpkin on a piece of paper, drops the paper in a box and hopes to have the closest guess to win the Starbucks gift card. Some enterprising people have taken the box of guesses and averaged them out and determined that the average of all guesses is usually very close to the actual weight. If, instead, the fair goers were somehow incentivized to work together so that they only had one guess, and if that guess were within, say 2 pounds of the actual weight, the entire crowd won a prize, it’s nearly a sure thing the crowd would lose every time, absent some form of cheating.
With this in mind, we should consider the wisdom of our risk assessment strategies.
 In the mean time, read Douglas Hubbard’s book: “The Failure of Risk Management”.
Researchers studying human behavior describe a trait, referred to as the availability heuristic, that significantly skews our estimation of the likelihood of certain events based on how easy or hard it is for us to recall an event, rather than how likely the event really is.
It isn’t hard to identify the availability heuristic at work out in the world: shark attacks, terror attacks, plane crashes, kidnappings and mass shootings. All of them are vivid. All of them occupy, to a greater or lesser extend, the news media. The recollection of these events, usually seen through the media, will often cause people to irrationally overestimated certain risks. For instance, the overwhelming majority, approximately 88%, of child kidnappings is perpetrated by a relative or caregiver. However, the raw statistics regarding kidnappings, constant Amber alerts and media stories about horrible kidnapping cases is the source of much consternation for parents. Consternation to the point that police in some jurisdictions are accusing parents who allow kids to play outside unsupervised of child neglect. The gun debate rages on in the U.S., with mass shooting tragedies leading news reports, even though the number of people who kill themselves with a gun significantly outnumbers those murdered with a gun.
The availability heuristic causes us to worry about shark attacks, plane crashes, stranger kidnappings and mass shootings, while we are far more likely to die in car crashes, or from diabetes, or heart disease, or cancer or even of suicide, however the risks from those are generally not prominent in our minds when we think about the most important risks we, and our friends and families, face. Maybe if, at the end of the TV news, the commentators recapped the number of car crash fatalities and heart disease fatalities, we would put better context around these risks, but probably not. As Stalin said: “a single death is a tragedy; a million deaths is a statistic.”
How does this related to information security?
Information security programs are, at their core, intended to mitigate risks to an organization’s systems and data. Most organizations need to be thoughtful in the allocation of their information security budgets and staff: addressing risks in some sort of prioritized order. What, specifically, is different between the ability to assess the likelihood of information security risks as opposed to the “every day” risks described above?
Increasingly, we are bombarded by news of mega breaches and “highly sophisticated” attacks in the media. The availability of these attacks in recollection is certainly going up as a result. However, just like fretting about a shark attack as we cautiously lounge in a beach chair safely away from the water while eating a bag of Doritos, are we focusing on the unlikely Sony-style attack, while our data continues to bleed out through lost or stolen unencrypted drives on a daily basis? In many cases, we do not actually know the specific mechanisms that lead to the major beaches. Regardless, security vendors step in and tailor their “solutions” to help organizations mitigate these attacks.
Given that the use of quantitative risk analyses are still pretty uncommon, the assessment of likelihood of information risks is, tautologically, subjective in most cases. Subjective assessment of risks are almost certainly vulnerable to the same kinds of biases described by the availability heuristic.
The availability heuristic works in both directions, too. Available risks are over-assessed, while other risks that may actually be far more likely but not prominently recalled, are never even considered. Often, the designers of complex IT environments appear to be ignorant of many common attacks and do not account for them in the system design or implementation. They confidently address the risks, as budget permits, that they can easily recall.
Similarly, larger scale organizational risk assessments that do not enumerate the more likely threats will most certainly lead to suboptimal prioritization of investment.
At this point, the above linkage of the availability heuristic to information security is hypothetical- it hasn’t been demonstrated objectively, though I would argue that we see the impacts of it with each new breach announcement.
I can envision some interesting experiments to test this hypothesis: tracking how well an organization’s risk assessments forecast the actual occurrence of incidents; identifying discrepancies between the likelihood of certain threats relative to the occurrence of those threats out in the world and assessing the sources of the discontinuities; determining if risk assessment outcomes are different if participants are primed with different information regarding threats, or if the framing of assessment questions result in different risk assessment outcomes.
A possible mitigation against the availability heuristic in risk assessments, if one is really needed, might be to review sources of objective threat information as part of the risk assessment process. This information may come from threat intelligence feeds, internal incident data and reports such as the Verizon DBIR. We have to be cognizant, though, that many sources of such data are going to be skewed according to the specific objectives of the organization that produced the information. Reading an industry report on security breaches written by the producer of identity management applications will very likely skew toward analyzing incidents that resulted from identity management failures, or at least play up the significance of identity management failures in incidents where multiple failures were in play.
In this series, I am exploring the intersection of information security and behavioral economics. As a long time information security person that recently began studying behavioral economics, I’ve come to realize that much of traditional information security programs are built using standard economic models.
For example, the Simple Model of Rational Crime (SMOC) has implicitly influenced the creation of security policies and conduct guidelines, as well as much of criminal law. Simply put, SMOC takes the traditional economic view of human decisions as it pertains to maximizing utility when in comes to committing crimes: people perform a cost-benefit calculation and decide whether or not to commit the crime.
We have four levers to push and pull on as it pertains to managing employee threat:
Many corporate security programs rely heavily on the first three levers assuming that if people clearly understand the expectations of them, clearly understand the consequences and have some expectation that they’ll be caught, employees will make economically rational choices after weighing the cost-benefit of whatever opportunistic misdeed lays in front of them. It’s hard to consider the possibility that a sane person would choose to risk their well paying job for a few hundred dollars or to cut a corner that saves them a few minutes. Anyone who would do such a thing must, by definition, not be of sound mind and therefore isn’t really good for the company. Right?
But this scenario happens all the time. Our policies and expectations are built on the understanding that people are indeed rational and make rational cost-benefit assessments before taking an action. A growing body of research points out that people are influenced by a great many things, from their mood to project deadlines to how tired they are. We don’t like level number four, because it’s expensive, inconvenient and we shouldn’t have to do it anyhow given the above conditions. But we should reconsider.
Dan Ariely’s book “The Honest Truth About Dishonesty” details many experiments that illustrate how the SMOC model doesn’t represent the actual behavior of people and is well worth a read for anyone responsible for designing security programs or security awareness training.
The take away for this post is that relying on employees “to do the right thing” as an integral part of a security program doesn’t make sense given what we know about the human mind. As mentioned in the previous post, reminders about honesty can help in some cases, but not in all. The integrity of key processes should not rely solely on policy and employment agreements, but rather be designed to prevent, or at least quickly detect, employee misdeeds. Such controls clearly won’t work for all organizations or in all circumstances due to cost constraints, politics, technological limitations and so on, but we need to be clear about what to expect when those controls are absent. Too many organizations are surprised when an employee violates policy, despite the policy being explicit on expectations, explicit on the ramifications of violating the policy and despite an elaborate security awareness campaign.
I recently read a post about improving security awareness using lessons from behavioral science. The field of behavioral economics and its intersection with information security has been a growing interest of mine, and the post I mentioned inspired me to start a series of posts, starting with this one, on the myriad opportunities there are to leverage the lessons of behavioral economics in improving information security programs.
Behavioral economics describes a set of nuances, biases and irrationalities in the way people, on average, thing. This does not mean that every single person will be influenced using these techniques. Also to be clear, these are my hypotheses and I do not mean to represent them as fact. This is intended to be an exploration of the linkage between behavioral economics and information security, to drive discussion and to refine my thinking on the matter.
According to Dan Ariely’s research described in his book “Predictably Irrational”, a group of people who are asked to recite the Ten Commandments, regardless of whether or not they remember all 10, prior to performing a task intended to incite cheating don’t cheat. Likewise, people do not cheat after signing a form in which they promise to abide by an honor code – an honor code that doesn’t really exist.
Ariely’s research found that people who are not asked to recite commandments or sign a honor code generally cheat when given the opportunity to do so, but they do not cheat to the full extent they could have. But if people begin thinking about honesty just before the point of temptation, they stop cheating completely. These effects don’t last long, however, and people must be reminded.
How can we apply this finding to information security?
1. If we put people in a position where cheating or stealing is possible, some number are going to do it. It’s apparently human nature. The threat of getting caught and losing one’s livelihood often doesn’t enter into the equation. Implement controls that affirmatively prevent cheating where possible.
2. Remind people about being honest at points where they have the opportunity to cheat or steal. A once a year conduct reminder isn’t sufficient. For example, an on screen reminder that it’s to be dishonest when completing an expense report form. Be careful, though, some research points out that people become blind to on screen warning messages over time. Possibly something more subtle in the background, stating that employees of the company are known for their honesty.