Data, versus Intuition, versus Data + Intuition

On Tuesday September 16, the Australian Financial Review carried this brief article by Patrick Durkin:

Forget data, executives go with their gut instinct

If we’ve got evidence, let’s go with the data. If all we we’ve got is opinion, let’s go with mine. That’s the favourite management advice of NBN Co chairman [sic] Ziggy Switkowski.

It seems he’s not alone.

A report by the Economist Intelligence Unit confirms what many of us suspected. When it comes to big decisions, executives prefer to go with their gut instinct and intuition, rather than data and analytics.

Close to 60 per cent of executives rely on their intuition or the advice and experience of a trusted colleague rather than on data, according to the survey of 1135 C-suite and board-level executives undertaken by the EIU in May this year.

And it seems Australian executives are more willing to go with their gut more than most.

Sixty-eight per cent of Australian respondents said they relied most on either their own gut instinct (40 per cent), or that of a trusted colleague (28 per cent), compared to 58 per cent for their global peers (where 30 per cent rely on their own gut and 28 per cent on the experience of others).

Just 29 per cent of executives globally, and only 22 per cent in Australia, said they placed the biggest reliance on data and analytics to making their latest big decision.

The mining and resources sector is our worst offender, with three-quarters of executives saying their gut instinct ruled in matters of big business.

According to Malcolm Gladwell’s best selling book Blink, our snap decisions based on small amounts of high-quality information, or gut instinct, often turn out to be highly accurate.

But head of PwC’s data team John Studley warns that Australia is badly lagging the field in terms of incorporating data into our decision making.

“Coles might run a price promotion and watch the data to see whether it is working or not, or it might be a targeted marketing campaign for mortgages in a bank where they will trial something among a certain cohort of customers and measure that using their data, that is becoming the successful path to decision making.”

Studley says the problem is a lack of skill and experience within Australian boardrooms. Twenty-five per cent admitted they lacked the skills or expertise to make greater use of data.

“It doesn’t mean building an enormous warehouse and populating it full of every data asset you can conceive of but you have to identify the key decisions and work back to find the data you need to best inform that decision.”

It’s not so bad. Our unconscious intuitions, which make themselves known to us via our ‘gut feelings’ and which we call ‘gut instincts’, are the sum of all of our experience with making these sorts of decisions. This gives us a deep reservoir of relevant knowledge, which our unconscious minds then collect, assess and evaluate. Our brains then summarise that evaluation into a single, simple emotional response to which our conscious brains are highly attuned, and which helps guide our decisions.

As is implied in the responses from Australian executives, this is a mode of decision-making with which they are quite comfortable. And for the most part, it works and works well. For example, most of our reactions when driving – moving the wheel, changing gears, checking our mirrors and the dashboard, maintaining our speed – are the result of processing in our unconscious brains. We trust these thousands of life-or-death decisions, made second-by-second, to the workings of our unconscious brains, while our conscious executive brains, up in our frontal lobes, get on with the important business of having a conversation with the person next to us, blocking out the noise made by the kids in the back, drawing up our shopping list, etc.

We are wise to follow the counsel of our instincts. As Blaise Pascal wrote in Pensées

The heart has its reasons, which reason knows nothing of. We feel it in a thousand things. 

However, problems arise when we find ourselves in new situations, where our reservoir of knowledge is only loosely related to the issue at hand, and thus cannot be safely applied. In fact, in these situations it can be dangerous to rely only on our gut reactions and intuitions – we will most likely misapply a paradigm, or pattern of thought, draw an incorrect conclusion, and repent our mistake at leisure.

Things become even more difficult when the situations that we observe can be explained by two different paradigms, and we have to choose between which explanation is the most correct. Often the decision will have to be made in the knowledge that an incorrect choice will lead to an inappropriate strategy, which is likely to be costly. The worst situation is when we fail to see the new paradigm developing, so that we continue to act according to a strategy which is increasingly inappropriate, at increasing cost.

These developments are ubiquitous in business and management. This is why it’s important to have a grasp of the situation at hand via regular reviews of recent data, and to keep an open mind about what is happening and how developments might be unfolding.

The question then isn’t: ‘Is it better to rely on data, than on intuition?’ We can’t avoid relying on our gut instincts, and we really shouldn’t make decisions without data. The question is: ‘How should we integrate both data and intuition into our decision-making?’

Often the data will give an unambiguous indication of what we ought to do. For example, if we see a green light while waiting at an otherwise-deserted crossroads, we should proceed. Our intuitions will align automatically and seamlessly into accordance with this decision.

It may be more difficult when we are faced with significant, or novel, or novel and significant decisions. The way that I think is best to proceed with significant decisions, in most areas of life but especially in business and investing, is

  • first, to gather as much appropriate data as possible, consider what it means for our situation, consider the options that arise from our new understanding, and evaluate those options according to objective (that is, transparent, clearly defined and universally-applied) criteria, producing a net assessment of the benefit or cost associated with each option;
  • then, to sleep on the decision, so that the unconscious mind has sufficient time to match its picture of the situation with appropriate instances from its deep well of experience, and present the conscious mind with its own net assessment in the form of a gut instinct.

Of course, there is no reason why these two approaches need be in agreement! The decision-maker, or makers, should then engage in further discussion with colleagues, friends and experts, to examine why they feel the way they do, and to find a way forward. The aim is to arrive at a decision which is best, given the circumstances, and using all available information to inform that decision.

Edward de Bono has some good things to say about this topic in his book Edward de Bono’s Thinking Course. From page 100:

If information is sufficient to make the decision for us then we, as humans, are superfluous. We are only called in to make decisions when an analysis of information is insufficient – that is to say, when we have to speculate or guess or apply human values and emotions. So the human element in decisions is vital. In the end all decisions are emotional.

And from page 138:

We need to place emotions, feelings and values in their proper perspective. In the end they are the most important part of thinking – but only if they are used in the end rather than at the beginning.

This last quotation indicates that, as is often the case, we need a process within which we can make best use of the knowledge at our disposal.

It would be good if the next survey of executives asked how they integrated factual data and unconscious intuition in their decision-making, rather than setting the two up in opposition. But even then, we might not get to the heart of the matter. As Pascal said, we may not know why we do what our instincts lead us to do. And as President Kennedy wrote, in the introduction to Ted Sorensen’s book Decision-Making in the White House: 

The essence of ultimate decision impenetrable to the observer – often, indeed, to the decider himself.


About Stebbing Heuer

A person interested in exploring human perception, reasoning, judgement and deciding, and in promoting clear, effective thinking and the making of good decisions.
This entry was posted in Decision-making and tagged , , . Bookmark the permalink.

1 Response to Data, versus Intuition, versus Data + Intuition

  1. Pingback: Data and Intuition: Matt Levine’s Take | The Stebbing-Heuer Project

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s