Big Data and Intuition- A Case of Babies and Bathwater

31 May

A recent symposium held at MIT reproduced the constant theme around organisational decision making and big data- gut instinct out and data driven decisions in. Whilst data and information are important and beneficial to organisations, such an anti intuition message  can carry huge amounts of risk, with one particular example springing to mind- the 2008 financial meltdown. In this article I’d like to address the role of intuition in decision making and its relationship with data, because regardless of the current narrative, intuition is vital to good decisions, you “just” need to know the limits of both.

The role of intuition in organisational decision making was defined by the work of Herb Simon, who provided a strong steer on what intuition is- pure and simple, intuition is recognition.  Intuition is that feeling something is wrong or right, you’re not sure why, it just feels that way. Psychologists suggest a pattern has been recognised, you may not be aware of the details, but the feeling is one of knowing with no explanation why, but it’s experience talking, you’ve seen it before. Researchers in the field, for example Gary Klein, identify that people operating in fields such as fire fighting use intuition derived from experience to generate mental models, assess them for effectiveness against a current situation, and adapt them. This model of decision making is fast, highly effective, and largely unknown to the person using it; it’s intuitive. This is tacit knowledge in action, highly effective information which exists outside of formal procedures and methodologies, and instead exists in people’s heads; the reason why collectively an organisation knows far more than it can say.

Data and information is formal in the sense that it sits there in front of you, it’s tangible. And as I discussed in an earlier article, the temptation is that data derived from highly sophisticated technology and methodologies is giving you an answer- the answer you have been looking for, the facts. The problem with just using data and facts to make decisions is that they can sit outside of context, and this where intuition comes in.

A lot which is written about big data is all the knowledge of the “world out there” it gives you. The data gives you new, faster and larger amounts of information and insight on your customers and competitors, it gives you the map, an answer. However, new technology and new platforms may not have changed the “world inside” your organisation, what the data tells you to do needs to be contrasted against the capabilities inside the organisation, and this where experience and intuition is vital. If a highly experienced member of staff says it just doesn’t feel right, listen, and put structured effort into finding out why, dig for that tacit knowledge, don’t just listen to the data.

Organisational initiatives fail because we think procedures and algorithms can suddenly turn that member of staff who has never delivered into an energetic go getter, or dominate a market which a company has failed to penetrate for the past 15 years, or predict human behaviour.  Data won’t change the internal capabilities of an organisation, it might point to a direction, but it’s the tacit knowledge, the intuition, which could tell whether you have the legs to get there. The 2008 financial meltdown looked impossible when analysing the data from models such as Value at Risk, but there were many people within the industry and on the street who intuitively knew it couldn’t stay on top of market volatility.

So, don’t analyse data in isolation, look at it within context, and that includes listening to intuitive doubts. To do this effectively you need to dig for that tacit knowledge, because quite often your best and most experienced staff will know far more than they can say. Effective intuition is linked to expertise, so increasing the visibility of tacit knowledge can increase the level of expertise within an organisation.  Expertise, as Gary Klein advocates, is an effective means of harnessing experience into the decision making process. By increasing the level of expertise, as opposed to always increasing methods and procedures, you increase the level of innovation and balanced decision makers in an organisation. I’ll discuss expertise further in my next article, but until then, when it comes to big data; don’t throw the baby out with the bath water.

2 Responses to “Big Data and Intuition- A Case of Babies and Bathwater”

  1. marciano49 July 8, 2013 at 9:09 am #

    Yes, especially in technology and decision making circles. Intuition is a natural process so I think we need to focus on ways of improving it not reducing it.


  1. Has intuition gotten a bad name? | counselorssoapbox - July 8, 2013

    […] Big Data and Intuition- A Case of Babies and Bathwater ( […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: