A Possible Life without Data Scientists

15 Aug

I read a very interesting article on smartplanet.com entitled “Why big data means job growth for non- data professionals” and from my perspective just this title summed the data explosion up perfectly. The vast increases in data if used correctly (and I mean a big IF) will have far more value to non-data professionals and to their organisations than anything a data scientist could produce. This is because data represents an opportunity to develop and accelerate human expertise far more than any spurious predictive model.

The quest for data scientists is still built on the beliefs that complexity is mathematically tractable. The moment a belief in a predictive model occurs then trouble begins, it’s the banking crisis model all over again. You can’t predict complex systems, but when you believe you can, you think you’re in control, that the model can do it for you, and that’s when human beings become passive. It’s a form of electronic social loafing, when predictive models become part of your cognitive dynamic, expertise takes a back seat and conflicting data or hunches get explained away. Just take a look at the Enron case study, all the data was there but the problems kept being explained away, with plenty of models, graphs and tables to help.

Better use of data does not lie in the hands of data scientists, it lies in the hands of experts, in any industry, using it to adapt and test their mental models. Good decision makers are not passive; they are adaptive and use vast amounts of tacit skills and heuristics to navigate complexity. Good decision makers also do not predict, they anticipate- cognitively, prediction is what leads you to lock your car keys in your own car, anticipation makes sure you don’t. The problem with following the examples of good decision makers is that very often they know far more than they can say. Unlocking this expertise and transferring it is true high value information but you’ve got to know how to do it, otherwise you could be following the wrong cues.

Big data and some of the latest BI platforms represent huge opportunities for experts in any field to operate at more adaptive levels which allow them to identify and lever risks rather than be buried by them. These technologies also represent the chance to help unlock tacit knowledge and turn novices into experts faster, and also broaden the range of expertise. When a focus is finally placed on how people actually use data to make decisions as oppose to how can more people use data to make decisions then we’ll finally see some real developments.

All decisions start with a hunch, intuition. Hours of research have taught me that, and the rules of thumb people generate to manage complexity are just as effective now as they have always been; we just need to identify them, support them and transfer them. So, forget data scientists, in about 5 minutes you can make almost anyone a better decision maker using data via Google, so focus on a usable, simple BI platform and then use it to support expertise.

Final thought- The data explosion represents huge opportunities for non-data professionals but they need to be used correctly and responsibly. Remember, the data sets are now are so vast you can prove almost anything you want via correlative statistics, so positive cases are of dubious value. At that point, ask an expert. I’ll explain further next time….

2 Responses to “A Possible Life without Data Scientists”

  1. John Coppins August 16, 2013 at 10:05 am #


    Interesting blog and I agree…. partly!

    A couple of points:

    1. Your description of a data scientist is more closely associated with a data miner. The difference between a data miner and a data scientist is the business understanding/expertise. So the ideal here is to merge the business capabilities of the subject matter expert with the mathematical capabilities of the data miner. I say ‘ideal’ here as these people are as rare as hen’s teeth. Unfortunately what we are currently seeing is an absence of quality in data science – mainly data miners re-badging themselves as data scientists which comes under the heading of a ‘very bad thing’! Now, whether we will ever get to the point of a large number of people encompassing the data acquisition, mathematical and business skills associated with a true data scientist is a different subject

    2. You focus on predictive modelling, which is fine but only one aspect of ‘big data’ analytics. For example, the process of discovery is not predictive but an equally valuable aspect of analytics (‘big’ or otherwise).

    I would suggest that most organisations do not bet the farm on predictive models and I suspect that they never will, although over time there will inevitably be occasions where we over rely on technology with the resulting systemic risk (I don’t think we will ever eliminate credit crunch style crises). However, I think we would agree that intelligent use of analytics and modelling to support expertise within an organisaiton is highly beneficial. In this way, I believe that the paradigm of self-sustained machine learning as a basis for decision making is unlikely to ever become mainstream.

    A couple of problems associated with relying on expertise are that it is thin on the ground (creating risk to the organisation should it become unavailable) and it introduces inherent bias an limitations. What do I mean by this? Expertise is based on experience and frequently looks for situations that have already been encountered. As such, the ‘new’ is likely to be missed and in an era where we are getting accessing to so many different and new data sources, this is likely to miss opportunities.

    My preference, and I suspect you would agree, is to use technology that supports the uncovering of these new relationships and opportunities and expose this to the expert decision maker who can assess whether this actually is an opportunity or whether it is just a spurious correlation lacking in organisations value. This should be more than just BI which is too limited in scope – there is real value to be had from anayltical technologies that go beyond a simple BI remit.


    • marciano49 August 23, 2013 at 9:36 am #

      Thanks John! Beacuse we come from two perspectives I really enjoy trying to meet in the middle. I’m currently working on research based on the neocortex which essentially uses memory, prediction, action and how this relates to data informed decision making. I’m weighting my work with technologies (big data etc) more in the direction of situational awareness to create better data informed decisions. This means seeing expertise not as “standard” cognition-memory, prediction, action but memory, anticipation action and the role software can play. So, Re- expertise and bias, expertise is experience based but the true nature of expertise is adaption. In the field studying experts we see a prediction then a simulation and then adaption; we get the opposite of bias, constant updating of the mental model based on the ability to see subtle pattern breaks. True, fixation can occur, but experts overwhelmingly maintain situational awareness, and getting this into data analysis is what I’m currently working on. Happy to share about the research if you’re interested, value your opinion, just get in touch…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: