An engine that drives innovation is how business experimentation is often referred to. Whether it is about deciding the size of the login button on a website or making multi-million dollar decisions, experimentation has become a major tool in the hands of business owners. What started as a good value addition has now evolved as a business necessity.
So what explains its growing popularity? Analytics India Magazine caught up with Akhil Behl, the head of data science at Fractal, to understand it.
AIM: Tell us about your professional journey.
Akhil Behl: I joined Fractal nine years ago in the first R&D data science team (called Fractal Sciences) at a time when the field was just emerging. Being trained in economics, experimentation was one of my interest areas. One of the things that I worked on was ‘Trial Run’ which is a cloud-based tool that brings analytics to business experimentation, allowing location-based businesses like retailers, quick service restaurants, bank branches, etc., to run experiments systematically and scale the process. So, I was originally the founding data scientist and then went on to work directly with clients – helping prove the value of our product to them and assisting the sales team in converting those clients.
AIM: How would you define business experimentation? Why is it important?
Akhil Behl: Business experimentation brings in the rigour of making sure that we establish a counterfactual in the technical sense. So, a counterfactual is understanding what would have happened if the experiment was not undertaken. A common example of this is in testing for new medicines. Whenever you’re giving a group of people a new medicine, you also want to have a control group or a counterfactual group. So, for example, you’re either giving a placebo or not giving anything to one group while you give the drug to be tested to the other group, and then you compare the two groups and evaluate their health outcomes. So, experimentation is really a discipline that works on establishing a reliable counterfactual in many different scenarios that may appear depending on the nature of the experiment you want to do.
Organisations that have been analytically savvy have been doing this for a long time, for example, digital businesses. The digital businesses built around Internet 2.0 have kept business experimentation as a part of how they do things for the last 15-20 years. In the non-digital, traditional business world, some industries have adopted these practices and some industries that have not; in many situations in the former category, business experimentation is adopted due to regulatory compliance requirements.
AIM: What is the importance of the experimental evidence?
Akhil Behl: Some of the reasons why experimentation is becoming more pervasive, and commonplace could be broadly divided into two buckets. In one case, experimentation is being used to replace human instinct. In the second case, experimentation is being used to choose between AI systems or other expert rule-based systems or machine intelligence-based systems.
In my assessment, there are three main reasons why experimentation is becoming commonplace – the rise of the digital economy, the increasing complexity of businesses, and a lot of decision-making via machine learning systems. In response, the fact that there’s enough precedent, the fact that business is getting more complex overall for human instinct to understand and direct, and the fact that a lot of decision making is algorithmic anyway, experimental evidence is becoming more and more important and ubiquitous.
There are three types of businesses – one that believes in experimentation and is actively investing in this toolkit. The second category includes businesses that believe in experimentation but haven’t adopted it at scale. The third one probably does not believe in the value of experimentation anyway. However, it is the second group that forms the majority of businesses today.
AIM: How can a company build a data-driven culture through a business experimentation framework?
Akhil Behl: Some factors help make the data-to-decision making process easier and better suited for adoption. One is regulatory compliance; the second is having top-level buy-in, especially if you have members in your executive teams who prefer at least corroborating insight with experimental evidence; the third is investing in a platform.
Business experimentation has the design phase, the execution and analysis phase, and lastly, the act phase. While businesses invest time and resources in the last two phases, not enough attention is paid to the design part. If you make critical mistakes in the design and planning phase, you run into a few risks in the last two phases, for example, being left with inconclusive tests. The worst issue if you do not design well enough, is you could make two different kinds of errors. Your analysis can either tell you that there is a positive effect where there is none, or worse, even a negative effect. You can also end up in the opposite situation where the test tells you there is no effect, and you might kill a good working idea. So, my suggestion for organisations that want to build a data-driven culture is to spend a lot more time in the design and planning phase of an experiment which should compress the time spent in the later analysis and act phases of the experiment.
AIM: When we talk about business experimentation, there are two schools of thought. Some people swear by its usefulness, while others outrightly dismiss it. What is your point of view?
Akhil Behl: Definitely, there are two schools of thought – there are believers in experimentation, and then there are non-believers in experimentation. The non-believers are those who think experimentation is too difficult, complex, or time consuming to be useful.
There is some truth to this. If you look at the physical sciences, it is really easy to run experiments there because one deals with the natural world where there is no strategic interaction. There aren’t sociological norms that drive behaviour, etc.
But when we speak of social sciences – Sociology, Economics, Psychology, Epidemiology – these disciplines are more complex than natural systems. They consider several strategic interactions and cultural and behavioural norms, which are pre-established within which people behave. Businesses operate in similarly complex contexts. These require a lot more planning and newer methodological tools. Because of the inherent difficulty with the domain, companies may shy away from trying out experimentation completely. Then some businesses may have tried out experimentation in the past but, because of reasons like the absence of the right tool or expertise, derived little or no value and given up.
However, while running business experiments is difficult, it is no more difficult than experiments in all of these other fields. It just needs a lot more care, planning, and the right toolkit.
AIM: What is the role of analytics in business experimentation, how important it is and how are the two interconnected?
Akhil Behl: Analytics is very important, and it can be used in business experimentation in several ways. Firstly, it can help a human being generate hypotheses if one has no prior hypotheses to start with. In this case, they can definitely lean on analytics, more specifically, observational analysis, to come up with hypotheses and then filter them out based on their instinct and understanding. Secondly, analytics can tell business decision-makers what kind of experimentation is appropriate given a specific situation. The third is the field of algorithms which falls in the exploration-exploitation paradigm which are combining experimentation and optimization in a single step in various decision-making scenarios in modern business.
As we’ve moved to a more and more machine intelligence-powered business ecosystem anyway, we will see an increased move towards experimentation. In addition, organisations overall will move towards more experimental decision making, even for cases where human decision making is required. At least from my perspective, I don’t see these two things as contradictory; it is more than experimentation is a tool for the human decision-maker and not a substitute.
