Randomised Control Trials in Operations


I was reminded by Ben Goldacre’s excellent erudite discussion on Radio 4 last week on the role of randomised control trials (RCT) in the development of public policy that the importance of RCTs in operations management practice is growing, but business schools do not spend a lot of time teaching the techniques and the coverage in operations management textbooks is negligible. Sadly it looks like the Goldacre radio programme is not going to be one of the programmes that the BBC leave online for longer than one week.  Goldacre discusses in detail the theoretical strengths and everyday weaknesses of the randomised control trials of drugs by pharmaceutical companies in his recent book Bad Pharma (subtitled “How drug companies mislead doctors and harm patients”, which the book elaborates in algesic detail). Both Bad Pharma and his earlier book, Bad Science, are very good books for anyone interested in quantitative methods to read. They both make the case for the significance of research methods, but in an approachable way  and in a domain that is salient to everyone.  The radio programme, Bad Evidence, explored “the potential for putting RCTs at the heart of the policy-making process, arguing that not only can they reveal if our existing policies are effective but RCTs have the potential to transform the way we create and implement social policy across the country, from education to health, from welfare to crime.” The programme’s content drew on a report prepared by the UK government Cabinet Office in June 2012: “Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials” by  Laura Haynes, Owain Service, Ben Goldacre and David Torgerson. In justifying the importance of RCTs to policy makers,  the websites of Amazon, Wikipedia and Netflix are used as examples of how RCTs are not just for drug trials, companies are using control trials to decide which website design generates the most income. Visitors to websites can be split into groups presented with different new designs and the old design and the rates of conversions logged and analysed to identify the best design. This is evidence-based service design, but evidence-based service design is not just for websites. Any service where customers can be identified individually and provided with one of a range of services can form the basis for a well-designed RCT, which takes in call centres, appliance servicing and universities. The neatest example of this in the design of operations in the commercial world is covered in a US NPR podcast From Harvard Economist To Casino CEO“,  summarised in the NPR blurb as “Gary Loveman used to be an economics professor at Harvard Business School. Now he runs Ceasars Entertainment Corporation, a giant gambling company. But he still thinks like an academic. He likes to say there are three things that can get you fired from Caesars: Stealing, sexual harassment and running an experiment without a control group.” Customers playing slot machines are identified by the  casino’s cards, so experiments can be designed to work out the optimum levels of benefits, such as free shows and meals,  to keep them spending money. The trialing of interventions described to stop  customers faced by an initial run of bad luck turning on their heels and heading for home is either brilliantly sophisticated or insidiously unethical, depending on your taste.

Even in the university we do not spend enough time thinking about how teaching innovations will be assessed, though as education becomes more and more mediated through virtual learning environments the setting up of RCTs becomes ever easier. In the real world of operations, as tracking customers and tracing products becomes easier the importance of RCTs in operations management practice will increase.



Leave a Reply

Your email address will not be published. Required fields are marked *