Test and learn

We're aiming to take a 'test and learn' approach to culture/behaviour change with a group of people in a specific customer facing job role. The role is a new one and we have to make sure that staff understand what it entails. They will have to develop the technical skills they need to do it well, and the social and interpersonal skills to do it in an engaging and effective way that gives the customer a good experience.

Our specific action is to 'Develop a behavioural change intervention strategy'. Developing an 'intervention strategy' seems to imply a 'plan and implement' (waterfall approach) antithetical to the 'test and learn' (agile approach) with its emphasis 'on short development cycles and constant testing – the learning from which is fed back into the development cycle.'

This has prompted us to consider what we mean by test and learn in this context and what that would involve. 'Test and learn' typically involves taking 'one action with one group of customers, a different action (or often no action at all) with a control group, and then comparing the results. The outcomes are simple to analyze, the data are easily interpreted, and causality is usually clear. The test-and-learn approach is also remarkably powerful.' (From: A Step-by-Step Guide to Smart Business Experiments)

So can we develop a 'test and learn intervention strategy'? Henry Mintzberg (a management expert) in a classic 1987 article, talks about 'crafting strategy', 'Now imagine someone crafting strategy. A wholly different image likely results, as different from planning as craft is from mechanization. Craft evokes traditional skill, dedication, perfection through the mastery of detail. … Formulation and implementation merge into a fluid process of learning through which creative strategies evolve.'

He continues with the image of a potter crafting a pot and develops the analogy to the point where he says 'My point is simple, deceptively simple: strategies can form as well as be formulated. A realized strategy can emerge in response to an evolving situation.' This thinking fits well with the test and learn approach.

Thomas Davenport, in an excellent article 'How to design smart business experiments' tells us that 'Tests are most reliable where many roughly equivalent settings can be observed'. That holds true in our situation when we are looking at many locations where the new job role will be. But he also says 'Formalized testing can provide a level of understanding about what really works that puts more intuitive approaches to shame. In theory, it makes sense for any part of the business in which variation can lead to differential results. In practice, however, there are times when a test is impossible or unnecessary.'

So our question is can we design an appropriate experiment that will lead to us forming an intervention strategy? Davenport offers an approach:

1: Create or Refine Hypothesis
2: Design Test
3: Execute Test
4: Analyse Test
5: Plan Rollout
6: Rollout
7. Add to Learning Library

Steps 1 and 2 are likely to be the most difficult to design. A hypothesis around 'behavioural change' involves a vast number of variables: manager style, tools to do the job, training that contributes, characteristics of job holder, customer expectations, etc. not to mention competing theories of behaviour change. An excellent handbook Reference Report: An overview of behaviour change models and their uses will be helpful in thinking this aspect through.

So we'll give it a go and see what we learn. What is your experience of test and learn around behaviour change? Let me know.