During my time at Neuronation, the founders were taking part in Google's 2019 "Sustainable Engagement Forum", which was a collection of product people from different apps sharing ways of applying behavioural economics to product development as a way of fostering increased engagement in mobile apps. As such, we were interested in running tests ourselves to see if we could tangibly improve the user experience of NeuroNation's flagship brain training app.
I conceptualised the feature and spec'd out the MVP, analysing the data (in Amplitude) to measure impact on retention and conversion post-release.
Our goals included:
(a) Research and understand behavioural economic theories and brainstorm how to apply these to a concrete product experience.
(b) Improve engagement, retention, and/or conversion through experimentation.
(c) Address user requests and wishes through our experimental ideas.
If you're interested in more details about self-determination theory and its sub-theory – organismic integration theory – and what these mean, scroll to the bottom. For the purposes of this case study, it gave me an idea: would giving users autonomy over an aspect of their experience with a product make them more likely to continue returning and continuing?
Now, in product development, we talk about personalisation constantly, and and it's core, it's often about using data to surface the best "thing" for a user at the best time. What we talk about less is something that sounds deceptively simple: giving users the decision, i.e. autonomy over their own experience.
As a brain training app, NeuroNation needs users to develop a habit in using the app frequently in order to reap the rewards, i.e. a more agile brain. While the program the users follow in the app is determined by factors such as their assessed strengths, areas for improvement, and amount of time they can dedicate to training, I wanted to see if giving them autonomy over some aspect of their program could make them more likely to continue training.
Hypothesis: Giving users increased autonomy over their brain training program will make them more intrinsically motivated to continue, and more likely to continue training, as measured by returning users on day 3, day 7, and day 14.
We decided to use difficulty of the exercises as the test. If users scored within the top percentile for a specific of exercise, we would give them the option to increase the difficulty for next time. This would function (a) as a reward for high performance, and (b) a motivator for coming back to try the harder exercise.
We also decided to give users the option to make the exercise easier if they scored lower, giving them the option to find a more appropriate level of challenge and again, motivate them to come back and try again.
We launched the experiment within 3 days, and allowed it to run for 3 weeks so that we could measure the impact on day 14 retention, i.e. the number of users returning to complete a brain training session within 2 weeks.
We found that there was a slight increase in users returning to complete an exercise, but it wasn't significant. However, we found that conversion had been impacted, i.e. the number of users signing up and paying for a premium subscription after having started training during the experiment window.
There was a 14% increase in conversion for over 30 year olds, but interestingly, for 18-30 year olds, who had historically converted at much lower rates given a general perception of brain training not being necessary for younger, conversion increased by 30%.
Introducing autonomous difficulty adjustments resulted in a 30% increase in conversion in 18-30 year olds.
We gathered qualitative feedback from users too, and found that they appreciated the ability to decide on difficulty for themselves, so the feature was kept and became part of NeuroNation's core USPs.
Below, you can see how the "challenge mode" screen looks, as found in NeuroNation's app store imagery.
Self-determination theory (SDT) is a psychological framework that explores human motivation and behaviour, focusing on the factors that drive individuals to engage in certain activities and pursue specific goals. It sets out that people have innate psychological needs for autonomy, competence, and relatedness, and the fulfilment of these needs is crucial for fostering optimal development and well-being.
The three basic psychological needs in SDT are:
Autonomy: This refers to the need for individuals to experience a sense of choice and volition in their actions. People desire to feel that their behaviour is self-determined and reflective of their own values and interests.
Competence: This involves the need to feel effective and capable in one's actions and pursuits. Individuals seek opportunities to master challenges and demonstrate their abilities, fostering a sense of competence in various domains.
Relatedness: This pertains to the need for meaningful connections and relationships with others. Social interactions and a sense of belonging contribute to the fulfilment of this need.
Organismic Integration Theory (OIT) is an extension of SDT that delves into the different ways individuals internalise and integrate external regulations for behaviour. OIT categorises types of motivation along a continuum, ranging from more self-determined to more controlled forms of motivation.
The main forms of motivation in OIT are:
Intrinsic Motivation: This is the most self-determined form of motivation, where individuals engage in an activity because they find it inherently enjoyable, interesting, or personally satisfying.
Extrinsic Motivation: This type of motivation involves engaging in an activity to attain external rewards or avoid punishments. Extrinsic motivation can be further divided into various subtypes, such as identified regulation (behaviour aligns with personal values) and external regulation (behaviour driven by external demands or pressures).
Amotivation: This represents the absence of motivation, where individuals lack a perceived connection between their actions and desired outcomes.
So, our feature gave users who performed well a reward in the form of an invite to try challenge mode, appealing to a sense of extrinsic motivation. As they start to take on challenge mode, and master other activities, unlocking challenge mode for several exercise types, they start to develop satisfaction from doing well at the exercises, shifting this extrinsic motivation more in the intrinsic direction. Intrinsically motivated users are more likely to find brain training inherently fulfilling, and therefore more likely to continue returning.
Or so we thought! Although we found a limited impact on retention, the perceived value of autonomy is likely what led to an increase in conversion.