Ensuring our in-class tech was an enabler, not a blocker

About the project

Walking through a square on my commute to work, there's a juggler who usually pitches up for a long day of busking. Often I’ll see him there, flinging four or five pins in the air as if it were second nature. Really, what I'm watching is the culmination of years of practice needed to make this seem effortless. I've only seen him drop his pins when something breaks his flow; a dog running over, a car horn beeping behind him — sudden disturbances which are out of his control.

In many ways, teachers are like this juggler. They manage to coordinate their students' needs with a curriculum, communicate with parents, grade work, plan for their lessons, be knowledgeable enough to confidently run lessons, and maintain their own lives. Certainly, unexpected external factors can make teachers drop all their pins, too.

As one ingredient in our ‘big old research pie’ at EF Class, we spoke to our teachers on a regular basis, and observed lessons that used the product as often as we could. Every time we got to see the product used in a natural setting, the single stand-out issue was always students finding excuses not to log in and use it.

In our data funnels, we were seeing teachers drop like flies – our North Star metric (teachers who have taught at least one ‘real lesson’ (at least 2 students for least 10 minutes) within a month of sign-up) was around 1.1%. But the nature of these problems meant they were largely out of our control. Or were they?

We took a step back. We knew that we’d created a fundamental flaw in our product by forcing teachers to enrol all of their students, and then expecting each student to log in every time they used it. Given that EF Class does not have the same model as tools like Google Classroom, but is instead used sporadically throughout the term, this felt like a massive own goal.

Working with a problem… question?

“Can’t we just get rid of student enrolment?” was almost certainly first uttered by an EF Class employee holding a beer, exasperated after a day of observing lessons where students will do anything they can to avoid learning.

To give you an idea of what teachers are up against, we’ve seen kids yeet their laptops five feet into the air, only to slam them as hard as they can into their desks upon landing, not unlike Kevin Nash delivering the ‘Jackknife Powerbomb’ to defeat Booker T in what would be his last World Championship Wrestling title win. We saw that particular move spiral into a laptop throwing competition between three of the more unruly kids. So yeah, high school classrooms are wild these days. ‘Forgetting’ their login credentials, or feigning tech issues is second nature to these kids.

We knew that this issue, coupled with how high a hurdle we’d created for a new teacher before they could get their first “aha” moment in the product, could be a killer. With all this flaring our anxiety, we sat down as a team to flesh out the main complications:

Production application:
Unlike a lot of other tools in the same space (Google Classroom, for example), EF Class follows a lesson-centric model:

The majority of teachers, aside from internal EF partners, view us as a ‘digital textbook’
The first step in our core loop is exploring and saving a lesson plan (before setting up a ‘class’)
Our lesson plans are the point where the user starts a lesson (instead of starting from inside a ‘classes’ area in the product).

Red flags:
Despite having a lesson-centric model, teachers had to first enrol all their students in a class before they could use EF Class with those students. We saw a number of red flags:

We’d created significant friction during the onboarding process. Teachers must sign up, create their first class, ask students to join, and students have to create their own accounts. Then they have to ensure their students remember their logins before they can ever teach an EF Class lesson
Our navigation in-product was more complicated than it needed to be
We’d created huge management overheads for time-pressured teachers.

In order to deliver the first real ‘aha’ moment and close our activation loop for teaching their first lesson, it seemed the smartest thing to do was to eliminate classes for new teachers (at least, until they effectively ‘level-up’ and become more confident users of the tool). This meant that teachers would stand a chance of seeing value in the product before we ask them to do any real heavy lifting.

Working backwards from this ideal

It’s one thing to work as a team of 30 to come up with millions of wacky ways to try and fix something, but another thing to actually find out if any of the ideas have legs.

Firstly, we had to see if our ideas would pass our overlords in the legal team. A few considerations we had to make included how we’d be storing cookies on students’ devices who, because they had created no accounts, had therefore not agreed to our T&Cs. Given that this extra step was a friction we wanted to avoid to allow students to access the lesson faster, we needed to find a legal way around it.

Besides this, any information submitted by students would have to be stored, at least temporarily, while a lesson was being run – without this, teachers would be unable to see how many students had joined a lesson, and the overall results for their class. The issue lay with the risk of students submitting whatever the hell they wanted to their teachers in open text activities: personal information, a job-lot of offensive material, bank details, you name it. At least with logged-in users, both parties are protected by our Privacy Policy – this had the potential to let students go rogue.

Thankfully, early feedback from the legal team on what we could and couldn’t do helped us guide our data policy for Quick-start lessons early on, and somewhat unintentionally, helped scope the tech investment a little better.

Forming the hashtag squaddie

Using a slightly adapted Basecamp methodology of working in 6 week sprints, we had the Product Manager oversee everything for the sake of sanity checking, but ultimately had the sprint led by the nominated designer (spoiler: it’s yer boy).

Once we had assembled a squad, our first job was to figure out what would be possible for each discipline in the time allocated. The biggest issues would be creating a bunch of new data endpoints and a cron job to wipe all Quick-start data in the back-end (something design and product had little control over), as well as working within the limits of a legacy front-end because designing and componentising a whole new flow in our accounts service was gonna take the whole sprint, and then some.

So how did we determine the scope?

Basecamp says ‘The secret to making this possible is something we call scope hammering. We take the chisel to the big block of marble and figure out how to sculpt a feature into the best six week version possible. It’s all about looking carefully at a feature and figuring out the true essence. Not “what can it be?”, but “what does it need to be?”’.

You better believe we hammered the bejeezus out of that big old chunk of marble.

As a newly formed team, we worked on what we thought would be the most optimal flow that took into account current tech limitations, legal issues, any weird edge cases of entering an anonymous lesson with a user account inside our app, and, of course, scope scope scope.

Figuring out what was possible early on, instead of finding out during a handover 3 months later, meant we were able to write up the optimal user flow and catch any major edge cases pretty quickly.

Bringing out the creativity

The next major job was design and front-end tech working hand-in-hand to create the easiest method, constraints allowing, for teachers to begin a Quick-start lesson and for students to join. Limitations aside, we now had a public-facing feature name with expectations we’d need to live up to.

Most prduct designers will be familiar with this phase. It involved a few days of locking myself in a room and coming up with lots of harebrained ideas, which only for the grace of our regular squad check-ins never quite reached a full reconstruction of the IASIP Pepe Silvia Scene.

Of course, as is the nature with a sprint project where every discipline is making new discoveries and reporting back, we found that what was and what could be in scope varied hugely between the start and end of any given week. A particularly exciting tangent of this feature opened up an opportunity to look into rebuilding the legacy areas the feature touched 👇🏼

This required a ton of testing, utilising unmoderated tools like Maze to test the interaction-based elements of our ideas, as well as guerilla usability sessions with anyone that had previous teaching experience we could find around the office (luckily, the crossover seemed to be about 30% of staff that week) for anything bigger. We used techniques like ‘slider scales’, based loosely on ‘Severity Ratings’ from the NN Group, except we directly asked the participants in order to capture their subjective opinion in a way we could still action.

Ultimately, though, with all the smaller concerns dealt with, this had to be a ‘build it and see’ type of test. As soon as we had a working beta, we tested furiously within the squad for several days to make sure we weren’t about to ruin the lives of a few thousand teachers.

We ran tons of 10 minute guerrilla tests on just about every variable to make sure we were getting this right. A great chance to look into our colour palette, colour and icon patterns, and to refine our copy for the feature.

Three amigos took tres mintutos

Because of the nature of working in a squad, we were checking in formally at least once a day, but sharing updates through Slack all day too. This meant that, when we came to the Three Amigos stage, the squad knew about 95% of their dependencies and any problems we might encounter.

By this point, even the weirdest edge cases had been fleshed out. Every member of the squad had given four weeks of detailed input at every stage, so we were all just there to make sure we hadn’t introduced any wildcards. The main thing for me at this stage was to make sure that the team felt celebrated in their success, and to show that I was ready to jump in as soon as QA needed a hand.

I found a new respect for QA engineers and manual testers the world over during this project. After losing my mind testing the same flows, the same behaviours, trying to replicate bugs, and trying to trick our product into breaking – approximately 12,000 times across every device imaginable – we felt ready to run a few internal tests with the whole EF Class team.

This was our big chance to prove the squad mentality worked, and that it had helped us ship a project in record time that had previously been in limbo for two thirds of a year. And it all went seamlessly! Even with two of our ‘teachers’ (read Mel and Stefan, squad web and iOS developers respectively) running lessons from Romania, it went without a hitch.

The squad’s wonderful Web and iOS developers Mel and Stefan gave the wider team the inaugural EF Class Quick-start lessons, and while working remotely. Two firsts for us!

Outcomes

Let’s put it in indisputable terms:

Since inception, 15% of all lessons are now run with Quick-start functionality, with minimal advertising around the feature. Without really trying too hard, and certainly without cannibalising much from our % of regular lessons taught, we have added massively to our North Star metric. Anecdotally, in conversations with our teachers, we hear how useful this feature is, and of course, hear lots of ideas on how to improve it.

Quick-start was adopted by teachers straight away, with little advertising. Without cannibalising regular lessons too much, we appeared to have targeted the exact group of users who were struggling before.

In our four day field study visit to Stockholm, Gothenburg, and Halmstad, we had the pleasure of observing around 12 lessons with massively varied student (and teacher!) abilities. Without bragging, the issue with onboarding 30 kids every lesson that we’d seen ad perpetuum before this had all but gone away. If anything, the process had now been made so seamless for these little rascals that we got to see all the other issues with our product we’d not previously spotted. And thus, the cycle of testing and sprinting began again!

After four days of observing lessons and seeing the success of the feature, we were confident enough to advertise the feature to all existing teachers via our weekly email campaign. And yeah, we really do reward you with confetti when you start your first lesson!

Special thanks to Chris Powell, Melania Moga, Vinod Aithal, Ștefan Ceriu, Sasha Makarenko, and Ziemek Obel