Bill Simmalavong founded this blog to explore technology in Education.

How Data Can Identify Engagement

Engagement, we all want our students to have it. We all want them to feel like they are having a great time going through the learning programs we spend hours constructing. But how do we know? How do we measure engagement online?

Let’s start by defining what I mean by engagement and I will be using Phil Schlechty’s (1994) definition but note that there are many variations.

“Students who are engaged exhibit three characteristics: (1) they are attracted to their work, (2) they persist in their work despite challenges and obstacles, and (3) they take visible delight in accomplishing their work.”

The question of engagement is not just unique to education. A lot of money goes into research and development for companies to learn about customer engagement and we need to fight the urge to try to solve the same issues from scratch. We just need to apply some of the principle findings to an education context to better help us create habit-forming and engaging curriculum.

A Look at Marketing

Marketers are big on customer engagement and retention, that is their core focus. With so many products and a limited amount of eyeball time, these firms need to make sure their money is being put to good use. They need to make sure that their campaigns and websites engage their customers so that they get a good return on investment. Most importantly this all needs to be measurable so that they can gauge whether a campaign has been successful or not. If we can’t get data how will we know the impact we have?

And I know sometimes in our line of work – with other responsibilities and whatever else – we don’t really have time to analyse the data. But we must!

When marketers look at websites and assess the effectiveness of the site, they will look at a range of metrics. Google Analytics is just one of the many programs out there that allow you to track what is happening with campaigns. And I know sometimes in our line of work – with other responsibilities and whatever else – we don’t really have time to analyse the data. But we must! It is the only way for us to know the impact of our work and whether our efforts are worthwhile. Would you continue to put in the time to exercise when you don’t see much gain? Of course, you wouldn’t and spending all that time developing learning sequences without measuring the impact is futile.

We can look at 4 metrics to help us start building out the narrative around how we are doing.

  1. Time Spent
  2. Tasks Performed
  3. Login Frequency
  4. User Reactions

Many of our student engagement questions can be solved with these 4 metrics in combinations. Marketers and those that specialize in this type of analysis will generally look at these performers to gauge the engagement of customers. These are also not the only metrics but we can start with a simplified version that allows us to get most of the results and give us a place to start. There will obviously be finer details that may need to be looked at on a case by case basis. It can also be said that “Login Frequency” is more apparent to some website than others. It could be replaced with something like return visits.

What Metrics?

Let me first describe what I interpret each of these metrics as.

Time spent – The amount of time spent on a page or website.
Task Performed – What the user does on the page or website.
Login Frequency – The time between login sessions.
User Reaction – Do they have a positive or negative reaction when visiting your page or website.

It is best to look at each of these in combination with one another and see if we can build a narrative around it. A lack of any one of these may signal various issues that a learner may be experiencing. In the context of being in a face to face classroom, body language will play a huge role in the intervention techniques that one may want to employ. In a virtual school, where we do not have this luxury, we must solely rely on data and various other forms of communication. In these examples, I will be focusing on the virtual classroom. In a face to face context, the data will often be accompanied by field notes about students where the narrative is a little clearer. This is also not to say that there will not be field notes for students in a virtual classroom, it just means that the data you can gather is a little different than a traditional setup. It also means that you are less likely to rely on say, gut reaction due to the lack of body language signals.

Using These Metrics to Tell a Narrative

Let us look at the possibility of having low time spent on the site but the other metrics are turning up fine. The student may be completing many tasks, may have frequent logins, and have generally positive feedback for the online classroom. The lack of time spent is a red flag. If the student submits good work but spends a little time on it, this may mean that the student is not working at their year level and may require a differentiate learning program. If a student submits poor work it may signal that the student is rushing through the work. Without the other 3 pieces of data, we would not be able to narrow the potential factors down to these most likely narratives.

What if the student shows that the only thing the student is lacking is in tasks performed? This may signal that the student may have anxiety when it comes to completing some of the tasks, or may be stuck on a task. A likely call or communication is needed to occur to determine what is preventing this student from completing the work assigned.

What if the student does good work, spends time on the site and performs tasks, but does not login frequently enough? In this case, the student may find the work non-engaging where they find it hard to come back, sit, and complete the work. An alternative study plan may be necessary for the course. Lack of logins is the most obvious sign that something wrong is happening and if it gets to this point the student will need extra support to re-engage. It could also be the case that the student is experiencing trouble managing their time and may need extra coaching in this. In this case narrowing down the potential issues a student may be experiencing.

If the student has a negative outlook on the course this may signify that they are being forced to complete the course. The student is compliant but is not working to their full potential. We are failing to provide the student with material that mateches their interests and outlook on life. It may also show that the subject matter may not match with what the student finds interesting and alternative material may be necessary.

If a student has positive results in all 4 metrics outlined above, there is a high probability that the student satisfies Schlecty’s (1994) definition of engagement.

In the end, if you are not sure if a student likes the work just ask! Know your students and build those relationships. Without these relationships and open lines of communications, it makes it difficult to intervene especially in a virtual school context. And remember that these metrics will only start to build out the narrative, it is with other conversations and field notes that you can make more accurate judgements and interventions.

Schlechty, P.  “Increasing Student Engagement.” Missouri Leadership Academy. 1994

The Importance of Modeling Growth Mindset

The New Way to Tackle Problems, 'OK Google'