University students using laptops and digital tablet, working together

Meta-Predicting Student Risk: What Does It Take to Support a Predictive Model?


Institutions are increasingly interested in how their student success efforts might benefit from a ‘360 degree view’ of the learner. While an approach that collects and synthesizes data from a large number sources may be a laudable, and capable of generating broad insights, a different point of view is summed up in the phrase “Less is More.” Indeed, there are pedagogical and logistical advantages to developing predictive models of student course achievement that make exclusive use of data from the learning management system (LMS).

From a teaching and learning perspective, it means focusing less on factors over which students have no control (like demographics and educational history), and more on concrete, real-time behaviors that can be changed.  While LMS use is not a direct indicator of learning and teaching, it can provide proxies for the deeper practices that can affect student study practices and ultimately their success.

Logistically, it is faster and less expensive to implement a predictive model with LMS-only data compared to connecting with a Student Information System, which usually carries a substantial amount of custom work to integrate.

The problem is that the accuracy of predictive models using LMS data can vary widely from institution to institution. While for some institutions we have been able to create a model with greater than 90% accuracy, for others the accuracy rate is much lower and is not a reliable source of information about students.

Why is that?

As it turns out, institutions and instructors use the learning management system in very different ways, which has a tremendous impact on the data that is generated.  Furthermore, the kinds of practice that are good for feeding a predictive model are also the kinds of practice that are good for teaching and learning on online environments.

In a report released today, we are publishing research by Diego Forteza, one of our data scientists at Blackboard, which describes an approach that we have developed to assess an institution’s readiness for predictive analytics using LMS data.  We call it a ‘meta-predictive’ model, because it predicts the ability of an institution to form an accurate prediction.

Based on this research, we recommend the following practices which, assuming that they are carried out in a pedagogically sound manner, will increase an institution’s ability to model student success, while at the same time (or even as a function of) increasing student engagement at scale.

  • Make consistent and frequent use of the LMS gradebook
  • Include LMS resources & activities that require early and frequent access by students
  • Pay attention to student enrollment so that online course rosters are the same as official records
  • Enforce deadlines so that no student ‘misses’ assignments
  • Design the online environment in a way that promotes regular and consistent student engagement

Let’s not forget that the ultimate goal of predictive analytics of student success is not to identify students at risk — it’s to help students succeed.  And the way we help students is by both intervening when they go off track, and optimizing the teaching and learning environment so as to remove barriers to progression.  When working with learning analytics, we are forced to think in both directions at the same time.  And for students, that is a very very good thing.