LOOM^TM

Loom Learning was proud to sponsor the Sloan Consortium’s 17^th Annual International Conference earlier this month (and especially the Dessert Reception at Epcot—those fireworks were spectacular!), where I gave a presentation on learning analytics.  My presentation served as an introduction and overview to the field of learning analytics, which we’re currently delving into as we develop our Weaver Analytics tool.  Here is the presentation-turned-blog-post, or you can see the presentation in full here.

Introduction to Learning Analytics

Major investments have been made by institutions throughout the 1990s on ERP systems with limited return.  But now we are seeing learning analytics begin to emerge.  What is this and what does it mean for higher education, especially as experience and intuition play a major part in decision-making in higher education still?

Although relatively new, there are already many meanings for the term learning analytics.  George Siemens, the educational technologist who led the first annual Conference on Learning Analytics and Knowledge at the beginning of this year (and learning analytics guru), defines the terms “Educational data mining,” “learning analytics”, and “academic analytics” in the following ways:

• Educational data mining An emerging discipline that is “concerned with developing methods for exploring the unique types of data that come from educational settings, and using those methods to better understand students, and the settings which they learn in.”
• Learning analytics “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.”
• Academic analytics Addresses a mix of administrative and learning analytics.  This is closest to what we call Business Intelligence in corporate settings.

In his definitions, “learning analytics” is more precise than “academic analytics.”  The former is used to describe the learning process specifically, whereas the latter describes how data analysis is used by the institution as a whole.  Each term, however, is focused around the data collected by institutions.

Big Data

So let’s quickly talk about this data.  There is so much of it now that we refer to it as big data.  Big data are datasets that grow so large that they become awkward or impossible to manage in the manners traditionally used.  So it becomes vital that we adjust so that we can access that data.  For example, Jeff Jonas, of IBM, says that organizations are as smart as what they know.  What they know, of course, comes from data, both structured and unstructured, which then forms the perceptions the organization holds as a whole.

Jeff Jonas also says that almost all data starts out unstructured, and is then structured by humans.  This especially makes sense when applied to educational institutions which are obviously focused around structured data.  Curricula, courses, syllabi… these are all intensely structured.  But we must also be aware that the way we implement structure may also be causing the problems we face.  Think, for example, on the ways we use standardized testing to measure success, and just how narrow those boundaries are.  I will come back to that particular point, later.

So there is the idea that we should be using this enormous amount of data that is being collected in the decision-making process, rather than our gut instincts as I mentioned earlier.  Ground-breaking, right?  But Big Data leaves us with new challenges.  There is, for example, the risk of data amnesia—where an organization loses track of certain data, leading to inefficiency.  Calling one prospective student three times, for example, to ask them if they’d like a campus tour when they just visited last month.

In order to use learning analytics effectively, we must be sure to define explicitly what we need to know and what data will most likely be able to give us that information.  But we have such a large quantity of data now that the manner in which we tackle this process has changed.  New models are required if we are to approach the data and extract from it what we need, because quantity changes things.  As David Gelernter said, “If you have three pet dogs, give them names.  If you have 10,000 head of cattle, don’t bother” (Hat tip: George Siemens and Phil Long).  Learning analytics holds the potential to give us the platform on which to build those new models.

The benefits of using learning analytics

I recently did my first mud run.  Have you heard of these?  They’re really popular right now, and after doing a few triathlons and getting into half marathons it seemed a natural progression (read: I got sucked in by friends).  Mud runs consist of wading and, in the rare instance when you come across terra firma, running through miles of mud and completing a number of obstacles.  It’s a challenge, and you know it’s going to get messy.  Not unlike wading through this huge amount of data to get at what you need.

There’s no denying that beginning this process is a huge undertaking, and that you’re bound to get a little bogged down in it all at some point.  So, more specifically than just allowing us to access our big data, what are the benefits of turning to analytics for institutions?

• Improve administrative decision-making and the allocation of resources.
• Can identify at-risk students and provide intervention to help them achieve success.
• Can create an understanding of an institution’s successes and challenges.
• Can innovate and transform academic and teaching models at an institution or system of institutions.
• Increase organizational productivity and effectiveness by providing up-to-date information and allowing quick response times to issues.
• Assist in determining both tangible and intangible value generated by the faculty (ie research and reputation)
• Can provide learners with insight into their own learning habits and suggest ways to improve (ie Signals at Purdue and Check My Activity at UMBC).
• Can also point learners to resources at their institution that they may otherwise not have been aware of.  John Campbell claims this as one of the most surprising and effective results of Signals at Purdue.

Another, and possibly the most universal, goal involved in the adoption of a learning analytics system by an institution is to improve retention.  Vincent Tinto defined a dropout as an individual and an institutional failure, noting that if a student does not define their own departure in those terms, then neither should the institution.  His model (see below) revolves around the idea that both social and academic integration play pivotal roles in the reasons and timing of a departure.

According to Dr. Alan Seidman’s student retention formula, in order that efforts to increase retention are successful, indicators must result in ACTION, which in turn must be EARLY, INTENSIVE and CONTINUOUS.   Both Seidman and Tinto have determined that it is not the responsibility of one department to monitor and improve retention, but rather the responsibility of the institution as a whole.  A learning analytics platform that allows the administration, faculty and students all to access reports, alerts, and help functions, would fulfill this requirement.  It is worth noting, too, that John Campbell has said that Signals cost approximately $49 per student to implement, and resulted in an 8% increase in retention rates among first year students.  So let’s talk a little about early alert tools.

Early alert tools and the LMS

With the advent of the Internet (20 years old this year!) came the Learning or Course Management System (the LMS or CMS); this of course then gave us the online or distance learner.  We can see the LMS as one source of data.  In one study that compared basic activities related to LMS participation, such as the pages viewed or the number of posts made, and duration of participation, significant differences were found between “withdrawers” and successful completers.  From this we can conclude that time spent and frequency of participation are important indicators for successful online learning.  It is this sort of conclusion that supports early alert tools within the LMS.

You can also see how this is important by looking at the “Academically Adrift” study, which found that 36% of the students that participated in the study were spending five or less hours a week on homework and getting a 3.16 grade average.  While this may sound good to students, it won’t be good enough to succeed in the real world.  The study also found that kids who never studied were more likely to be living with their parents two years out of college.  I think really if we just turn that warning into an early-alert, we’ll get wonderful results.

However, the LMS also presents particular challenges when used as an analytics tool.  Although it reflects an individual’s activity as focused around a system, it does not take into account interactions that occur outside of the LMS, for example, in Facebook, Twitter or on blogs.  Activity that occurs offline also goes unaccounted for, which could include library use, individual time with a tutor or academic advising.  Many of these activities contribute to “integration” as Tinto would define it; important factors in determining retention rates.  Tablets and smartphones do hold, however, huge potential in this respect for their ability to fill in some of those gaps by capturing location and range of activities.

In an effort to measure how different factors may affect a student’s ability to learn, a school in Australia developed the smiley scale.  School children were asked a series of questions relating to various aspects of school life, and asked to indicate their answer by pointing to one of a series of smiley faces that best reflects their level of agreement.  Although used primarily for young children, such a scale could very well be used to gauge the level of satisfaction an online learner has with his or her course, experience of learning online, and with external factors that may affect their mood and therefore their ability to learn.  The simplicity and ease of the scale is quick and unobtrusive.  A scale such as this one could be used to indicate areas of discomfort for an online student (especially in areas where the data is particularly unstructured, such as in the social aspects of a student’s life—peer group interactions and extracurricular activities as highlighted by Tinto’s model), which may act as obstacles to gaining online success.

Getting at the data

So this is my new tattoo.  It took forever!  No, I’m just kidding.  This is actually the tattoo of John Mosley, a 22-year-old who, like many of us I’m sure, wanted an entire Where’s Waldo scene on his back.  Unfortunately, however, he was unable to afford such a lavish tattoo.  But his opportunity came when tattoo artist Rytch Soddy gave him a challenge: Raise £500 for London’s Great Ormond Street Children’s Hospital, and the tattoo is free.  In the end Mosley raised £2000 and got his tattoo, which took 24 hours to complete.  But of course he wasn’t told when Waldo was being drawn in, because where’s the fun in that?

But joking aside, tracking a single, struggling student is like searching for Waldo.  You must look for clues, and rely on your own ability to find them.  And there may very well be more than one Waldo.  Online classes can vary, as you all know, substantially in size; can be taught by multiple instructors; one instructor may have multiple courses in a semester or have head of department responsibilities.  All of this, of course, makes the search even harder.

As with any advance in technology, there are, of course, downsides to using learning analytics.  The most prominent issue facing learning analytics concerns privacy, security and ownership.  There needs to be significant attention paid to what the student is giving up in making those choices about privacy, with moves made into this area by the institutions themselves.  Other concerns, though, include that relying on analytics could make measurement the target, at the expense of the human element.  Similarly, there must be a careful balance between understanding what computers do best and what humans do best, which means that the process can never be entirely automated.  This is also true because learning analytics can, and will, be gamed at some point, by somebody.

In business, analytics is known as Business Intelligence, or BI.  Not only are your Netflix and Amazon accounts measuring your buying patterns in order to make suggestions of future titles you might like, but health care is becoming evidence-based, using collected data to make clinical decisions rather than relying solely on the judgment of the sole physician.  So with this in mind, it’s becoming even more apparent that analytics as a trend is not something that should, or can, be ignored, despite the pushback from some individuals due to the various concerns I touched on.

One reason that it is so important that we get at this data and make sense of it is that we want to be able to predict which students will be successful and which won’t so that we can replicate and encourage the behavior we want and intervene in and improve the behavior we don’t.  But to do this we must first know which factors make a student successful.

Predicting success

Jonah Lehrer wrote a blog post on this, on predicting success.  In this article he notes that for a long time we have assumed that the largest contributor to the success of an individual was largely a case of genetics; inheriting particular talents.  So the Williams sisters have the tennis gene, Bill Gates has the technology gene, and Jackie Chan has the martial arts gene.  But recent research has begun to discount this belief, as our genes do not provide us with specific skills and abilities.  For example, there is no kung-fu movie actor gene.  So what has been found instead?  That talent is really about deliberate practice.  It takes hard work, and lots of it, to become very good at something.

Lehrer does not stop there.  He goes on to note that this realization raises more questions:  What allows someone to practice for a long time?  Why are some people better at deliberate practice?  What factors influence the hard work that has been deemed necessary for success?

Before we answer those questions it’s worth noting that deliberate practice works.  In one example, out of a set of kids that were competing in a spelling bee, the ones that spent hours drilling themselves, versus the kids that were quizzed by others, did better, even though this type of practice is consistently voted as one of the least favorite methods of self-improvement.  (Studying is boring, we all know that.)

So why are some kids better at drilling themselves?  Why do some kids simply just spend so much more time in deliberate practice?  It appears that what’s really important here, is a psychological trait known as grit.

Grit had been measured in this study by using a short survey that featured statements to measure an individual’s consistency of passions (ie “I  have been obsessed with a certain idea or project for a short time but later lost interest”) and consistency of effort (ie “Setbacks don’t discourage me”) using a scale (not unlike our smiley scale).  The study found that those with grit were more single-minded about their goals and were more likely to keep going even when faced with struggle and failure.

So what does this mean for learning analytics?  Well, one thing that Lehrer found was that there is a major inconsistency between how we measure talent and the causes of talent.  Take, for example, the NFL Combine.  Players preform in short bursts—40 yard dashes or catching drills… what-have-you—under conditions in which they are highly-motivated, in order to measure their potential and what they’re capable of on the field.  The problem with this, however, is that the field doesn’t look like the NFL Combine.  Success in the real world depends on sustained performance, rather than maximum performance.  To be successful, NFL players ought to be putting their all into every practice, going home and eating right, studying the playbook on weekends, reviewing game tapes.  These are the daily practices that require grit, and lead to high and consistent performances on a daily basis.

Our standardized testing has faced similar criticisms.  The SAT, the GMAT, even end-of-grade testing in earlier grades have not proved that the students that get the highest scores on the day of the test always go on to be the most successful college students.  Institutions are always under fire for “teaching to the test”, and challenged as to whether scores on standardized tests are true indicators of college success and whether they show a correlation to salary.  Instead, there is a growing recognition that skills such as grit and self-control explain a large proportion of the variation between individuals when it comes to college (and life) success.  Despite having little to do with intelligence as measured by IQ scores, factors like grit are often hugely predictive when it comes to real world performance.  Of course, grit is not something that can be measured on a single afternoon in the NFL Combine or on the LSAT.

But what if we had a series of data taken from learning analytics that showed a student’s grit?  This may be the best way to identify the at-risk student at the earliest possible moment.  This is the sort of possibility held in learning analytics.

And finally, let’s just quickly put what we’ve learned into practice, as I ask you to quickly rate this blog post:

(See what I mean about the possibility of analytics being gamed…….)