LOOM^TM

Is all the fuss about learning analytics just hype?

As academic and learning analytics begin to emerge, they face not only huge expectations, but also implications, from all fronts.  We certainly have many expectations for Weaver™, into which more features we will be incorporated into every new release (Weaver™ Alpha has just been released into the wild).  Of course, as with any innovation, the learning analytics naysayers do exist, which is one reason why we are taking a sure but methodically sound approach to the development process.

At the core of academic and learning analytics is data mining.  In her 2006 paper on data mining, Lisa Bowen Ayre noted that the term refers to “the application of special algorithms in a process built upon sound principles from numerous disciplines including statistics, artificial intelligence, machine learning, database science, and information retrieval.”  Colleges collect huge amount of data on their students, and with the advent of the learning management system (LMS), the amount and types of data has increased.  Typically, however, most colleges use the information they glean about students solely for marketing efforts and in order to comply with accreditation requirements.  But couldn’t this same information be used for the direct benefit of the students?  And couldn’t the benefits to the institutions be broadened as well?  Surely academic and learning analytics would have some vastly positive effects on making that information available to students, to aid in increasing transparency, and perhaps even taking a step towards making it easier to establish actual gains in knowledge, which so far has been notoriously difficult to measure.

But the act of moving towards transparency and an increased availability of data is not without its concerns.  David Jones calls learning analytics a “two-edged sword,” pointing to his early days teaching as a university academic.  He contrasts his methods (which included “a lot of very different things…Not all of them worked as I planned, but they all helped something interesting grow”) with those of a notable few among his colleagues, who were able to get away with poor methods of educating students due to the opaque nature of the system at that time.  Although those said colleagues slipped through as long as deadlines were met and certain grade distributions were produced, it was those very same standards that allowed Jones the freedom to use his own innovative techniques.  Could learning analytics, while spawning incredible benefits across educational institutions, also make allowances for similar loopholes on an even larger scale?

That seems to be one of several arguments made by cynics, who seem to arrive at the conclusion that all this fuss about learning analytics, due to the complexities involved, is just hype.  Of course, because critiques of learning analytics are new, any form of criticism remains few and far between.  George Siemens responded to the void left by what he felt was a lack of sensible debate (“concepts need strong critiques”) by posting a discussion forum to address the situation.  Most who have added to the thread here raise concerns with the end goal of eventually operating learning analytics most effectively and responsibly, and comments in which there is the sense of real push-back have been notably rare.  Tackling the collection and use of data, Viplav Baxi notes that fundamentally “there are many arguments for or against statistical analyses and other forms of analytics (such as those generated by “intelligent” systems). The arguments address generalizability (do the analytics imply that we can take general actions and predict outcomes), appropriateness (are the analytics appropriate to generate for the domain under consideration), accuracy (did we have enough information, did we choose the right sample), interpretation (can we rely on automated analytics or do we need manual intervention or both), bias (analytics used to support an underlying set of beliefs), method (were the methods and assumptions correct), predictive power (can the analytics give us sufficient predictive power) and substantiation (are the analytics supported by other empirical evidences).”  But he also comments on the presence of inadequate analytics that are already prone to a certain amount of reductionism that can do more harm than good, and the benefits the emergence of learning and knowledge analytics might do in education.  There is also the possibility that we may begin to look only at cold, hard data for clear facts and patterns, exposing us to the potential that we miscalculate the complexities that those numbers hide.  We are, after all, still looking at human behavior.  This miscalculation might appear as we begin to question the extent of a computer’s ability to read human subtleties, as when Murray Richmond asked, “what does the number of ‘interactions’ tell us about the ‘quality or value’ of a discussion?” A related matter is that we run the risk of reducing students and faculty (those who will be “measured”) to the status of commodities in relation to the larger entities of their respective institutions.  Yet another common concern is that of ethics; learning analytics must still take into account the rights of the learner.

But every argument against the development and adoption of academic and learning analytics leads to a single conclusion—that whatever stage learning and analytics reach, they will always have to be carefully (humanly) managed.  This, I think, has greater implications for how institutions will implement and run academic and learning analytics, rather than if they implement and run them.  The questions raised in Siemen’s forum will need to be tackled by institutions, and they will be better off knowing where they stand on these issues, legally as well as ethically, before they encounter them.  This alone creates space for entrepreneurship; although many institutions are already thinking seriously about learning analytics and the frontrunners have even begun to use them in making decisions regarding policy, many more institutions will find themselves trying to catch up.  Businesses like us are already beginning to recognize the opportunities here, as learning analytics will require the software and services necessary to effectively manage those concerns I’ve already discussed or implied.  So although there are invariably risks associated with managing schools’ data, especially as each has its own idiosyncrasies and curricula, huge opportunities exist also: the ability to consolidate the lifecycle of each student, from enrollment through to graduation and even beyond (as the potential exists to collect subsequent career and salary data for alumni that is otherwise not gathered without undergoing massive reports), to improve the quality of courses by comparing student’s expectations before the start of a course to their actual results and own perceptions of what they achieved, and to align trends appearing throughout the educational sector to what and how courses are delivered, to name just a few.  Entrepreneurial efforts will aid in innovation in learning analytics and the speed at which they’re implemented.  We will undoubtedly see different approaches begin to emerge, which will keep the use of learning analytics in check and make the business of it all far more robust.  So rather than being lead to believe that analytics is just the latest trend, a passing fad that will never really take off, we believe that learning analytics face challenges that will be overcome through the combination of rich academic discussion and entrepreneurial endeavor that is already coming to light.

What are your concerns regarding learning and academic analytics?  Do you see these as being manageable, or of destroying any possibility of making learning analytics a viable option across the educational sector?

Harriet May hmay@loomlearning.com