Mark Jones, Data Check: Recent “Evidence” in Support of Blended and Online Learning (27 January 2013)

The Arts and Science webpage for “Blended Learning”[1] claims that

A 2010 meta-analysis of more than a thousand empirical research studies showed that learning outcomes for students taking blended courses were significantly better than those of students receiving purely face-to-face instruction. The mean effect size in studies comparing blended with face-to-face instruction was +0.35, p < .001.

Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning. Center for Technology in Learning, U.S. Department of Education.

This claim is misleading in several ways.  First, the relevant data sample used by the 2010 meta-analysis is not “more than a thousand empirical research studies,” but, more precisely, twenty-three:

The computerized searches of online databases and citations in prior meta-analyses of distance learning as well as a manual search of the last three years of key journals returned 1,132 abstracts. In two stages of screening of the abstracts and full texts of the articles, 176 online learning research studies published between 1996 and 2008 were identified that used an experimental or quasi-experimental design and objectively measured student learning outcomes. Of these 176 studies, 99 had at least one contrast between an included online or blended learning condition and face-to-face (offline) instruction that potentially could be used in the quantitative meta-analysis. [. . .] Of the 99 studies comparing online and face-to-face conditions, 45 provided sufficient data to compute or estimate 50 independent effect sizes (some studies included more than one effect). [. . .] Most of the articles containing the 50 effects in the meta-analysis were published in 2004 or more recently.  The split between studies of purely online learning and those contrasting blended online/face-to-face conditions against face-to-face instruction was fairly even, with 27 effects in the first category and 23 in the second.[2]

Second, the usage of “blended” in the meta-analysis differs so significantly from the usage in Arts and Science at Queen’s as to make the conclusions of the meta-analysis inapplicable.  The meta-analysis uses “blended” to describe any combination of “online learning components . . . with face-to face instruction to provide learning enhancement.”[3]  Given this definition, it is scarcely surprising to find that “blending” enhances learning.  At Queen’s, however, “blended” means that online components have been added to replace and reduce contact hours:  as the Arts and Science webpage puts it, “While blended courses have the same number of student learning hours as traditional lecture courses, the nature of the learning hours is different and fewer contact hours are involved.”  Simply adding online resources will predictably “enhance” learning outcomes, but the same may not be true where contact hours are subtracted.

Third, the Queen’s webpage simply ignores the significant “caveats” made by the U.S. Dept. of Education study:

However, several caveats are in order: Despite what appears to be strong support for blended learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium. In many of the studies showing an advantage for blended learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages. (p. xviii)

The authors also warn that  “many of the studies suffered from weaknesses such as small sample sizes; failure to report retention rates for students in the conditions being contrasted; and, in many cases, potential bias stemming from the authors’ dual roles as experimenters and instructors” (p. xviii).

In sum, the claim that “more than a thousand empirical research studies showed that learning outcomes for students taking blended courses were significantly better than those of students receiving purely face-to-face instruction” is a misleading use of this source.

*

Appendix:  Arts and Science Webpage on Blended Learning, as of 27 January 2013:

Blended learning integrates in-class, face-to-face learning with online learning in a purposeful, thoughtful and complementary way to enhance student engagement. (Garrison and Vaughn 2008).

Blended Learning for Students [PDF] describes blended learning from a student’s perspective.

Improved student engagement is achieved by focusing on in-class interaction to promote active and collaborative learning, and minimizing or eliminating the passive transmission of information.

In Arts and Science a blended model generally involves moving fundamental knowledge acquisition out of the classroom—using interactive online materials to deliver enriched content, to guide students through the textbook, to verify comprehension—and devoting classroom time to applying, integrating and synthesizing the knowledge. This model is sometimes referred to as the “flipped classroom.” While blended courses have the same number of student learning hours as traditional lecture courses, the nature of the learning hours is different and fewer contact hours are involved.

In addition to enhancing student engagement, several benefits are associated with blended models. Flexibility is increased for students through the learner-centred approach, which allows students to progress at their own pace online and to review complex concepts as needed. For instructors, the blended approach frees up class time and offers opportunities to integrate teaching and research—to explore a research perspective, discuss case studies, or to pose a provocative question.

Redesigning a course as a blended model is not seen as a cost-saving model, but can be an effective way of increasing enrolment in high-demand courses using existing resources.

Blended models are particularly effective for large classes, where it is more challenging to engage students. The Faculty’s first innovations in blended learning were initiated in 2011 by instructors in two high-enrolment Arts and Science courses: Human Geography (GPHY 101) and Principles of Psychology (PSYC 100).

The Faculty is promoting the development of blended learning models for large introductory classes through a strategic Course Redesign project. Starting in 2012/13, Introduction to Sociology (SOCY 122), Women, Gender, Difference (GNDS 120) and Ancient Humour (CLST 205) are being delivered as blended courses. In an independent initiative, the Department of Film and Media is offering Film, Culture and Communication (FILM 110) in a blended format as well.

The next phase of the Course Redesign project (2013/14) includes the following courses: Differential and Integral Calculus (MATH 121), Introductory Biology of Cells (BIOL 102), General Chemistry (CHEM 112), Gender, Race and Popular Culture (GNDS 125) and Theatre in the Age of Film and Television (DRAM 205).

All blended courses, including those initiated outside of the Course Redesign project, are being assessed to evaluate their effectiveness in terms of student engagement and learning; the results of these assessments will be reported to the university community.

Read more about the Faculty’s blended learning initiative.

View Resources for blended learning.

A 2010 meta-analysis of more than a thousand empirical research studies showed that learning outcomes for students taking blended courses were significantly better than those of students receiving purely face-to-face instruction. The mean effect size in studies comparing blended with face-to-face instruction was +0.35, p < .001.

Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning. Center for Technology in Learning, U.S. Department of Education.

Notes

[1] Queen’s Faculty of Arts and Science.  “Blended Learning.” http://www.queensu.ca/artsci/academics/teaching-and-learning/blended-learning, 27 Jan. 2013; reproduced in the appendix to this post.

[2] Barbara Means et al., Evaluation of Evidence-Based Practices in Online Learning: a Meta-Analysis and Review of Online Learning Studies (U.S. Department of Education, rev. ed. Sept. 2010) (http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf), pp. xii-xiii, boldface added.

[3] See Means et al., p. 9:  “this review distinguishes between two purposes for online learning:

  • Learning conducted totally online as a substitute or alternative to face-to-face learning
  • Online learning components that are combined or blended (sometimes called “hybrid”) with face-to-face instruction to provide learning enhancement.”
Advertisements
This entry was posted in Data Checks, Virtualization / Online learning. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s