Mark Jones, Comments on the Most Recent Academic Planning Survey (19 December 2012)

The new “Survey for Quality Assurance in Online Courses” (reprinted at the end of this post) is a mess.  An academic planning survey should be designed to gather views and information useful for policy recommendations.  This one looks more like it’s  designed to gather ammo against recent efforts to send blended, online, and other course variants to the Curriculum Committees.  At any rate, it is going to be useless for gathering information.

It doesn’t help that it opens with misdefinitions, confusing its own categories from the outset:

For the purpose of this survey, consider the following as definitions of courses:

Traditional: live lectures

Blended: a mixture of online materials and classroom activities

First, there’s more to “traditional” modes of teaching than the lecture–e.g.,  the seminar, the studio session, the lab, the tutorial.  But the real problem here  is the misdefinition of “blended” courses.  Many (perhaps most) “live lecture” courses now mix in “online materials” and would be “blended” by this definition.  But, as our Senate Task Force should certainly know, the definition of “blended” now used in Arts and Science is more rigorous:  according to an Arts and Science memo of May 2011, the “blended model generally includes online materials that replace lecture-style delivery of fundamental content, fewer contact hours, [and] in-class activities that promote active and collaborative learning” (emphasis added).  The current Arts and Science webpage on blended learning says likewise: “In Arts and Science a blended model generally involves moving fundamental knowledge acquisition out of the classroom. . . . fewer contact hours are involved” (emphasis added).  In sum (and in actual practice), “blended” courses are those in which online material is used to displace some but not all of the face-to-face contact with students.

Since the survey begins by sowing confusion about the meaning of “blended,” all of its responses concerning “blended” will be uninterpretable and will have to be thrown out.

Consider, for instance, the first question:

Do you think that there should be a requirement for Faculty level approval (e.g. Curriculum Committee) when a traditional course becomes blended by introducing online activities? (emphasis added).

Given the survey’s own definition of “blended,” I cannot tell whether this means, when an instructor voluntarily adds online activities, or if it means, when the course is officially blended, i.e., when it is advertised as routinely having more online instruction in a trade-off with fewer contact hours.  Some will respond on one assumption, others on the other, and all responses will therefore be ambiguous and useless.

Other questions are messy in other ways.  Consider, for instance, nos. 5 and 6:

5. How often, if at all, do you think traditional (face-to-face) courses should be reviewed by a department?

6. How often, if at all, do you think blended courses should be reviewed by a department?

Since periodic departmental review of courses is not a requirement at present (that I know of), it is not very clear what is envisioned here as course review:   by “courses” do we mean the calendar descriptions, the course materials, or everything, including classroom delivery?  Likewise, what is meant by “by a department”?  Does this mean by the Head or undergraduate chair, by an  undergraduate committee, or what?  (Queens-Blyth course variants are now nominally approved “by departments,” but in practice this often means by an undergraduate chair or a Head.)  The ambiguities mean that people are going to answer these questions on the basis of widely varying assumptions.  Both questions allow the following choices of answer:

  • No need for review once approved
  • Only when a concern has been noted (low USATs, student complaint)
  • Every offering
  • Every 2 years
  • Every 3 years
  • Every 5 years

There is no option for “none of the above” or “other,” and yet the permitted options are gappy.  What if one thinks it should be every 10 or 15 years?  What if one thinks it should be “only when a concern has been noted,” not by students, but by instructors, or by  external reviewers?  Yet the survey does not allow one to skip this question, either:  even if no option seems right, one must choose one in order to continue with the survey.

Then there is question 17:

Would you be in favour of using a Learning Management System (Moodle, D2L, MedTech) that included embedded course design elements encouraging good educational practices (such as defining objectives, linking objectives to assessment, etc.)?

  • Yes, it is a good way to promote standardization among courses
  • Yes, it is a good way to inform students of expectations
  • Yes, it is a good way to improve course design and thus quality
  • No, it infringes on academic freedom
  • Other, please specify…

At least we get an “Other” option this time.  But (a) you cannot tell whether this is asking about “traditional” or about blended and online courses (a crucial difference).  A lot of “traditional” lecture courses already use Moodle, and the rubric is no help:  “Questions 17-19 relate to your efforts to use online resources to improve course quality.”  And (b) “in favour of using” is unclear:  at first it seems  I’m being asked if I would personally like to use an LMS for a given course, but the 1st and 4th response options suggest that I’m being asked if I’m in favour of having this system imposed on everyone (please, no!).  And, of course, the phrasing of the question seems to presume that the the LMS’s “encourage good educational practices.”

There are other problems as well.  But these examples should suffice to  illustrate.  I would like to ask the Task Force to forego highly constraining surveys and concentrate more on open-ended questions.  I would also like to ask that it do this in an open forum so everyone can see what responses the Task Force has been receiving.  The Senate motion by which the present Task Force was struck provided that it should “us[e] the consultation process established by the 2010-2011 APTF” (Senate Minutes, Nov. 2011, p. 8).  The 2010-2011 APTF did not use such surveys.  It used town-hall meetings and a blog-style website on which community comments were open and visible.  It posted queries and propositions and asked for general feedback.  It then posted its drafts and asked for feedback on them.  The community could see what the feedback was and pursue it in dialogue with the APTF and each other.  Such methods are more likely to generate productive response.  They are also more conducive to clarity and trust.

Fortunately, the APTF website is still up and running (scroll down and click on “comment”).  Use it, not the surveys, to talk with your Task Force.

Appendix:  The Survey

Survey for Quality Assurance in Online Courses

As part of our mandate to review the current mechanisms, and to make recommendations for ways to ensure that Queen’s delivers high quality online courses, the Academic Planning Task Force is collecting information and meeting with individuals across the university who are involved in curriculum and course design and quality assurance. For the purpose of this survey, consider the following as definitions of courses:

  • Traditional: live lectures
  • Blended: a mixture of online materials and classroom activities
  • Online: a fully online course

Toward that end, we are hoping that you can provide us with answers to some or all of the following questions.

Questions 1 and 2 deal with the procedures for reviewing courses in relation to incorporation of online elements.

1. Do you think that there should be a requirement for Faculty level approval (e.g. Curriculum Committee) when a traditional course becomes blended by introducing online activities? (select all options that apply)

  • No: such changes are consistent with academic freedom
  • Yes if it changes the learning objectives
  • Yes if it changes how the learning objectives are assessed
  • Yes if it changes contact hours
  • Yes if it changes the types of activities in which students are engaged

2. Do you think that there should be a requirement for Faculty level approval (e.g. Curriculum Committee) when a residential course (face-to-face or blended) forms the basis of a fully online (distance) course? (select all options that apply)

  • No: such changes are consistent with academic freedom
  • yes if it changes the learning objectives
  • yes if it changes how the learning objectives are assessed
  • yes if it changes the types of activities in which students are engaged

Questions 3-12 deal with the procedures for reviewing courses in relation to incorporation of online elements.

3. What mechanisms are in place to ensure the quality of courses within your unit? (select all options that apply)

  • self-report student satisfaction such as USATs
  • peer review by a curriculum or other committee
  • outcome measurement
  • minimum standards that must be met, set either by professional body or research findings
  • not aware of any mechanisms

4. In your Unit, are any additional or different measures applied to courses with an online component?

5. How often, if at all, do you think traditional (face-to-face) courses should be reviewed by a department?

  • No need for review once approved
  • Only when a concern has been noted (low USATs, student complaint)
  • Every offering
  • Every 2 years
  • Every 3 years
  • Every 5 years

6. How often, if at all, do you think blended courses should be reviewed by a department?

  • No need for review once approved
  • Only when a concern has been noted (low USATs, student complaint)
  • Every offering
  • Every 2 years
  • Every 3 years
  • Every 5 years

7. If your answer is different for question 5 and 6, please explain your rationale.

8. If traditional courses were to be reviewed this should be the responsibility of the:

  • Instructor alone
  • Instructor and Centre for Teaching and Learning
  • Department Head or designate
  • Undergraduate Committee in Dept or unit
  • Curriculum Committee of Faculty or School

9. If blended courses were to be reviewed this should be the responsibility of the:

  • Instructor alone
  • Instructor and Centre for Teaching and Learning
  • Department Head or designate
  • Undergraduate Committee in Dept or unit
  • Curriculum Committee of Faculty or School
  • Other, please specify…

10. If your answer is different for question 8 and 9, please explain your rationale.

11. Fully online courses are generally hosted outside departments. In the Faculty of Arts and Sciences, online courses are offered through Continuing and Distance Studies (CDS), with approval from the Department. If fully online courses were to be reviewed, this should be the responsibility of the;

  • Instructor alone
  • Instructor and Centre for Teaching and Learning
  • Department Head or designate
  • Undergraduate Committee in Dept or unit
  • Curriculum Committee of Faculty or School
  • CDS
  • Other, please specify…

12. If your answer is different for question 8 and 11, please explain your rationale.

Questions 13-16 gage your satisfaction with the procedures for course approval and review procedures in your unit for different types of courses.

13. With which unit are you primarily affiliated for teaching purposes?

  • Arts and Science
  • Applied Science
  • Health Sciences
  • Education
  • Law
  • Business
  • Other, please specify…

14. Are you satisfied with the course approval and revision processes for traditional courses?

  • Not familiar with the process
  • Yes, I am satisfied
  • No, I am not satisfied

15. Are you satisfied with the course approval and revision processes for blended courses?

  • Not familiar with the process
  • Yes, I am satisfied
  • No, I am not satisfied

16. Are you satisfied with the course approval and revision processes for online courses?

  • Not familiar with the process
  • Yes, I am satisfied
  • No, I am not satisfied

Questions 17-19 relate to your efforts to use online resources to improve course quality.

17. Would you be in favour of using a Learning Management System (Moodle, D2L, MedTech) that included embedded course design elements encouraging good educational practices (such as defining objectives, linking objectives to assessment, etc.)?

  • Yes, it is a good way to promote standardization among courses
  • Yes, it is a good way to inform students of expectations
  • Yes, it is a good way to improve course design and thus quality
  • No, it infringes on academic freedom
  • Other, please specify…

18. In incorporating online activities, is your MAIN goal to?

  • reduce formal lectures for workload related issues
  • replace lecture-based instruction for pedagogical reasons
  • utilize web-based resources designed for indivdual use
  • promote active or peer-to-peer learning exercises
  • divert content selectively

19. If you answered option “divert content selectively” in the previous question, then answer: When moving content on line, the rationale is: (select up to three options]

  • To put basic background material online, allowing you to focus on more challenging content in class
  • To put more challenging content on line so that students can develop and understanding at their own pace
  • To free up contact time for active, small group learning activities to deepen understanding
  • To put all content on line so that students can recieve individualized and frequent feedback before coming to class to apply what they have learned
  • To increase the content of the course

Current number checked: 0 Minimum: 0 | Maximum: 3

The final questions (20-23) ask if you have had assistance that helped you improve a course when incorporating online elements.

20. If you have developed online resources as part of a traditional course, what types of suport, if any, have you been given? (select all that apply)

  • Informal support from a colleague
  • Centre for Teaching and Learning
  • Other instructional design or development personnel
  • Faculty Office
  • IT Services
  • Library
  • Copyright Office
  • Educational resources (List serves, webinars, conferences) outlside of Queen’s
  • Other, please specify…

21. Do you think that the support available helped improve the quality of the course?

  • not really
  • marginally
  • significantly
  • the support was vital

22. If you have developed a new online course what types of suport, if any, have you been given? (select all that apply)

  • Informal support from a colleague
  • Centre for Teaching and Learning
  • Other instructional design or development personnel
  • Faculty Office
  • IT Services
  • Library
  • Copyright Office
  • Educational resources (List serves, webinars, conferences) outside of Queen’s
  • Other, please specify…

23. Do you think that the support available helped improve the quality of the course?

  • not really
  • marginally
  • significantly
  • the support was vital
This entry was posted in Academic Planning Task Force, Virtualization / Online learning. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s