Science of teaching and learning

Exploring the impact of explicitly stated learning objectives on student outcomes in computer sciences courses

B. M. Stoesz, C. Penner (Computer Science), & R. Guderian (Computer Science) 

Learning objectives (LOs) are defined as declarative statements that identify the knowledge and skills that students should learn by the end of a lesson, module, or course (Sana et al., 2020), and are an important aspect of a student-centred approach to pedagogy. Well-written LOs guide instructors and students as they develop structured courses, and engage meaningfully with course material, respectively. Explicitly stated LOs are associated with higher scores on assessments and students are more satisfied with courses (Smith & Tanner, 2010), but this result may depend on the degree of control set within the studies.  

Using naturalistic research methodology (study of individuals or groups in their natural setting with minimal interference), the influence of LOs on outcomes will be examined in two different computer science courses. Specifically, this study will test whether the presence (vs absence) of explicitly stated LOs increased student perceptions of their learning and/or grades on assignments, exams, and the course overall. Depending on the course, LOs appeared within a learning management system (LMS; D2L Brightspace) as a content item before one or more course components allowing user-activity for LOs (e.g., content access dates, times, and durations) to be analyzed with respect to course artifacts (e.g., achievement on assignments, exams, and the final grade of the course).  

Conventional methods of educational research assume, in part, that an ideal context that is unchanging exists, and that the results obtained from the ideal context can be applied to another ideal context with the same effect. A different approach was taken in this study as the postsecondary educational environment is not ideal in reality – students’ degree of engagement in their courses and instructors’ application of instructional strategies differ. Therefore, if previous findings are robust, then this study is expected to show that the presence of LOs have a positive impact on learning. In addition, instructors can be more confident that even small improvements to their course development will better support their students’ learning journeys.  

Teaching and learning research at the University of Manitoba: A SoTL rose by another name? 

J. Cape 

The purpose of the proposed study is to conduct an environmental scan of research published by UM faculty over the past ten years that investigates an aspect of teaching and learning, and that would fall under the “big tent” (Chick, 2014) of SoTL research. While other kinds of teaching and learning research—formal and informal—certainly occur at UM without being disseminated in peer-reviewed contexts, publication suggests a level of value placed on the research. This work seeks to catalog the UM authors undertaking this work, their disciplines, the publications where the work is disseminated, and to examine the questions investigated. 

Globalization, universalization, and harmonization of the scholarship of teaching and learning: Lexicon of common SoTL-based terms 

H. Malyk 

Varied definitions and interpretations of SoTL-based concepts challenge the meaning of SoTL and its discourse (McKinney, 2002) and constrain SoTL engagement rather than enhance original inquiry and authentic teaching practices. Terminological inconsistency within the broad field of SoTL is an apparent barrier to its capacity building, as it pushes SoTL to the periphery of pedagogical research and makes it the “road less traveled” (Simmons, 2017) and “a borderland” in which scholars from different disciplinary cultures come to “trade their wares” (Chick, 2013; Huber & Morreale, 2002), transforming it into a vague undervalued “big tent” (Chick, 2013; Hewson & Easton, 2022) “under which a wide variety of work can thrive” (Huber & Hutchings, 2005, p. 4). This study attempts to address this inconsistency by exploring and analyzing the nature and substance of SoTL concepts, their definitions and terminological applications to facilitate a requisite cultural shift for enhanced educational practices within 4 M Framework of SoTL Impact (Simmons, 2020) and “dispel the fog of learning through SoTL” (Tagg, 2010). 

Impact of self-reflection on professional growth and identity development of educational developers in higher education 

L. A. Doan & B. M. Stoesz

Self-reflection on teaching is a critical component of professional development for instructors in higher education (Szűcs, 2018). A reflective method that includes creative (fictionalizing some biographical events), expressive (writing about traumatic events and the emotions experienced), and reflective (reflecting on one’s life, including questions regarding one’s identity and beliefs) writing contributes positively to the development of professional identity (Lengelle et al., 2016). The reflective practices by teachers in K-12 education and community colleges have received much more attention in the research literature than those of university educators ​(Jaramillo Cherrez & Jin, 2020),​ or educational developers. Given the research on the benefits of self-reflective practice, and that limited research has been conducted with educational developers, our primary goal is to examine whether educational developers in higher education participate in self-reflection and how this practice may or may not impact their professional growth and identity development. 

Immersive virtual reality training for postsecondary educators 

B. M. Stoesz, H. Morris (Student Advocacy), B. Usick (Student Engagement & Success), N. Harder (Nursing), D. Sutherland (Graduate Studies), M. Menzies (Human Rights & Conflict Management), & D. Gerhard (Computer Science) with support from M. Quesnel, J. Plohman, I. Arowolo, W. Snow, and A. Budhiraja (student)

Training postsecondary educators to implement pedagogies and deal with various teaching challenges more effectively supports the goal of striving for teaching excellence at the University of Manitoba. A novel way to prepare faculty members, sessional instructors, and graduate students for success in teaching involves the use of extended/virtual reality (XR/VR). Designing a teaching and learning XR/VR environment would enable educators to develop teaching skills in a safe and judgement-free setting prior to using their newly acquired skills in the classroom and/or laboratory. In this proposal, we outline an innovative three-year postsecondary educator training strategy to support the development of skills to enhance teaching confidence and competency and facilitate more positive interactions with students. Our goals throughout the initiative include developing a micro-credential to enhance teaching skills that may help educators deal with challenging teaching issues, gathering and analyzing qualitative and quantitative data to determine whether this experiential learning/training experience is effective, revising the micro-credential based on feedback from educators (and other UM stakeholders), and creating additional micro-credentials based on identified training needs. By working with multiple partners across the University of Manitoba community, this initiative is more likely to meet the needs for postsecondary educator training and enhancing the student learning experience. 

Academic integrity attitudes and behaviors: A survey of students at the University of Manitoba

B. M. Stoesz & J. Plohman in partnership with D. Rettinger (University of Tulsa), J. Stephens (University of New Zealand), & G. Curtis (University of Western Australia) 

The University of Manitoba has been leading the way in Canada in the area of academic integrity. In past decades, the University of Manitoba has participated in the McCabe Academic Integrity Survey, developed by the late Don McCabe, founder of the International Center for Academic Integrity (ICAI). The data from the surveys was used to help guide the development of various resources for promoting academic integrity and preventing misconduct. Fast forward to 2023, the survey has been re-developed to reflect changes in society, culture, and technology, and is now an online (rather than paper-based) survey. The purpose of this study is to understand University of Manitoba students’ perceptions, beliefs and behaviors related to academic integrity. This data can be used to gather or generate resources to promote academic integrity. The data can also be used as a benchmark to compare changes in student attitudes and behaviors over time with future points of data collection. 

Changing cultures of integrity at Queen’s University and the University of Manitoba: The analysis of historical documentation using advanced text-mining techniques 

L. Scholle-Collen (Queen’s University) & B. M. Stoesz with support from L. Doan 

Queen’s University and the University of Manitoba, akin to many Canadian universities, trace their origins to religious foundations with a focus on ministry education. Over the years, these institutions have grown substantially and have gradually distanced themselves from their initial religious affiliations. Today, these universities boast a diverse range of academic programs spanning law, medicine, arts, sciences, engineering, business, and more. Examining these historical shifts can reveal conflicts and tensions related to academic integrity that have persisted within Canadian universities over the last several decades. Gaining a comprehensive understanding of current integrity culture in higher education from past events is imperative (see Eaton & Hughes, 2022; Gallant, 2007). Some researchers have delved into academic integrity policies at Canadian universities (Stoesz & Eaton, 2022; Stoesz et al., 2019) and quality assurance processes (Thacker & McKenzie, 2022) to uncover the prevailing culture of academic integrity within higher education. In this study, we propose a different approach to unearthing the transformations in the integrity culture within two Canadian universities. We are conducting a comprehensive analysis of archived university senate meeting minutes from Queen's University and the University of Manitoba using advanced text-mining techniques. This involves using natural language processing (NLP) and data mining algorithms to systematically analyze the content contained within the archived documents. Through this approach, we expect to uncover significant themes and concepts and the ebb and flow of integrity-culture within two Canadian universities as they evolved from their religious foundations to become diverse, multifaceted institutions. 

How much can we trust online survey data? The degree of fraudulent responding in and methods for fraud detection from a population-based online survey on barriers and persistence in adult job-related training

W. M. Snow & A. E. De Jaeger 

Research in many disciplines, including higher education, relies heavily on self-report surveys. Increasingly, survey administration is moving away from in-person to online and digital environments, the advantages of which can include increased anonymity and convenience for participants as well as reduced research costs. Collecting survey data in digital and online environments, however, increases the capacity for fraudulent participation in research. We investigated the degree of suspected invalid responding in an online survey on barriers to entry and persistence in adult job-related training and education among Manitobans and extended upon an algorithm designed to facilitate detection of suspected fraudulent responding. This research will have important implications for designing more effective, rigorous methods of data collection to reduce potential illegitimate participation in online survey research and to ease the burden of detection of such instances for researchers going forward. We also argue for more standardized reporting in online survey research studies on the extent of suspected fraudulent responding and methods used to define data as fraudulent to increase the validity of findings and foster trust in survey data collected online. Such measures are essential to ensure data integrity in online survey research in education and beyond. 

Academic integrity and accommodations: Challenging the misconceptions

Stoesz, B. M., Attas, R., & Sanni-Anibire, H. (2024). Academic integrity and accommodations: Challenging the misconceptions. In A. Chaudhuri (Ed.), Discipline-based approaches to academic integrity. Victoria, BC: BCcampus. Retrieved from https://pressbooks.bccampus.ca/ubcacademicintegrity/part/academic-integrity-and-accommodations-challenging-the-misconceptions/ 

Within the topic of academic integrity, accommodations for students experiencing learning challenges have largely been ignored in research and practice. Instructors’ lack of knowledge and accompanying skills may result in misconceptions about the intersection between academic integrity and the learning needs of students; in particular, the misconception that accommodations are “situationally sanctioned cheating” that allow students to “gain an unfair advantage” rather than facilitating learning and success must be addressed. This chapter explores questions that students, instructors, and other postsecondary community members may have about accessibility and equity, and how these considerations show respect, fairness, trust, responsibility, honesty, and courage and contribute to building cultures of academic integrity.  

Studying the effectiveness and usability of an online authorship verification tool in the laboratory and in the classroom 

B. M. Stoesz, M. Quesnel, & R. Guderian (Department of Computer Science) 

Contract cheating is the outsourcing of academic work to a third party with an estimated 23% of postsecondary students engaging in this behaviour (Curtis et al., 2021). An approach to reduce or detect contract cheating that has not received much attention is to quiz students to verify their assignment authorship. The premise is that if students engaged in, completed, and submitted their own academic work, their memory of the details of their work should be better than if they had outsourced the work. The goal of this study is to examine the effectiveness of a tool designed to quiz students on their submitted assignments to verify their authorship and explore whether it is a practical, reliable, and valid approach to deter or detect contract cheating. 

Bias in student ratings of instruction: A systematic review of research from 2012 to 2021

Stoesz, B., De Jaeger, A., Quesnel, M., Bhojwani, D. & Los, R. (2022). Bias in student ratings of instruction: A systematic review of research from 2012 to 2021. Canadian Journal of Educational Administration and Policy/Revue canadienne en administration et politique de l’éducation, (201), 39–62. https://doi.org/10.7202/1095482ar 

Student ratings of instruction (SRI) are commonly used to evaluate courses and teaching in higher education. Much debate about their validity in evaluating teaching exists, which is due to concerns of bias by factors unrelated to teaching quality (Spooren et al., 2013). Our objective was to identify peer-reviewed original research published in English from January 1, 2012, to March 10, 2021, on potential sources of bias in SRIs. Our systematic review of 63 articles demonstrated strong support for the continued existence of gender bias, favoring male instructors and bias against faculty with minority ethnic and cultural backgrounds. These and other biases must be considered when implementing SRIs and reviewing results. Critical practices for reducing bias when using SRIs include implementing bias awareness training and avoiding use of SRIs as a singular measure of teaching quality when making decisions for teaching development or hiring and promotion. 

Student perceptions of the visual design of learning management systems

Stoesz, B. M., & Niknam, M. (2023). Student perceptions of the visual design of learning management systems. Canadian Journal of Learning and Technology, 48(3), 1-22. https://cjlt.ca/index.php/cjlt/article/view/28154/20689  

Approximately 2,034,957 Canadians were enrolled in courses at postsecondary institutions in 2015/2016 (Statistics Canada, 2017), and an estimated 610,487 (30%) are completing online courses in any given year (Bates, 2015). This project examined students’ visual perceptions of the learning management systems (LMS) they use and how it influences their learning experiences. Findings contribute to our knowledge of LMS interface design.  

Faculty perspectives of academic integrity during COVID-19: A mixed methods study of four Canadian universities

Eaton, S. E., Stoesz, B. M., Crossman, K., Garwood, K., & McKenzie, A. (2023). Faculty perspectives of academic integrity during COVID-19: A mixed methods study of four Canadian universities. Canadian Journal of Higher Education, 52(3), 42–58. https://doi.org/10.47678/cjhe.vi0.189783 

Researchers from four Canadian universities and an industry partner are collaborating to better understand faculty perceptions and needs related to academic integrity in Canadian higher education. The survey designed for this project was informed by previous academic integrity research (e.g. McCabe, 1993), and the findings are expected to lead to the development of specific supports for faculty in Canadian higher education institutions to promote academic integrity. 

Using Google analytics to measure engagement with a teaching and learning centre during COVID-19

Stoesz, B. M. (2022). Using Google analytics to measure engagement with a teaching and learning centre during COVID-19. Journal on Centers for Teaching and Learning, 14, 39-57. https://openjournal.lib.miamioh.edu/index.php/jctl/article/view/245 

Postsecondary educators and students were among the first affected by COVID-19 pandemic safety protocols and were required to transition quickly from face-to-face to unfamiliar remote teaching and learning environments. To support this transition, support staff at teaching and learning centres (TLCs) also pivoted their support strategies and developed and delivered more online resources and virtual professional development workshops. The purpose of this study was to examine whether the demands for online and remote teaching and learning support was evident in the Google Analytics data of the TLC webpages of a research-intensive Canadian university. To facilitate interpretation of observed trends, we aligned the timing of messages emailed to faculty and new resource launches on the TLC webpages to the analytics data. Results confirmed that TLC webpages are vital sources of information for faculty development, and targeted communications increased faculty engagement with teaching resources and professional development opportunities. As pandemic safety protocols ease and educators and students return to their postsecondary campuses, they will face new challenges. In response, TLCs must continue to monitor faculty engagement and their changing support needs, and they must continue to adjust approaches to offering information and professional development opportunities as necessary. The examination of website analytics data is just one measure that can support evidence-informed decision making for this purpose. 

Tensions and partnerships: Understanding research ethics in the scholarship of teaching and learning (SoTL)

De Jaeger, A. E., Stoesz, B. M., Doan, L. A. (2022). Tensions and partnerships: Understanding research ethics in the scholarship of teaching and learning (SoTL). In: Fedoruk, L.M. (eds) Ethics and the Scholarship of Teaching and Learning. Ethics and Integrity in Educational Contexts, vol 2. Springer, Cham. https://doi.org/10.1007/978-3-031-11810-4_4 

The topic of ethics has received much attention within the SoTL community; however, limited studies have examined both REB reviewers’ and SoTL researchers’ perspectives on partnerships when preparing or reviewing SoTL research. In this chapter, we describe the findings from our study that aimed to address this gap by gathering researchers’ and REB reviewers’ experiences with and perceptions of SoTL research and situate findings within the micro-meso-macro-mega (4M) framework. 

International students’ knowledge and emotions related to academic integrity at Canadian postsecondary institutions

Sanni-Anibire, H., Stoesz, B. M., Gervais, L., & Vogt, L. (2021). International students’ knowledge and emotions related to academic integrity at Canadian postsecondary institutions. International Journal of Educational Integrity, 17, 21 (2021). https://doi.org/10.1007/s40979-021-00088-4 

This study investigated the knowledge of academic integrity and associated emotions of a small sample of international students studying at Canadian postsecondary institutions (n = 60) using survey methodology. Depending on the survey item, 25–60 participants provided responses. Many respondents appeared knowledgeable about academic integrity and misconduct and reported that expectations in their home countries and in Canada were similar. There was, however, disagreement on the concept of duplicate submission/self-plagiarism, indicating an important gap in educating students about specific aspects of policy in postsecondary education in Canada. In addition, more than a third of respondents provided neutral responses to a situation involving contract cheating, suggesting a lack of certainty in how to respond when witnessing peers’ engagement in outsourcing academic work. Many respondents reported feeling confident upon reading the academic integrity and misconduct policies of their Canadian postsecondary institution, although nearly one third indicated feeling fearful, anxious, and/or confused. These negative feelings were associated with reduced knowledge of academic integrity and misconduct. Future research should further explore the experiences and emotions of international students related to academic integrity and misconduct to better understand the successes and challenges that they face in their postsecondary studies in Canada. Our findings have important implications for the delivery of academic integrity education, enhancing supports and resources, and refining academic integrity policies and procedures to improve the experience of students who come from abroad to study in Canada. 

Development of the Online and Blended Teaching Readiness Assessment (OBTRA)

Los, R., De Jaeger, A. & Stoesz, B. M. (2021). Development of the Online and Blended Teaching Readiness Assessment (OBTRA). Frontiers in Education, 6(673594). https://doi.org/10.3389/feduc.2021.673594 

Readiness for teaching online can be defined as having the competencies required to deliver a high-quality online course, where readiness can be equated with the qualities exemplified by the ideal online instructor. This project seeks to determine the characteristics that exemplify effective online instructors and create a first-of-its-kind online teaching readiness assessment with psychometric properties. The assessment will provide instructors with immediate feedback as to the areas they excel and can improve upon, assist in identifying faculty training/development opportunities, determine strengths and areas in need of enhancement in instructors’ online teaching skillsets, and further understand the barriers and attitudes towards online teaching. 

Defining the visual complexity of learning management systems using image metrics and subjective ratings

Stoesz, B. M., Niknam, M., & Sutton, J. (2020). Defining the visual complexity of learning management systems using image metrics and subjective ratings. Canadian Journal of Learning and Technology, 46(2), 1-21. https://www.cjlt.ca/index.php/cjlt/article/view/27899/20497 

Delivering courses online can be effective but learning via learning management systems (LMS) may be attenuated by poor visual design. This project aimed to determine if existing models of complexity for webpages can be extended to LMS and explore how individual differences influence ratings of complexity. Findings inform the development of online learning environments.  

If you would like to discuss or become involved in a teaching and learning research project at The Centre for the Advancement of Teaching and Learning, please contact the STL team.