The Use of AI-Detection Tools in the Assessment of Student Work

May 6, 2023

People have been asking if they should be using detection tools to identify text written by ChatGPT or other artificial intelligence writing apps. Just this week I was a panelist in a session on “AI and You: Ethics, Equity, and Accessibility”, part of ETMOOC 2.0. Alec Couros asked what I was seeing across Canada in terms of universities using artificial intelligence detection in misconduct cases.

The first thing I shared was the University of British Columbia web page stating that the university was not enabling Turnitin’s AI-detection feature. UBC is one of the few universities in Canada that subscribes to Turnitin.

The Univeristy of British Columbia declares the university is not enabling Turnitin’s AI-detection feature.

Turnitin’s rollout of AI detection earlier this year was widely contested and I won’t go into that here. What I will say is that whether AI detection is a new feature embedded into existing product lines or a standalone product, there is little actual scientific evidence to show that AI-generated text can be effectively detected (see Sadasivan et al., 2023). In a TechCrunch article, Open AI, the company that developed ChatGPT, talked about its own detection tool, noting that its success rate was around 26%

Key message: Tools to detect text written by artificial intelligence aren’t really reliable or effective. It would be wise to be skeptical of any marketing claims to the contrary.

There are news reports about students being falsely accused of misconduct when the results of AI writing detection tools were used as evidence. See news stories here and here, for example. 

There have been few studies done on the impact of a false accusation of student academic misconduct, but if we turn to the literature on false accusations in criminal offences, there is evidence showing that false accusations can result in reputation damage, self-stigma, depression, anxiety, PTSD, sleep problems, social isolation, and strained relationships, among other outcomes. Falsely accusing students of academic misconduct can be devastating, including dying by suicide as a result. You can read some stories about students dying by suicide after false allegations of academic cheating in the United States and in India. Of course, stories about student suicide are rarely discussed in the media, for a variety of reasons. The point here is that false accusations of students for academic cheating can have a negative impact on their mental and physical health.

Key message: False accusations of academic misconduct can be devastating for students.

Although reporting allegations of misconduct remains a responsibility of educators, having fully developed (and mandatory) case management and investigation systems is imperative. Decisions about whether misconduct has occurred should be made carefully and thoughtfully, using due process that follows established policies.

It is worth noting that AI-generated text can be revised and edited such that the end product is neither fully written by AI, nor fully written by a human. At our university, the use of technology to detect possible misconduct may not be used deceptively or covertly. For example, we do not have an institutional license to any text-matching software. Individual professors can get a subscription if they wish, but the use of detection tools should be declared in the course syllabus. If detection tools are used post facto, it can be considered a deception on the part of the professor because the students were not made aware of the technology prior to handing in their assessment. 

Key message: Students can appeal any misconduct case brought forward with the use of deceptive or undisclosed assessment tools or technology (and quite frankly, they would probably win the appeal).

If we expect students to be transparent about their use of tools, then it is up to educators and administrators also to be transparent about their use of technology prior to assessment and not afterwards. A technology arms race in the name of integrity is antithetical to teaching and learning ethically and can perpetuate antagonistic and adversarial relationships between educators and students.

Ethical Principles for Detecting AI-Generated Text in Student Work

Let me be perfectly clear: I am not at all a fan of using detection tools to identify possible cases of academic misconduct. But, if you insist on using detection tools, for heaven’s sake, be transparent and open about your use of them.

Here is an infographic you are welcome to use and share: Infographic: “Ethical Principles for Detecting AI-Generated Text in Student Work” (Creative Commons License: Attribution-NonCommercial-ShareAlike 4.0 International). The text inside the infographic is written out in full with some additional details below.

Here is some basic guidance:

Check your Institutional Policies First

Before you use any detection tools on student work, ensure that the use of such tools is permitted according to your school’s academic integrity policy. If your school does not have such a policy or if the use of detection tools is not mentioned in the policy, that does not automatically mean that you have the right to use such tools covertly. Checking the institutional policies and regulations is a first step, but it is not the only step in applying the use of technology ethically in assessment of student work.

Check with Your Department Head

Whether the person’s title is department head, chair, headmaster/headmistress, principal, or something else, there is likely someone in your department, faculty or school whose job it is to oversee the curriculum and/or matters relating to student conduct. Before you go rogue using detection tools to catch students cheating, ask the person to whom you report if they object to the use of such tools. If they object, then do not go behind their back and use detection tools anyway. Even if they agree, then it is still important to use such tools in a transparent and open way, as outlined in the next two recommendations.

Include a Statement about the Use of Detection Tools in Your Course Syllabus

Include a clear written statement in your course syllabus that outlines in plain language exactly which tools will be used in the assessment of student work. A failure to inform students in writing about the use of detection tools before they are used could constitute unethical assessment or even entrapment. Detection tools should not be used covertly. Their use should be openly and transparently declared to students in writing before any assessment or grading begins.

Of course, having a written statement in a course syllabus does not absolve educators of their responsibility to have open and honest conversations with students, which is why the next point is included.

Talk to Students about Your Use of Tools or Apps You will Use as Part of Your Assessment 

Have open and honest conversations with students about how you plan to use detection tools. Point out that there is a written statement in the course outline and that you have the support of your department head and the institution to use these tools. Be upfront and clear with students.

It is also important to engage students in evidence-based conversations about the limitations tools to detect artificial intelligence writing, including the current lack of empirical evidence about how well they work.

Conclusion

Again, I emphasize that I am not at all promoting the use of any AI detection technology whatsoever. In fact, I am opposed to the use of surveillance and detection technology that is used punitively against students, especially when it is done in the name of teaching and learning. However, if you are going to insist on using technology to detect possible breaches of academic integrity, then at least do so in an open and transparent way — and acknowledge that the tools themselves are imperfect.

Key message: Under no circumstances should the results from an AI-writing detection tool be used as the only evidence in a student academic misconduct allegation.

I am fully anticipating some backlash to this post. There will be some of you who will object to the use detection tools on principle and counter that any blog post talking about how they can be used is in itself unethical. You might be right, but the reality remains that thousands of educators are currently using detection tools for the sole purpose of catching cheating students. As much as I rally against a “search and destroy” approach, there will be some people who insist on taking this position. This blog post is to offer some guidelines to avoid deceptive assessment and covert use of technology in student assessment.

Key message: Deceptive assessment is a breach of academic integrity on the part of the educator. If we want students to act with integrity, then it is up to educators to model ethical behaviour themselves.

References

Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-Generated Text be Reliably Detected? ArXiv. https://doi.org/10.48550/arXiv.2303.11156

Fowler, G. A. (2023, April 3). We tested a new ChatGPT-detector for teachers. It flagged an innocent student. Washington Post. https://www.washingtonpost.com/technology/2023/04/01/chatgpt-cheating-detection-turnitin/

Jimenez, K. (2023, April 13). Professors are using ChatGPT detector tools to accuse students of cheating. But what if the software is wrong? USA Today. https://www.usatoday.com/story/news/education/2023/04/12/how-ai-detection-tool-spawned-false-cheating-case-uc-davis/11600777002/

_________________________________

Share this post: The Use of AI-Detection Tools in the Assessment of Student Work – https://drsaraheaton.wordpress.com/2023/05/06/the-use-of-ai-detection-tools-in-the-assessment-of-student-work/

This blog has had over 3 million views thanks to readers like you. If you enjoyed this post, please “like” it or share it on social media. Thanks! Sarah Elaine Eaton, PhD, is a faculty member in the Werklund School of Education, and the Educational Leader in Residence, Academic Integrity, University of Calgary, Canada. Opinions are my own and do not represent those of the University of Calgary.


Handbook of Academic Integrity (2nd Ed.) is In Production!

May 4, 2023

I am beyond excited to share that all of the chapters for the Handbook of Academic Integrity (2nd ed.) are fully approved and in production!

This is a massive volume, with 112 chapters, 109 of which are brand new. There are 3 chapters carried forward from the first edition into the new one.

I wanted to share an explanation about how the Handbook will be available online.

The electronic version of the Handbook of Academic Integrity is — and will continue to be — available in two (2) formats. One format is the dynamic online version and the other is the static online version.

Dynamic Online Version

The Dynamic Online Version of the Handbook is also called the “living” version. As and when an author decides to update a previous edition contribution, the updated version will replace the previous version on Springerlink (while the previous version is still accessible). In this “living” format, the Table of Contents is not organized section-wise, instead the chapters are added alphabetically as and when the online-first publication is completed for a chapter. 

A screenshot of the website of the living edition of the Handbook of Academic Integrity (2nd edition).

As we progress with production, the living version of the Handbook of Academic Integrity will include

  • first edition chapters that were not updated in the second edition;
  • revised first edition chapters (with a hyperlink to the previous version);
  • and the new chapters that were added to the second edition.

Here is a link to the Living version of the Handbook of Academic Integrityhttps://link.springer.com/referencework/10.1007/978-981-287-079-7

Static Online Version

Each edition of the Handbook is also be published in its entirety once the book is complete.

This means, that the first edition of the Handbook edited by Tracey Bretag, is and will continue to be, preserved in its entirety online.

Screenshot of the website for the first edition of the Handbook of Academic Integrity living edition

Here is the link to the static online version of the first edition: https://link.springer.com/referencework/10.1007/978-981-287-098-8

Springer has assured me that the first edition link will never expire and will remain online in perpetuity.

The static online version for the second edition will be available when the entire edition is ready to go to print. That may not be for several months yet because this is a massive project and also it is detail-oriented work that must be completed to exacting standards.

In the meantime, chapters from the new edition will be added to the living version in an “online first format” as they ready. 

The First Chapter of the Second Edition Has Been Published

Screen shot of a chapter in the Handbook of Academic Integrity (2nd edition).

I confess that I do not know the inner workings of the publisher in terms of all of the production details. This means that I do not know the order in which chapters will become available online and nor do I have specific dates for when chapters will be available online.

Having said that, I am happy to share that one chapter from the new edition is now available online. Here are the details:

Stoesz, B.M. (2023). Academic integrity through ethical teaching and assessment: Overview and current trends. In: Eaton, S.E. (eds) Handbook of Academic Integrity. Springer, Singapore. https://doi.org/10.1007/978-981-287-079-7_89-1 

This is the first of over 100 new chapters that will all be published over the next several months. Keep your eye on the website for the living edition for more chapters coming soon!

_________________________________

Share this post: Handbook of Academic Integrity (2nd Ed.) is In Production! https://drsaraheaton.wordpress.com/2023/05/04/handbook-of-academic-integrity-2nd-ed-is-in-production/

This blog has had over 3 million views thanks to readers like you. If you enjoyed this post, please “like” it or share it on social media. Thanks! Sarah Elaine Eaton, PhD, is a faculty member in the Werklund School of Education, and the Educational Leader in Residence, Academic Integrity, University of Calgary, Canada. Opinions are my own and do not represent those of the University of Calgary.


Invitation to Participate: Research Study on Artificial Intelligence and Academic Integrity: 

April 19, 2023

The Ethics of Teaching and Learning with Algorithmic Writing Technologies 

On the right there is a black robotic hand and forearm. On the left there is a human hand and forearm. The forearm is tatooed. One finger from each hand is touching the other.
Photo by cottonbro studio on Pexels.com

Academic misconduct has taken various forms in present-day educational systems. One method that is on the rise is the use of artificially generated software compositions. The capabilities and sophistication of these new technologies are improving steadily. We are conducting a study to gauge the sophistication of the current artificial intelligence (AI) software-generated text. To that end, we are recruiting participants to evaluate the level of writing level of small compositions (260 words in length at most).

Your participation in this study would be to evaluate two small pieces of text presented in a survey and optionally make comments on your observation. We appreciate your consideration in this matter. This research provides an opportunity for the participants to contribute to the state of AI software used for various educational purposes. Participation in this study is voluntary, and you are free to terminate the survey and withdraw at any time and for any reason without censor. There are no known physical, psychological, or social risks associated with participation in the study.

All demographic data collected will be kept strictly confidential. Only the researchers listed in this letter will have access to the raw data. The data (in electronic format) will be retained indefinitely. Participation in the study will be asked for some basic demographic information and then presented with a 260- word length composition. After reading, the participants will be asked to evaluate the level, assign a mark to the composition, and note any pertinent observations. The second piece of composition, also of the same length, will be followed by the same set of questions. The total anticipated time for completing the survey is about 9-12 minutes, but it can vary based on reading speed and consideration afforded to the assigned grade.

If you have any questions or concerns about your participation in this study, you can contact the Principal Investigator, Dr. Sarah Elaine Eaton, seaton (at) ucalgary.ca

This study is funded by a University of Calgary Teaching and Learning Grant. This study has been approved by the Conjoint Faculties Research Ethics Board at the University of Calgary: REB22-0137.

To take the survey, click here.

_________________________________

Share this post: Invitation to Participate: Research Study on Artificial Intelligence and Academic Integrity: The Ethics of Teaching and Learning with Algorithmic Writing Technologies – https://wp.me/pNAh3-2U3

This blog has had over 3 million views thanks to readers like you. If you enjoyed this post, please “like” it or share it on social media. Thanks! Sarah Elaine Eaton, PhD, is a faculty member in the Werklund School of Education, and the Educational Leader in Residence, Academic Integrity, University of Calgary, Canada. Opinions are my own and do not represent those of the University of Calgary.

 


A Comprehensive Academic Integrity (CAI) Framework: An Overview

April 14, 2023

This post is a reprint of a self-archived document available here: https://prism.ucalgary.ca/handle/1880/116060

Background

As I write this, the Handbook of Academic Integrity (2nd ed.) is well underway. All the chapters have been submitted and are at various stages of review, revision, and production. Page proofs should start going out to contributing authors this month. This has been a massive project: 150+ authors, 109 chapters, 9 section editors, and me herding all the cool cats who have made it happen.

After all the chapters had been submitted, I realized that we had something new and fresh with this edition. We have pushed the boundaries beyond persistently historical ideas about academic integrity only as a matter of student conduct. So, I wrote an introduction for the handbook that synthesizes some its through lines. The common threads of this updated edition are summed up in this Comprehensive Academic Integrity (CAI) framework.

I wrote this intro in two epic writing sessions, fuelled by gallons of coffee, Vegemite on toast, and a pizza that a friend had delivered to my house because he felt sorry for me eating Vegemite on toasted bread crusts. (I am not kidding.) Anyway, first, I drafted the chapter in full. Then I asked a couple of other section editors and contributors to the handbook to provide me with an open peer review of the draft. In the chapter I acknowledge them by name and I re-iterate my gratitude to them here. Thanks are due to Guy Curtis, Brenda M. Stoesz, Rahul Kumar, Beatriz Moya, and Bibek Dahal for their feedback that helped me to improve the chapter. In the second writing session, I incorporated just about all of their suggestions and completely re-vamped the visual image to the one you see below. The CAI Framework is a high-level synthesis of all the chapters in the handbook and as such, every single author who has contributed to the handbook (as well as those they have cited in their respective chapters) all deserve credit.

According to the publisher’s rules around self-archiving and pre-prints, I am not allowed to share the entire chapter with you ahead of publication. But I can share a summary of it, so I’m doing that here. I’ve also self-archived a copy of this overview (minus the background commentary about Vegemite and pizza) in our university’s digital repository. On the off-chance you want to cite the “official” version of the summary, I have included instructions below. You’ll have to wait for the Handbook to be published to read the full chapter, but in the meantime, I hope this overview is useful.

How to cite this overview

Eaton, S.E. (2023). A Comprehensive Academic Integrity (CAI) Framework: An Overview. Calgary, Canada: University of Calgary. https://prism.ucalgary.ca/handle/1880/116060

Overview

For years scholars and other experts have called for a more holistic approach to academic integrity (e.g., Bertram Gallant, 2008; Boud & Bearman, 2022; Bretag et al, 2014; Carrol & Duggan, 2005; Löfström et al., 2015; Morris & Carrol, 2016; Turner & Beemsterboer, 2003). The CAI framework synthesizes ideas that have been repeated for decades in various iterations.

The central argument behind a wholistic framework is that academic integrity must encompass, but extend beyond, notions of student conduct, and should be considered a foundation of all aspects of education. In this framework, I do not propose a new definition of academic integrity in part, because several useful definitions already exist (see Bretag, 2016; ICAI, 2021; Tauginienė et al., 2018). Instead, this framework can be used with existing definitions.

The Comprehensive Academic Integrity (CAI) framework includes eight (8) essential elements that includes, and extends beyond traditional notions of academic integrity merely as a student responsibility:

  • everyday ethics
  • institutional ethics
  • ethical leadership
  • professional and collegial ethics
  • instructional ethics
  • student academic conduct
  • research integrity and ethics
  • publication ethics

Figure 1. Comprehensive Academic Integrity (CAI) Framework

A circle with eight colourful swirls (one each in green, blue, pink, dark yellow, darker purple, red, bright yellow, medium purple, and green). There is black text associated with each swirl. This image is a graphic representation of the Comprehensive Academic Integrity (CAI) framework.

Keywords

academic integrity, student conduct, student affairs, research ethics, research integrity, publication ethics, instructional ethics, pedagogy, everyday ethics, experiential learning, definition, ethical decision-making, morals, values, virtues, leadership, equity, diversity, inclusion, accessibility, justice, decolonization, Indigenization

Postscript

Appreciation to Kieran Forde at the University of British Columbia for his most awesome interpretation of the graphic as a “colourful swirly donut”. Who doesn’t love donuts?! Thanks, Kieran!

References

Bertram Gallant, T. (2008). Academic integrity in the twenty-first century: A teaching and learning imperative. Wiley.

Boud, D., & Bearman, M. (2022). The assessment challenge of social and collaborative learning in higher education. Educational philosophy and theory, 1-10. https://doi.org/10.1080/00131857.2022.2114346   

Bretag, T. (2016). Educational integrity in Australia. In T. A. Bretag (Ed.), Handbook of Academic Integrity (pp. 1-13). Springer Singapore. https://doi.org/10.1007/978-981-287-079-7_2-1 

Bretag, T., Mahmud, S., Wallace, M., Walker, R., McGowan, U., East, J., Green, M., Partridge, L., & James, C. (2014). ‘Teach us how to do it properly!’ An Australian academic integrity student survey. Studies in higher education, 39(7), 1150-1169. https://doi.org/10.1080/03075079.2013.777406

Carroll, J., & Duggan, F. (2005, December 2-5). Institutional change to deter student plagiarism:  What seems essential to a holistic approach? 2nd Asia-Pacific Educational Integrity Conference, University of Newcastle, Callaghan, Australia.

Eaton, S. E. (forthcoming). Comprehensive academic integrity (CAI): An ethical framework for educational contexts. In S. E. Eaton (Ed.), Handbook of Academic Integrity (2nd ed.). Springer. 

International Center for Academic Integrity (ICAI). (2021). The fundamental values of academic integrity (3rd ed.). https://academicintegrity.org/resources/fundamental-values

Löfström, E., Trotman, T., Furnari, M., & Shephard, K. (2015). Who teaches academic integrity and how do they teach it? Higher Education, 69(3), 435-448. https://doi.org/10.1007/s10734-014-9784-3

Morris, E. J., & Carroll, J. (2016). Developing a sustainable holistic institutional approach: Dealing with realities “on the ground” when implementing an academic integrity policy. In T. Bretag (Ed.), Handbook of Academic Integrity (pp. 449-462). Springer Singapore. https://doi.org/10.1007/978-981-287-098-8_23

Tauginienė, L, Gaižauskaitė, I, Glendinning, I, Kravjar, J, Ojsteršek, M, Ribeiro, L, Odiņeca, T, Marino, F, Cosentino, M, Sivasubramaniam, S, Foltýnek, T. Glossary for Academic Integrity. ENAI Report 3G [online]: revised version, October 2018. https://www.academicintegrity.eu/wp/wp-content/uploads/2023/02/EN-Glossary_revised_final_24.02.23.pdf

Turner, S. P., & Beemsterboer, P. L. (2003). Enhancing academic integrity: Formulating effective honor codes. Journal of Dental Education, 67(10), 1122-1129. https://doi.org/10.1002/j.0022-0337.2003.67.10.tb03705.x

_________________________________

Share this post: A Comprehensive Academic Integrity (CAI) Framework: An Overview – https://drsaraheaton.wordpress.com/2023/04/14/a-comprehensive-academic-integrity-cai-framework-an-overview/

This blog has had over 3 million views thanks to readers like you. If you enjoyed this post, please “like” it or share it on
social media. Thanks!

Sarah Elaine Eaton, PhD, is a faculty member in the Werklund School of Education, and the Educational Leader in Residence, Academic Integrity, University of Calgary, Canada. Opinions are my own and do not represent those of the University of Calgary.


How to Talk to Your Students about ChatGPT: A Lesson Plan for High School and College Students

April 7, 2023
bionic hand and human hand finger pointing

Photo by cottonbro studio on Pexels.com

This article by Ben Edwards in ArtsTechnica (April 6, 2023) is worth a read, “Why ChatGPT and Bing Chat are so good at making things up”.

Edwards explains in clear language, with lots of details and examples, how and why large language models (LLMs) such as ChatGPT make up content. As I read this article, it occurred to me that it could serve as a really great way to have pro-active and generative conversations with students about the impact of artificial intelligence for teaching, learning, assessment, and academic integrity. So, here is a quick lesson plan about how to use this article in class:

Education level

Secondary school and post-secondary (e.g., community college, polytechnic, undergraduate or graduate university courses)

Lesson Plan Title: Understanding ChatGPT: Benefits and Limitations

Learning Objectives

By the end of this lesson students will be able to:

  • Understand how and why AI-writing apps make up content.
  • Explain the term “confabulation”.
  • Discuss the implications of fabricated content on academic integrity
  • Generate ideas about how to fact-check AI-generated content to ensure its accuracy

Lesson Preparation

Prior to the class, students should read this article: “Why ChatGPT and Bing Chat are so good at making things up by Ben Edwards, published in ArtsTechnica (April 6, 2023)

Come to class prepared to discuss the article.

Learning Activity

Class discussion (large group format if the class is small or small group format with a large group debrief at the end):

Possible guiding questions:

  • What is your experience with ChatGPT and other AI writing apps?
  • What were the main points in this article? (Alternate phrasing: What were your key takeaways from this article?)
  • What are some of the risks when AI apps engage in confabulation (i.e., fabrication)?
  • Discuss this quotation from the article, “ChatGPT as it is currently designed, is not a reliable source of factual information and cannot be trusted as such.”
  • Fabrication and falsification are commonly included in academic misconduct policies. What do you think the implications are for students and researchers when they write with AI apps?
  • What are some strategies or tips we can use to fact-check text generated by AI apps?
  • What is the importance of prompt-writing when working with AI writing apps?

Duration

The time commitment for the pre-reading will vary from one student to the next. The duration of the learning activity can be adjusted to suit the needs of your class.
  • Students’ pre-reading of the article: 60-minutes or less
  • Learning activity: 45-60 minutes

Lesson closure

Thank students for engaging actively in the discussion and sharing their ideas.

Possible Follow-up Activities

  • Tips for fact-checking. Have students in the class generate their own list of tips to fact-check AI-generated content (e.g., in a shared Google doc or by sharing ideas orally in class that one person inputs into a document on behalf of the class.)
  • Prompt-writing activity. Have students use different prompts to generate content from AI writing apps. Ask them to document each prompt and write down their observations about what worked and what didn’t. Discuss the results as a class.
  • Academic Integrity Policy Treasure Hunt and Discussion. Have students locate the school’s academic misconduct / academic integrity policy. Compare the definitions and categories for academic misconduct in the school’s policies with concepts presented in this article such as confabulation. Have students generate their own ideas about how to uphold the school’s academic integrity policies when using AI apps.

Creative Commons License

This lesson plan is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). This license applies only to the lesson plan, not to the original article by Ben Edwards.

Additional Notes

This is a generic (and imperfect) lesson plan. It can (and probably should) be adapted or personalized depending on the needs of the learners.

Acknowledgements

Thanks to Dr. Rahul Kumar, Brock University for providing an open peer review of this lesson plan.

 _________________________________

Share or Tweet this: How to Talk to Your Students about ChatGPT: A Lesson Plan for High School and College Students – https://drsaraheaton.wordpress.com/2023/04/07/how-to-talk-to-your-students-about-chatgpt-a-lesson-plan-for-high-school-and-college-students This blog has had over 3 million views thanks to readers like you. If you enjoyed this post, please “like” it or share it on social media. Thanks! Sarah Elaine Eaton, PhD, is a faculty member in the Werklund School of Education, and the Educational Leader in Residence, Academic Integrity, University of Calgary, Canada. Opinions are my own and do not represent those of the University of Calgary.

%d bloggers like this: