AI in education

AI in Education

Survey/study of

• 2023-08: Perception, performance, and detectability of conversational artificial intelligence across 32 university courses
• 2023-10: Employees secretly using AI at work.
• 2023-10: Survey shows students using AI more than professors.
• 2023-11: ChatGPT has entered the classroom: how LLMs could transform education

AI improves learning/education

• Mollick, Ethan R. and Mollick, Lilach and Bach, Natalie and Ciccarelli, LJ and Przystanski, Ben and Ravipinto, Daniel, AI Agents and Education: Simulated Practice at Scale (June 17, 2024). The Wharton School Research Paper. doi: 10.2139/ssrn.4871171
  • Can enable personalized education.
• Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
  • GPT4 can out-perform human tutors.
• Keppler, Samantha and Sinchaisri, Wichinpong and Snyder, Clare, Backwards Planning with Generative AI: Case Study Evidence from US K12 Teachers (August 13, 2024). doi: 10.2139/ssrn.4924786
  • Teachers benefit from using AI as a co-pilot to aid in tasks (planning, how to teach topic, explore ideas).
  • There is smaller utility in using AI purely as a text-generator (to make quizzes, workbooks, etc.).
• Effective and Scalable Math Support: Evidence on the Impact of an AI- Tutor on Math Achievement in Ghana
• AI Tutoring Outperforms Active Learning

AI harms learning

• A real-world test of artificial intelligence infiltration of a university examinations system: A “Turing Test” case study
  • Current grading systems cannot detect AI.
• Bastani, Hamsa and Bastani, Osbert and Sungu, Alp and Ge, Haosen and Kabakcı, Özge and Mariman, Rei, Generative AI Can Harm Learning (July 15, 2024). The Wharton School Research Paper.doi: 10.2139/ssrn.4895486
  • Access to ChatGPT harmed math education outcomes.
• 2024-09: AI Meets the Classroom: When Does ChatGPT Harm Learning?

Software/systems

• GPTutor (code)
• EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education
• Eureka Labs (founded by Andrej Karpathy) aims to create AI-driven courses (first course is Intro to LLMs)

Individual tools

• Chatbot (OpenAI ChatGPT, Anthropic Claude, Google Gemini)
• NotebookLM: Enables one to "chat with documents".

AI for grading

• Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K-12 Education (preprint)

Detection

• Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays
  • GenAI can simulate student writing in a way that teachers cannot detect.
  • AI essays are assessed more positively than student-written.
  • Teachers are overconfident in their source identification.
  • Both novice and experienced teachers could not identify texts generated by ChatGPT vs. students

AI Text Detectors Don't Work

• RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

AI/human

AI out-performs humans

Tests

• 2023-07: SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
• 2024-06: A real-world test of artificial intelligence infiltration of a university examinations system: A “Turing Test” case study
  • AI scores higher than median students.

Creativity

• 2023-09: Best humans still outperform artificial intelligence in a creative divergent thinking task
  • Best humans out-perform AI at creativity. (By implication, median humans may not.)
• 2024-02: The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks
• 2024-02: Felin, Teppo and Holweg, Matthias, Theory Is All You Need: AI, Human Cognition, and Causal Reasoning (February 24, 2024). doi: 10.2139/ssrn.4737265
  • Argues that human "theory-based" creativity is better than AI "data-based".
• 2024-07: Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?
  • Top human (professional author) out-performs GPT4.
• 2024-09: Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
  • LLMs can be creative
• 2024-09: Creative and Strategic Capabilities of Generative AI: Evidence from Large-Scale Experiments

Professions

• 2024-03: Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study
  • GPT4 improves medical practitioner work; surprisingly, GPT4 alone scored better than a human with GPT4 as aid (on selected tasks).
• 2024-10: Perspectives on Artificial Intelligence–Generated Responses to Patient Messages
• Humanity's Last Exam
  • Effort to build a dataset of challenging (but resolvable) questions in specific domain areas, to act as a benchmark to test whether AIs are improving in these challenging topics.

AI improves human work

• 2023-07: Experimental evidence on the productivity effects of generative artificial intelligence
• 2023-09: Dell'Acqua, Fabrizio and McFowland III, Edward and Mollick, Ethan R. and Lifshitz-Assaf, Hila and Kellogg, Katherine and Rajendran, Saran and Krayer, Lisa and Candelon, François and Lakhani, Karim R., Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (September 15, 2023). Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 24-013, The Wharton School Research Paper doi: 10.2139/ssrn.4573321
• 2023-11: Generative AI at Work (National Bureau of Economic Research)
• 2023-12: The Uneven Impact of Generative AI on Entrepreneurial Performance (doi: 10.31219/osf.io/hdjpk)
• 2024-07: Generative AI in Real-World Workplaces: The Second Microsoft Report on AI and Productivity Research

Coding

• 2023-02: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
• 2024-09: Cui, Zheyuan and Demirer, Mert and Jaffe, Sonia and Musolff, Leon and Peng, Sida and Salz, Tobias, The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers (September 03, 2024). doi: 10.2139/ssrn.4945566

Forecasting

• 2024-02: AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy

Creativity

• 2024-07: Generative AI enhances individual creativity but reduces the collective diversity of novel content
• 2024-08: An empirical investigation of the impact of ChatGPT on creativity

Counter loneliness

• 2024-07: AI Companions Reduce Loneliness

AI passes Turing Test

• 2023-05: Human or Not? A Gamified Approach to the Turing Test
• 2023-10: Does GPT-4 pass the Turing test?
• 2024-05: People cannot distinguish GPT-4 from a human in a Turing test
• 2024-07: GPT-4 is judged more human than humans in displaced and inverted Turing tests

Uptake

• 2024-03: Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
• 2024-05: Humlum, Anders and Vestergaard, Emilie, The Adoption of ChatGPT. IZA Discussion Paper No. 16992 doi: 10.2139/ssrn.4827166
• 2024-06: Delving into ChatGPT usage in academic writing through excess vocabulary
• 2024-09: The Rapid Adoption of Generative AI
• 2024-10: Growing Up: Navigating Generative AI’s Early Years – AI Adoption Report (executive summary, full report)
  • 72% of leaders use genAI at least once a week (c.f. 23% in 2023); 90% agree AI enhances skills (c.f. 80% in 2023)
  • Spending on genAI is up 130% (most companies plan to invest going forward)

See Also

• UNESCO. Guidance for Generative AI in Education and Research