There’s a little chatbot getting a lot of attention right now. Maybe you’ve heard of it? ChatGPT. Launched by OpenAI in November 2022, this generative artificial intelligence (AI) technology is literally blowing people’s minds with its ability to answer prompts by generating full written essays and formulating complete ideas.
The education community – at all levels – is in a tailspin as it figures out how to accept, monitor and potentially embrace ChatGPT’s game-changing capabilities. What’s to stop a student from feeding the chatbot its assigned essay questions and submitting the AI-generated responses? What does this mean for the future of individual thought and creativity – and teacher assessment?
Christian Terwiesch, a professor of operations, information and decisions and co-director of the Mack Institute of Innovation Management at the Wharton School of the University of Pennsylvania, has already begun his research into the power and possibilities of ChatGPT and related technologies. He recently published the white paper Would ChatGPT Get a Wharton MBA? to see if the technology was smart enough to pass an exam in a typical Wharton Master of Business Administration course — in this case, his operations management course, which is about using business practices to make a company or organization more efficient.
We caught up with Professor Terwiesch at Wharton San Francisco, where he is currently teaching executive MBAs, to talk about ChatGPT and its potential for a great GPA.
Wharton Global Youth: We’re all talking about ChatGPT, but not everyone fully understands how it works. Why is this such an incredible AI advancement?
Christian Terwiesch: The big new thing here is that it is not an algorithm [responding to] a set of commands. It is a model that has learned patterns and by going through millions of millions of texts [content on the internet] has found a way of predicting in the world of natural language which word will be used next. Just like when you’re typing on your computer or on your phone and it makes suggestions of how to finish a word. There’s this predictive element in there, and if you train it on enough content, it is able to figure out the rhythm, style and context and make some good predictions on how to continue. It’s iPhone text completion on steroids.
Wharton Global Youth: Why did you decide to document how ChatGPT3 performed on the final exam of a typical MBA core course? Is there a story behind it?
Terwiesch: Just before the winter break, my kids and I were sitting around the dinner table. My kids are in their 20s and one of them is really interested in design. He had experimented with Midjourney (an AI-based art generator) and Dall-E (creating images from text) and was blown away by the power of the technology. My other son is interested in computer science and had been talking about ChatGPT and its coding capabilities. As the discussion emerged, it came up: ‘OK, Dad, do you think this thing can pass your exam?’ Since we were sitting at the computer anyway, I took my first exam question, cut and pasted it into the bot command line, and I was just blown away. It was not a super difficult question, but I was blown away by the correctness and the quality of the argumentation, reasoning and language.
And so I said, let me try out others. Literally, this mixed feeling emerged. Mostly, I was overwhelmingly enthusiastic about how good it was, But then, there were surprising mistakes — things where middle school math would suffice. My overall sentiment was amazement and awe, but there were these wrinkles in there and that made it interesting.
Wharton Global Youth: What did you learn from your research about ChatGPT’s academic performance?
Terwiesch: I think three pieces. It did really well on simple questions. The content was correct, the argumentation and the justification, the wording, the explanation was beautiful. So, A+ there. On more complex questions, it struggled. It did require a human hint and even with a human hint, some of the math was wrong. The math was wrong at a level that is not particularly difficult. The score was more in the B to B- range. And the third one was the creativity element. The last thing I did was asked it to generate exam questions for me. These questions were good, but not great. They were creative in a way, but they still required polishing. I could imagine in the future looking to the ChatGPT as a partner to help me get started with some exam questions and then continue from there.
Wharton Global Youth: As a professor, how do you feel about ChatGPT’s ability to do solid and sometimes stellar work?
Terwiesch: The question that high school students, college students, MBA students and [all of us] in the educational industry are wrestling with is how we should deal with the usage of technology on tests. Should we ban it? There’s been some really good thought processes happening here at Penn. I think we need to ask ourselves: ‘Why do we test? What is the purpose of the test?’ Do we test because we want to certify skill? Do we test because we want to have a sense of where our learners are at the moment? Or do we test to engage the students in the content? I think it’s only on the third type of testing where we have to adjust course and reimagine all those test questions. Because what used to take a student three hours in the library to write a little paper, now the student can do in five minutes without even engaging the material. In that sense, the test no longer fulfills its purpose. I think there we should imagine new forms of giving students reasons to engage with the material. If anything, I think that ChatGPT is going to be our friend. It’s going to open up amazing opportunities to engage students in learning.
Wharton Global Youth: How would you guide a high school student who is considering using ChatGPT to write an essay?
Terwiesch: First of all, make sure you comply to the ethics policy in your classes. Your teacher and your institution have given you rules, so don’t play foul. Cheating is always something you will look back on in life and regret later.
My biggest advice having played with the tool at least a little bit is to hit the repeat button, so to speak. Generate multiple answers and try to challenge your thinking. Try to hear opposing points of view. Encourage it to contradict itself. Ask it to go left, but then ask it to go right. And then get stimulated by these very different responses and find the one that resonates with you, then write your essay from there. Use it to explore the landscape as opposed to using it as a cheap writing tool. You’re not taking full advantage of the technology if all you’re doing is using it as a writing machine. Of all these perspectives you’ve just seen, which one do you want to put in your paper? Use the technology to help stimulate your thinking.
“I can’t wait to imagine a world where teachers, where doctors, where social workers are leveraged in their productivity and their capability, so that they can help patients, children and people who need it.”
Wharton Global Youth: What impact will generative AI have on critical thinking and creativity?
Terwiesch: Let’s start with creativity. Creativity has two steps to it; it’s a very evolutionary process. You need to create and you need to select. It doesn’t do you any good to have 100 ideas unless you pick the best idea to run with. I think it’s going to be amazing at helping us to come up with new ideas, even in the current format, which is just recombining and taking a synthesis of what is out there. It’s going to create options for us. We still have to do the selection. We still have to do what a venture capitalist or a big corporate innovation office does. We have to select from those 10 selections the one that is going to create value. That is requiring the critical thinking. Right now, ChatGPT is a little hit or miss. Some of the suggestions are total garbage and in operations I don’t like that. In operations, I like things that are reliable and predictable. But in innovation, I’d much rather have you give me 10 ideas of which one is outstanding and nine are total crap, as opposed to giving me 10 okay ideas.
Wharton Global Youth: You are the co-director of Wharton’s Mack Institute of Innovation Management. What does this technology mean for the future of innovation?
Terwiesch: I think it’s too broad to call ChatGPT one innovation. Think about how we use the internet or the iPhone? It’s just so multi-dimensional. It’s not one innovation; it’s a name or a catch-all phrase for thousands of exciting innovations in medicine, education, law and business. There’s no “it,” if you will.
An innovation is broadly defined as a novel match between a solution and a need. Here we have a new class of solution approach for all kinds of needs. There are going to be hundreds of thousands of needs that are going to get met now in a new way. This could be a doctor doing a prescription without seeing a patient, a teacher grading a homework assignment, it might be an architect sketching out a drawing for a new house. Whatever that purpose is, AI is going to have a major role in it, but the individual innovations are going to be very different from each other. Think about the internet. It has changed so much in our lives that it would be funny to call the innovation the internet. The innovation is how we now file our taxes, speak to each other over a video line as opposed to a landline. It’s a new solution that will enable thousands of innovations.
Wharton Global Youth: Are you already thinking about other areas of research?
Terwiesch: It’s really fascinating to imagine a world where technology is connecting customer and firm 24/7. But at the same time, it just generates new challenges because we have limited resources. In health care, physician and nursing burnout is already a major topic in the medical community (Dr. Terwiesch is also a professor of health policy). So, if you want to make this [connectedness] happen, we need some form of AI and automation to deal with all of the information. How this is going to play out is absolutely fascinating to study. We’ve seen first reports that ChatGPT doesn’t only do well on the MBA exam, but also does well on medical school exams. I can’t wait to continue that work and imagine a world where teachers, where doctors, where social workers are leveraged in their productivity and their capability so that they can help patients, children and people who need it.
Conversation Starters
Dr. Christian Terwiesch says you should consider using ChatGPT to “generate multiple answers and try to challenge your thinking.” How do you feel about this? Do you agree or disagree? Have you done this already? Share your story and ideas about how you are using ChatGPT in the comment section of this article.
Is using ChatGPT to write an essay unethical? Why or why not?
Dr. Terwiesch talks about the power of ChatGPT to ease the information burden in health care. How do you see this technology transforming different industries? Where might generative AI make a strong impact?
