AI Tutoring has Arrived! (Or has it?)

A video featuring Sal Khan (of Khan Academy) and his teenage son using the latest version of ChatGPT as an AI tutor has been making some waves in the education space recently, eliciting a range of reactions from excitement to concern to confusion. I took a closer look at this AI demonstration in an effort to determine what ChatGPT is actually currently capable of and what aspects of this demo made use of technology that is not currently available to the average user. 

If you have not yet seen the video, here it is:

I think most people will agree that ChatGPT does a surprisingly good job tutoring the student. It doesn't just get the problem right, it skillfully guides the students through the problem in a friendly, patient, and supportive manner. And, perhaps more importantly, it scaffolds the learning process quite well. (Scaffolding is the process of providing the right amount of support to a student during a learning process. Too little help leads to frustration, while too much ends up doing the work for the student.) 

So, what did I discover? Basically, that this level of support is not yet available to the general public.

My first step in determining this was to attempt to recreate the interaction shown in the demo. I am quite familiar with the capabilities of ChatGPT, but I couldn't think of a way to get the AI to see in real time a second app open on an iPad, as is shown in the demo. Maybe I just wasn't thinking creatively enough?

In an effort to rule this out, I simply asked ChatGPT if it was capable of seeing other content on my iPad. 

Here's a screenshot of the interaction:

ChatGPT seemed pretty confident that it is not yet able to see other content on my iPad.

But I wanted to se sure this wasn't merely a hallucination. While ChatGPT knows a lot about its own capabilities, it doesn't know everything. Further, the new model had only been released a few days prior to the above conversation, so I thought that maybe it just hadn't updated its knowledge base yet.

I continued to ponder how I might recreate the demo.

Stumped, I decided to rewatch the video itself to see if I could gain any clues as to how they'd set it up. That's when I noticed a subtle, but important detail in the video that revealed that the version of ChatGPT used in the demo is substantially different from what is currently available to the general public, even if the underlying model being used is the same. 

​Below is a screenshot taken from the demo video, just before Sal's son connects ChatGPT to the other app on his iPad. There is a small icon in the lower left hand corner of their version of ChatGPT that looks like a little TV:

Here is the same screenshot, just zoomed in:

And, here is what happens after Sal's son taps that icon:

The above screenshot is a bit blurry, but you can still see that the pop-up says "Screen Broadcast." This seems to be the feature that allows ChatGPT to see the rest of the iPad. And, it's a feature that is missing from the currently available version of  ChatGPT.

For example, here is what the version I have looks like:

There is no TV icon on mine. Just a pause button.

At this point, I knew I was simply not going to be able to recreate the demo exactly as it had occurred. However, I wondered if a workaround might not allow me to come close. I wondered what might happen if I uploaded a screenshot of the same math problem and then asked ChatGPT to walk me through it, step by step.

Unfortunately, this didn't work either. At least not initially.

Here is my conversation with ChatGPT:

As you can see, while ChatGPT did lay out its explanation in a "step by step" manner, it didn't walk me through the problem, in real time, as I was hoping for. (It also didn't refrain from telling me the correct answer.) 

At first, I assumed this meant that the model they were using might actually be different under the hood, after all, even though they were calling it by the same name. But then I thought that the difference I observed might not have to do with the way I prompted ChatGPT, versus how Sal Khan did in the demo.

​So, I decided to try again, using Sal's exact wording.

This time, it responded a lot more like ChatGPT did in the demo:

This revealed a lot to me.

I big part of effectively using generative AI comes down to skillful prompting (as anyone who has ever experimented with an AI image program will surely know). "Prompt engineering" is more of an art than a science, and often requires a fair bit of trial and error to get out what one wants from a particular program. 

When I think about students trying to use ChatGPT for homework help, I can't help but think that most would likely use a prompt closer to my first, ineffective version than to the longer, more detailed prompt that Sal uses in his demo. And, if I'm being honest, I think most students would simply upload the problem and ask ChatGPT, "How do you solve this?"

If they were to do that, here's what they would get:

This explanation is definitely not wrong. It's just a far cry from the highly interactive, remarkably empathetic, and skillfully scaffolded version we all got to see in the video.

So, even if OpenAI releases an updated version of ChatGPT that has the exact same functionality as the version used in the demo, the program will still have to be prompted correctly to achieve a similar type of interactive experience. 

That said, I couldn't help but wonder how long it will be before interactions like the one shown in the video become commonplace. So, I turned, once again to ChatGPT. This time I asked it how long it would be before AI tutors will replace human tutors. 

​Here is the response I got:

I appreciated ChatGPT's response, particularly this section:

Human tutors do more than teach academic skills; they understand emotional cues, build meaningful relationships, and provide motivation and encouragement in ways that are deeply personalized and nuanced. While AI can simulate some of these aspects, the genuine human connection and understanding are challenging to replicate fully with current technology.

​I couldn't have said it better myself.