In Classrooms, Teachers Put A.I. Tutoring Bots to the Test

Discussions - Public
1

Hey team, here is a very interesting article in the New York Times about experiments in using OpenAI-based tutors for education in the Newark school district. Not sure if you’ll hit a paywall or not. If so, sorry :/

Some quotes in case there’s a paywall or you don’t have time:

Newark has essentially volunteered to be a guinea pig for public schools across the country that are trying to distinguish the practical use of new A.I.-assisted tutoring bots from their marketing promises. It is one of the first school systems in the United States to pilot test Khanmigo, an automated teaching aid developed by Khan Academy, [and using OpenAI’s GPT models under the hood]

Newark students began using Khan’s automated teaching aid in May. The reviews have so far been mixed.

Mr. Rodriguez described the bot as a useful “co-teacher” that allowed him to devote extra time to children who needed guidance while enabling more self-driven students to plow ahead.

Down the hall in Ms. Drakeford’s math class, the bot’s responses to students sometimes seemed less like suggestions and more like direct answers. When students asked Khanmigo the fraction question posted on the classroom’s white board, the bot answered that the word “mathematician” contained 13 letters and that seven of those letters were consonants. That meant the fraction of consonants was seven out of 13, the bot wrote, or 7/13.
“That’s our biggest concern, that too much of the thinking work is going through Khanmigo,” said Alan Usherenko, the district’s special assistant for schools, including First Avenue, in Newark’s North Ward. The district did not want the bot to lead students through a problem step by step, he said, adding, “We want them to know how to tackle the problem themselves, to use their critical thinking skills.”

“Our engineering team corrected the A.I. a few weeks ago,” Khan Academy said in an email on Tuesday, “so that it no longer gives the answer to this question.” [However, we later] asked Khanmigo the same fraction question. In student mode, the tutoring bot explained the steps and then directly provided the answer: “the fraction of consonants in the word ‘MATHEMATICIAN’ is 7/13.” In teacher mode, which is designed to walk educators through problems and answers, the bot provided a different — incorrect — response. Khanmigo said erroneously that there were eight consonants in the word “mathematician.” That led the bot to provide a wrong answer: “8 consonants/ 14 total letters = 8/14”

Participating districts that want to pilot test Khanmigo for the upcoming school year will pay an additional fee of $60 per student, the nonprofit said, noting that computing costs for the A.I. models were “significant.” … Whether schools will be able to afford A.I.-assisted tutorbots remains to be seen. … the financial hurdles suggest that A.I.-enhanced classroom chatbots are unlikely to democratize tutoring any time soon. Mr. Nellegar, Newark’s ed tech director, said his district was looking for outside funding to help cover the cost of Khanmigo this fall. … “The long-term cost of A.I. is a concern for us,” he said.

My take is that the technology is too immature (and perhaps not even the right technical foundation) to rely on for this type of tutoring. Going all in on AI, like IBL education is doing, is probably not the right move at this time :p

BB-1496

3 Likes
2

I have a subscription to NYTimes, so here’s a gift link that anyone should be able to use.

3 Likes
3

It’s interesting… because I do see the latest iterations of AIs doing a much better job of “showing their work”: citing sources, explaining why particular answers were given. And really, it’s that kind of explaining that should be offered as tutoring, rather than just spitting out the answers. So it seems capable of it, maybe it just doesn’t know how to hold back the answers yet?

Yeah… computing will always get cheaper, but to keep getting better, AIs will always need to use as much computing power as we can afford to throw at them. (Aside: The Bitter Lesson).

We certainly live in interesting times…

4

I have spend a couple of years teaching in a primary school which completely changed my view of technology in context with education. I wrote multiple paragraphs as response here, but decided it probably isn’t the right place. So sticking to the topic.

I would say tech can help individual students and private/personal learning. But it doesn’t have a place in a public school setup. So even if it becomes mature enough to provide tutoring, it should only do just that - tutoring 1 on 1.

It would really be unfortunate if the AI hype train ends up taking more money from education budgets where teachers spend personal money to buy school supplies.

2 Likes
5

Thanks for sharing, and interesting :zap:

Very cool! I’d be curious to read what you wanted to share about your experience :slight_smile: Perhaps an off-topic thread?

6

There was an excellent keynote at the Open edX conference this year about the use of technology in eduction. The short version is that there has been a lot of hyped technology over the years but most of it has made very little difference and ultimately never got widely adopted since it doesn’t really help learning.

7

I got a slightly different takeaway-- it’s not that it didn’t help learning, it’s that instructors are looking for and select for software that helps them do what they’re already doing better, and they’re not in a frame of mind to change their entire paradigm of instruction just because a technology makes it possible and it might improve matters overall.

Like, there are institutions which have adopted learning technologies and are doing cool things with them that aren’t as easily done with traditional methods, but these are the (usually well-funded) exceptions rather than the widely available equalizing forces we might hope for.

The availability of AI technology will not change the incentives or values of administrators, politicians, or citizens. Some school districts are built to act as funnels of public funds into the pockets of education contractors. Others invest into the classrooms and teachers. Others have cash to spare for AI tech after taking care of school supplies.

It is greatly dependent on the district how well it spends its money, and if it’s not going to be on AI, it will be on some other toy. There are districts in the US that have taken to buying an iPad for every student, which is certainly more expensive than this. In any case, I wouldn’t look at this as any worse (or better) for the purpose.

Most of the e-learning tech that is developed tends to benefit people who are already well positioned. I don’t see this as necessarily bad, since if any person’s life is improved and the improvement value is greater than the cost (seems very likely it is, as we keep getting paid) and no one is harmed, there’s a net benefit to society. However it also means that this improvement is likely undercapitalized-- there would likely be much more improvement if these advantages could be more widely utilized, and to get there we need to better understand the actual clients and their needs, and the product must be aligned with them.

I think perhaps AI should be used in a more adversarial manner for instruction-- writing essays that learners must find a way to validate, for instance, or crafting arguments the learner should learn how to refute or undermine to practice debate. Some instructors are already doing this. Maybe one day the LLMs will get to where they are ‘hallucinating’ less, but to be frank, the LLM already makes a great imitation of someone chronically online with a good grasp of English who doesn’t feel the need to vet its sources very closely and doesn’t mind occasionally bullshitting to convince you. And learning how to engage with that sort of entity is extremely important in this day and age, and has always been important historically as well. It should be used to teach skepticism rather than fact.

LLMs may, in the future, be able to more readily make statements that we would view to be in line with consensus reality. However humans have only ever had a tenuous grasp of the truth since the beginning, and epistemology remains a discipline without a solution algorithmic or otherwise. If the LLMs get to the point where they always give answers the current experts agree with, they won’t teach humanity anything. It’s only when they say something that we don’t agree with that turns out to be correct, or something that is blatently false that makes us investigate and become enlightened, that we really learn.

Of course, this shows the REAL problem-- the fact that most schools (at least, the ones I’m familiar with-- other countries might be better at this) are oriented toward the lowest rungs of bloom’s taxonomy. Companies are trying to sell AI to ‘tutor’ because they want a way to get knowledge and basic understanding to kids quickly. But we’re in the information age. They can get information at the touch of a button. Kids don’t need information so much as they need to be able to validate it and synthesize it themselves. But as mentioned earlier, instructors pick tools for doing what they’re already doing-- and the LLMs can’t even get that right.

A company that builds tooling around using AI for teaching skepticism will be leveraging the technology very well indeed, and learners who use the tool may benefit greatly, but we can expect that this will not be widely leveraged since most schools are not teaching skepticism and argument. It will be sold to a much narrower audience, but one which may pay very well for it.

If there is an angle for teaching skepticism that can be sold to a much wider audience, that would be the most wonderful thing! However there could be systematic resistance to such a tool. It could be very disruptive to the classroom to have students asking their teacher to provide evidence for their claims, even if that is a highly desirable sign of educational development. If it could be introduced and have that effect slowly enough that removing it would be too arduous, that would be the best result.

8

@Fox Reading your post really brought a smile to my face. It was such a different take that I haven’t encountered before.

@gabriel The reason I didn’t post my thoughts is that, I have a unhealthy aversion towards the tech in the educational space because of personal experiences and as Fox points out, it is highly dependent on a specific location and the system around it.

This part kind of hits the nail with regards to my negative response around tech in education.

My experience has been that the “well funded” schools are sometimes well funded at the expense of other poorly funded schools. I am not sure about the US’s way to budgets. But from my limited experience, a school in a poor neighborhood doesn’t get the same patronage as the one in a better off location. Flashy things like computer labs, projectors per classroom (big deal in Indian public schools)…etc., get funded in one school, while asking for extra stationary needs to be made with the proof that the admin staff is not selling extra on the side to make money.

I agree that it is complex systemic issue involving politicians, administrators, school headmasters …etc., and tech is not probably not at fault here. It kind of accelerated or amplified this sort of class division and most of it had nothing to do with educational outcomes.

We weren’t tech averse, we had a classroom laptop and a printer which printed hundreds of pages every day. It really helped improve literacy and numeracy in kids who didn’t have a print rich environment at home. At the same time, I also saw teachers deferring to Khan Academy videos, for Maths instruction in English, for children for whom English was second sometimes third language and the schools celebrating the teacher for it. At the end each tool is only as capable as the person wielding it and teacher are asked (encouraged/noticed/positively incentivized) to wield a new tool every other year.

3 Likes