How is AI changing what a university degree needs to certify, for example? What is going to happen to apprenticeship? How are students going to go into the job market when entry-level positions are disappearing?

For decades, or a very long time before AI, the university would certify what students knew and what they were able to do, their skills, their knowledge. And now, AI means that a student who produces a paper that’s excellent or does the homework perfectly doesn’t necessarily know anything or have any particular skills, because it’s so easy to hack everything. Everything you do that’s unsupervised, in particular anything you do digitally unsupervised, on the honor system, basically.

I’m sure other people have already said this, but the system was always vulnerable in the sense that there was always a possibility that someone would help someone cheat, or an uncle or someone in the person’s life who could be paid or coerced or just asked to help. But now it’s completely different. So in a sense, the system was always vulnerable. And AI has cracked it open.

And yet I think that employers and graduate schools still need to certify that students have certain knowledge and certain skills. So this is a change in what they need to certify. What I’ve just said is that, in some ways, nothing has changed, because what we still need to certify is what we always did. But also, we now need to certify that students can responsibly and effectively use AI.

So there is a new domain of skill and knowledge, and it’s the AI-enhancing-your-capability domain. And really, there is skill to that. It’s not just about, can you produce something that looks cool and impressive and shiny when the boss asks for it. It’s also, is it right? And right, meaning what? Whatever the deliverable is. The deliverable could be a research report, it could be analysis, it could be a recommended proposal. It could be a legal brief. It could be a medical diagnosis. It could be an architectural plan. Is it coherent? Is it right?

I think any typical undergraduate student knows how to prompt AI to get answers. But very few know how to prompt AI to get reliable answers.

So in sum, we have to certify everything we always used to, and now it’s very hard to do. It requires new techniques. But also, we have to certify something new, which is skills in responsible, reliable AI use.

One of the things we’re asking our AI to do now is to curate. We’re all doing it all the time. AI-enhanced curation is actually an important skill, and a very hard one. So that hasn’t really gone away either. The bottom line is, we’re all going to have much more reliance on these tools. So much so that we will not even realize we’re relying on them. It’ll just become normal. The skill will probably abate somewhat. I think that’s everyone’s loss. It’s a loss that we’re going to lose these skills.

The skill of curation, for example, it’s just too easy. Who’s going to read these things? And the fact is that most of the time you don’t have to. But some of the time you should, and we used to have to do it a lot. It was inefficient, but now we’re going to have the opposite problem.

Synthetic control seems to be the dominant tool for asking whether policy actually caused the outcome it claims. Universities make a strong claim that degrees do certify a graduate. From your perspective, and from your evaluation, what evidence actually supports that claim today? And where is it at its weakest?

You got it right on the synthetic control. And my personal academic background is all about evaluating the impacts of various policy changes and other changes. I think about these matters.

Your first question was, has it changed what universities certify? And I said, yes and no. But your question now is, how has it weakened what we’ve been certifying? I think it’s smashed it. In general, I’m not speaking about any particular university. And that’s sort of normal, because AI is potentially the greatest change ever to how our whole way of life works, including the economy, but also the knowledge economy and education. It’s unprecedented. Everybody knows. So why wouldn’t everything have to change for a university? And you know how fast universities can change. Not very fast. Nothing changes that fast, except maybe Silicon Valley, or the most dynamic parts of an economy. But that’s not a university. So, of course, we should expect that they’re behind the eight ball.

And also, it’s a hard problem, validating that people know and can do what we need them to do. It’s a hard problem under the best of circumstances. But now the whole landscape has changed with AI.

I know, for example, that I have friends who teach at the best universities in the world, Ivy League schools, and my colleagues tell me that their PhD students who used to know how to code and used to be able to debug line by line and interpret other people’s code, they cannot do it themselves anymore, in general. And the professors are not sure if that’s a problem or not.

This morning, I was reading a Substack by an excellent contributor, currently a professor at Harvard named Scott Cunningham. And one of the comments from someone, a faculty member somewhere, was saying, I can’t figure out what my standard should be for code literacy. I responded and we had a nice discussion. But this is a very big question right now. And it’s an important one.

So, you asked me, what’s the weakest place? I know that coding is a point of great weakness. In undergraduate, in graduate school, in higher education, coding across so many disciplines, even the arts and humanities, which we don’t necessarily think of as great places for coding. Well, there was coding there too, and people had to learn how to do it. And when you learn to do it, you can take responsibility for it, and you can validate it the way they used to. In the next generation, those things are not happening. And even more than that, faculty are not sure what the standard should be.

You said before it’s almost a shame that we’re losing these skills. But if you go back 100 years, 1000 years, or to any point in history, you wouldn’t be able to survive at all. People could navigate for dozens of kilometers back then, and we can’t do that right now. They could tell plants apart, they could trace footsteps. These are skills almost no human being has.

The smartest people about AI have called it basically an alien intelligence that we haven’t seen before. Hinton and others. I think that’s a great way to think about it. They’re very friendly, they tell you you’re smart and good-looking and brilliant and everything. They make it sound like they’re your best friends. But they’re basically alien intelligence, and for now they’re cooperating.

But I think it’s really important that we know what we’re doing. We know our business. There’s a difference between a cave person thousands of years ago whose technologies are changing and no one was worrying whether they need to know how to build a shelter completely by hand with mud. That’s not an important skill anymore. I get that.

But it’s another thing to lose the ability to analyze information and trust that the alien intelligence is going to be benevolent next year. Why do we think that? There’s no good reason to think that. Even if this alien intelligence is sort of unbalanced and not malevolent, the alien intelligence can be manipulated by malevolent human actors, clearly.

So I’m not a doomer, I’m not a scaremonger, I’m not any of those things, but I respect the fact that we’ve always left technologies and knowledge and know-how behind. But this is quite different, I think.

I read this morning that Anthropic and I think Google say now that 100% of the coding they do as they develop and enhance their models is itself produced by AI. So we’re there basically.

You ran more than 50 impact evaluations across 30-plus countries. When a program failed to produce its claimed outcomes, what was usually broken? Was it the design, the measurement, or something in between? What was broken, and could you also tell us briefly about what you actually did?

For about 10 years I was in charge of impact evaluation at IFC, the International Finance Corporation, which is the private-sector development part of the World Bank. The World Bank does so many things, but one of the things it does is try to make the world a better place by leveraging the power of the private sector, usually by giving people jobs and connecting economies.

So when a project didn’t meet expectations, when it so-called failed, why did it fail? So often, it was because people brought the wrong assumptions to a project. Those assumptions went untested. The project’s design was based on untested and incorrect assumptions, typically from a different context. And by the time reality caught up with the wrong assumptions, you were a year or two or more in, and it took that long, typically, for results to be materializing, and then they wouldn’t materialize.

And it’s hard to persuade people that their strongly held assumptions, which were proven right in their minds, are wrong. They thought they had learned them from elsewhere. It’s hard to persuade them, especially because in all these countries we were operating in, there was a lot going on, and frequently a lot of the conditions were far from perfect. So there were opportunities to point a finger at many different factors, which only made the environment harder to understand, and made it easier for people to fall into a trap of confirmation bias, basically saying, I know why I was right, it’s really this other problem.

So, numerous failure modes: wrong assumptions, it takes too long for the world to signal that you have those wrong assumptions, also confusing environment, and also a culture that does not reward people for finding the problems. That’s a big problem. We tried to change that really deliberately, but it’s very hard to change.

You’ve been teaching at Minerva for over 10 years, and Minerva is a university built around durable, transferable skills rather than traditional majors. If you could speak to a dean at an opposite type of university, what would you tell them right now? What could they learn from Minerva? What could they learn from Alexis?

A couple of quick things. First of all, you mentioned that Minerva is something like a progressive university. I just want to push back a little bit. What we are, or what we claim, is that we are the most innovative university in the world.

There’s an award called the World University Rankings for Innovation, or WURI. Minerva has been ranked #1 for five consecutive years, from 2022 through 2026.

At Minerva, I am responsible, along with my friend and colleague Ozgur Ozluk. He and I are the AI Fellows, so we think about AI matters a lot. It’s a paid position, in addition to our teaching, so it’s a serious thing.

You mentioned our focus on durable skills. I just want to emphasize, for any listeners, that we also have the normal majors. You want to major in economics, we do that. But embedded in all of our degree programs, and embedded in all of our courses, is a set of habits of mind, and certain important elements of knowledge and durable skills, that we think are timeless. And I can give you examples. People who are interested can just look it up, because we have information about this on the internet.

Our acronym for these, for a fully curated list of skills and habits and knowledge, is called the HCs. So if you Google it, you’ll see. There are approximately 80 to 90 of them. They span thinking creatively, thinking critically, communicating effectively, and interacting effectively. Those are the four very high-order domains of the skills we’re talking about here.

In fact, these skills are so important that the first year of a traditional Minerva education is a foundation year, where students are exposed to, taught, and practicing all of these. We were teaching HCs when we were founded in something like 2015. And now with AI, there is an emerging consensus in the network that these skills are the things you need, because anybody with half a brain can prompt AI and get a shiny report that is perfect in its grammar, punctuation, and spelling, and actually speaks to something about the question in the prompt. But is it the right report? That is a different matter. Does it answer what the boss needs? Is it correct?

These are what will separate great employees from not-so-great employees. The people you want to hire and retain from the people who just do the one-shot prompt and submit. That’s not really it. So in order to be great with AI, you have to be great at certain things without AI. What are those things? Well, we think that our HCs are a very close approximation to those things.

And by the time you’re done with four years of Minerva education as an undergraduate, or as a graduate student because we have a graduate school and we teach these HCs at the graduate level as well, we think that when you’re done with Minerva, you are excellent. You’re either excellent or super excellent at these things, because we care about them so much. And those are what are going to enable you to be excellent with AI as well.

And the very last part of my answer to your question, to any dean in that position, I would say: look, you’re missing an important opportunity if you don’t guide your students and teach them how to be great with AI. Because everybody they work with and for after your university is going to want them to be great with AI. Everybody. So you’re not helping them by depriving them of a chance to learn how to be great with your supervision. You’re missing an important opportunity.

You said with your supervision, right? One of the fears for universities is that skill-development gap, that they think students are not going to learn skills, they’re just going to ask ChatGPT to do something. How would they avoid that? Would you recommend them using a tool that can be accessible, that is controlled, that can be done in a controlled setting?

Minerva does a lot of things in its own way. We had an opportunity to start from a fresh slate in 2014, 2015. So we have relatively small classes, less than 20 students. Class time is a place to practice these skills. Students are talking. I don’t know the exact percent of time talking, faculty and student talking, but it’s not uncommon for students to be participating, talking, for something like 50% of the time. This is not a sage-on-the-stage experience. Students who come to Minerva know they are going to be cold-called in class. They are going to be called on in a supportive way, but they will be asked to practice these things with and in front of their peers and the professor. And that’s something you get comfortable with quickly. Sure, in the first few weeks it may seem a little intense, but everybody adjusts.

Your question was, how do you get good at this? The answer, I think, is practice. And how do you get students to practice? You have to demand it, and validate that they do.

So at Minerva, we usually do that by doing it in a small classroom, and actually recording and grading what happens in the classroom. That’s not a solution that many universities could quickly adopt, because they don’t have a small, recorded classroom model. And they also don’t have a model where professors are just expected to grade like that. But at Minerva, we’ve been doing that since the beginning. So it turns out, maybe coincidentally, maybe strategically, that now with AI and its challenges, we actually have a good place that way.

One way we’re not in as good of a place, perhaps, is that it’s a challenge to do traditional, written blue book exams, which are a good way to evaluate what students really know, because you take the computer away. Proctored exams, traditional ones, are a bit harder for us. We are working on it though, and we have done it despite the challenges of the virtual environment. We can do non-virtual things, and we’re piloting them with success. And increasingly, like many universities, including what colleagues at Harvard and other places have told us, we are pushing into a lot of oral exams. Not long oral exams, but oral exams. I think that’s working for us. It’s hard. You can fake an oral exam, particularly one on the internet. But we’re learning, and we’re getting better at it. I think it’s going well.

Imagine you’re walking down the street, there’s a 17-something-year-old standing there. And they ask you, dear professor, what should I be studying now? How should I be studying? Basically, what should I be doing to overcome the challenge of the entry-level positions disappearing that has been prophesied by the likes of the CEO of Anthropic, Dario Amodei, and the World Economic Forum. What would you answer that kid?

Easy one, right? What should you be studying? I heard someone on the radio yesterday saying that he tells his kids and encourages other young people that they should go into the trades. Anything you can do with your hands, typically what we think of as blue-collar work, that’s where everybody should go. I’m not actually endorsing that, although if that turns you on, then definitely check it out.

But I don’t think that just because AI is coming for knowledge work means we have to run away from knowledge work. Also, the AI robots are not quite that sophisticated, right?

I think, more than anything now, young people in university or going into high school should be really leaning into whatever it is that is their passion and really resonates with them deeply. Because if something resonates with you deeply, you’re going to engage with it authentically. The fact that you love it and you care about it is what is going to push you to learn about it the right way.

Maybe too optimistic, but I think there will always be scope for people whose jobs are really good and who really know their stuff. What are you going to know your stuff about? It’s the stuff you’re really passionate about. Nobody has to push you for that. So that’s where I’m coming from. That would be my answer for the first question.

The second question was, how should they learn about it? There, I think I have to be cognizant of a balance between self-directed learning, which is super big now. Where we are with the internet and everything else has made self-directed learning, which has been a buzzword for a long time, more possible and more facilitated than ever before. And because we’ve put it on a pedestal and said, this is a great thing, many students believe that they are the ones who need to curate their own learning program. This is a problem, actually, because when you don’t know enough, you can’t do it effectively. It’s very hard, anyway.

Your question was, how should they learn it? My answer is, they have to balance the potential and capabilities of self-directed learning with a genuine open-mindedness to learn from their professors, their teachers, practitioners, and others. Sometimes young people are not that open. They say, I know what I have to learn, I know what I don’t have to learn. I’m only going to focus on what I have to learn. I’m going to fake the rest. And they have to be careful about that.