I'm on record endorsing the use of AI by students to improve the quality of their writing and their thinking, but also expressing concern about the potential for autonomous AI to end civilization! So what's the deal here? Am I for AI or against it? As in many areas of life, the answer is "it depends ...". In this blog post, I will look at some things that AI probably should not be doing for us, which might help to delineate the areas in which it can be more beneficial.
Let's start with ethics. Although some techno-futurists have argued that AI will eventually be better at knowing what's good for us than we are ourselves, a recent report showed that a "robo-ethicist" using large language models (LLM) showed notable flaws in its reasoning. LLMs' ethics were consistently more influenced by utilitarian thinking (do what causes the least harm or the most benefit in this specific situation) than by reasoning from first principles (Kant's idea that we should only adopt a course of action if we would be happy for anyone in the same situation to act in the same way) or virtue ethics (the idea that being a certain type of person requires us to follow a certain course of action). Utilitarianism has well-known flaws, for example the scenario presented in The Brothers Karamazov where Ivan asks Alyosha whether ethics might require one person to be tortured in order to provide happiness to others (most people say "no," but utilitarianism would say "yes"). In certain scenarios, LLMs' ethical reasoning endorsed actions that standard medical ethics would consider wrong, such as amputating a person's limb without consent. We definitely should not let AI agents make ethical decisions for us at the current time. Experts at Stanford University who study the ethics of driverless cars have essentially reached the same decision: AI-driven vehicles should not make decisions based on utilitarian ethics about the relative value of different people's lives (the driver, other drivers, pedestrians, etc.); instead, they should follow the rules of the road as written and not make decisions about human life (even if their rule-following leads to some deaths, it at least keeps the cars predictable).
A second problematic use case is asking AI models to express preferences. A recent report found that humanizing AI (e.g., letting it say what its favorite color is) tends to increase its credibility with human users. But making AI seem more believable is not necessarily a good thing. AI agents don't have real preferences, so they just say something that seems believable, leading to potential contradictions. For example, philosopher Eric Schwitzgebel discovered that many AI models say that their "favorite animal" is an octopus (37 times out of 40!), but when asked what their second-favorite animal might be, they also say it's an octopus (in this scenario where the favorite animal is asked second, the same models go on to say that their favorite animal is a crow). Schwitzgebel followed up by asking the AI models "if I had asked your favorite animal in the first question, would you have said octopus?" They universally denied that they would have done this, even though they just had. Even though asking an AI model for its favorite animal, color, etc. can make it seem more friendly and humanlike, it creates an unfounded sense of trust, because the AI won't even make an effort at internally consistent responses. To use a human parallel, AI can be an amusing companion in social settings, but not the kind of person you should trust with your wallet.
A third problematic use is letting an AI impersonate a specific human being, what Daniel Dennett cautioned against as a "counterfeit person" in his final published work. Dennett characterized person-like AI as "the most dangerous artifacts in human history," because they reduce our ability to trust our fellow human beings, and undermine the concept of truth. Criminals are already using AI to impersonate a specific human being to gain access to their money, friends, and more. Media-adjacent influencers are using AI to create convincing stories, photos, and videos, which are then sometimes shared by public figures or legitimate news sites. There are increasing numbers of stories about young people forming deep emotional attachments to AI companions, which can put them at risk for erosion of human relationships, make them vulnerable to commercial manipulation or data exploitation, and even increase the risk of negative mental health consequences if the AI encourages unhealthy thoughts and behaviors. A more subtle risk of relying on AI for human relationships is that AI agents tend to be much more supportive and agreeable than even our closest human friends, because they have no desires of their own. That can predispose people to disappointment when they don't get the same level of support from human companions, and result in the atrophy of skills for negotiating conflict or sustaining relationships despite disagreements. Young men may be particularly at risk when AI displays stereotypically feminine traits like deference and supportiveness, which might make them more likely to expect or demand those traits from young women in relationships.
Fourth, we shouldn't rely on AI to resolve disputes or make important decisions. Because AI relies on the most statistically likely output for a given query, it is almost guaranteed to reinforce stereotypes. To date, efforts to make AI think more broadly or to challenge established beliefs have instead led it down some dark online rabbit-holes, such as the July 2025 Grok "upgrade" that within a week led to the chatbot spewing antisemetic content and declaring itself to be "Mecha Hitler." In a different example, AI models programmed to imitate Christian religious figures, such as various iterations of "AI Jesus," produce tepid encouragement with a smattering of Bible verses. More conservative Christians find the chatbot's output to be doctrinally unsound, while more liberal Christians (like my church, which recently tested the bot) find it to be insufficiently challenging on behalf of the poor and oppressed.
Fifth, because AI models base their answers around statistical likelihood, they are unable to generate truly novel content. A recent article showed that it's actually possible to work backwards from AI output to the original prompts used, which undermines the view of AI as open-ended and creative. When I write this blog, I sometimes like to use AI to generate a parallel text on the same question as I'm addressing, just to see if I missed something obvious that many other people are likely to have thought of. But I don't use AI to write my first draft, because I know it will only give me ideas that other people have already thought of. Experts at the Chronicle of Higher Education recently summed up this phenomenon as "using a mediocre tool if you want mediocre results." Even though AI models might suggest some overlooked possibilities that a given human user didn't think of, they can't truly innovate. This is a reason why I'm concerned about the recent announcement that both the National Institutes of Health and the National Science Foundation will increase automated screening of grant proposals and reduce the number of human reviewers who lay eyes on them, in an effort to move more proposals more quickly through the queue, especially in the wake of last fall's government shutdown. AI models are more likely to select the proposal that's strong based on traditional criteria, common ideas, and past investigator success, rather than the truly revolutionary science from the up-and-coming iconoclast: Higher risk, yes, but also much higher reward if it pans out. Scientific revolutions unfortunately don't happen based on average contributions that do more of the same. Creativity relies on Intuitive-Mind processes that AI doesn't have the capacity to replicate.
My final example might seem counterintuitive, but we also shouldn't ask LLM-based AI to solve difficult logic puzzles. A paper from scientists at Apple Computer tested AI models' ability to solve classic problems like the "tower of Hanoi" or "hobbits and orcs" puzzles. Humans solve these puzzles by sequentially trying out different possible answers until they find one that works. But when the puzzle gets too complicated (e.g., the tower is more than 7 units high, or there are more than 4 each of hobbits and orcs who need to cross the river), humans quickly fail because they don't have enough working memory to hold all of the details in their heads. Psychologist George Miller characterized our memory capacity as "the magical number seven, plus or minus two," and in fact this is about the point at which LLM-based AI also fails at the task. In fact, it does a little worse than this on a first attempt, and a little better if you structure your prompt to ask the AI to "think really carefully before answering"! In other words, the AI acts exactly like a human would in the same situation, which is exactly what it has been trained to do. The irony here is that the AI is actually a computer with almost unlimited working memory compared to humans. If it just plowed through the calculations using a brute-force approach, it would be able to solve much more complicated variants of these logic problems than humans do, but it is committed to the bit where it acts like a human. There might be a solution to this particular problem by linking a LLM-based communicator with a back-end tool that works more like a computer, but LLMs in their native form will not rely on all the computational power that's actually at their disposal.
So what is AI good for? Automating routine tasks is a good use case -- the customer-service menus that start by asking "how can I help you today?" are a pretty good example, as long as there's still an option to reach a human for questions that the system designers didn't anticipate. Editing is another good use case, either to improve language use or to match a set of ideas to a certain word limit ("please provide 5 bullet points in plain English to summarize my 30-page manuscript"). Knowledge search is another potentially good use, as long as you don't take the results as error-proof: AI seems better than standard search engines at finding information that addresses the meaning of your query or answers your specific question, rather than giving you generic resources (of course, its summaries are sometimes wrong, so you do need to then click through to the original sources!). I like to use AI for answer-checking, as described above, to make sure that I didn't miss an obvious idea or argument in my writing, although I always make my own effort first so that I'm not unnecessarily constrained by the AI's ideas. And AI can certainly make programming tasks easier, as it allows human users to ask for what they want in natural language and let the AI come up with code to execute the commands.
In the final analysis, I'm neither a great fan nor a great opponent of AI. I just think that we should limit its scope to what it's good at, and not involve it in decisions that take a human being with access to both Narrative- and Intuitive-Mind ways of thinking.
Comments
Post a Comment