The End of Textuality

I’ve been turning over the post I wanted to write about AI — “What I really think about AI” was my working title. Recently I finally looked up some early attention I had given the subject for Lesbians Who Tech and was shocked to find it was 3 years ago, in the spring of 2023. Mark and I had just met which means we were launching Germ right into the beginning of the GenAI explosion.

I have a lot of blog posts adding up and I have to get them out the door. You’re going to see a lot of prose from me this month and it might not be as polished as you’re used to but I’m going to forgive myself in order to unburden myself to you. In the age of LLMs authoring everything we’re writing and reading to each other, what does it mean to be sitting on this original prose? I don’t know, that’s not where what my purpose is here. Or maybe it is—that’s the beauty of the essay. You know, the kind you write yourself? To get your thoughts out?

Here’s what I wrote in an early Germ newsletter in 2023, based on a talk I gave at LWT early that year:

If you follow tech news you’ve surely seen that ChatGPT is here and it can pass the bar, diagnose your rare illnesses, and make up citations and quotations as effortlessly as George Santos.

ChatGPT—and similar products like Google’s Bard—have been developed for fluency, not truth. They are supercomputer bullshitters. They have no relationship with semantic reality, only with grammatical reality. That is why Emily Bender calls them “stochastic parrots,” because they parrot stochastically, or randomly, based on the probability of what words will make up a conceivable-sounding sentence. Interestingly, Bender and her co-author Timnit Gebru, who was fired from Google’s Ethical AI team for exactly predicting this set of problems, were seriously concerned that these large language model (LLM)-based models were being designed to optimize for fluency and human-ness without sufficient attention to their correctness, relationship to bias or misinformation, and so forth. Well, they were right. These LLMs were trained on uncleaned swaths of internet text including, like, Nazi websites, and have already been shown to be wrong, racist, and sexist AF. [And if you’re interested in Big Tech’s general abandonment of information integrity, I will always point you back to Safiya Noble’s book Algorithms of Oppression.]

There is a lot more to say here about the wanton disregard for best research practices and responsible innovation and frankly human rights that are unfolding here but what I want to talk about is… Plato.

I’ve been thinking about Phaedrus .

When I taught my Stanford class “Hiphop, Orality, and Language Diversity,” I started the quarter with Plato’s Socratic dialogue “Phaedrus.”

In the dialogue, Socrates meets up with a young rhetoric fan named Phaedrus, and they take an erotically charged walk to a riverbank (sample line, translated not that loosely: “Is that a scroll under your robe, or are you just happy to see me?”) to talk about the invention of writing.

And while Phaedrus is bullish on the invention of writing, Socrates is bearish. Harkening back to the Egyptian myth in which King Theuth is skeptical of the invention of writing, Socrates recounts how writing’s inventor, Thamus, tells King Theuth, “I have invented a potion for memory and for wisdom.” Theuth rebuts:

“You have not discovered a potion for remembering, but for reminding; you provide your students with the appearance of wisdom, not with its reality. Your invention will enable them to hear many things without being properly taught, and they will imagine that they have come to know much while for the most part they will know nothing. And they will be difficult to get along with, since they will merely be appear to be wise instead of really being so.”

Socrates goes on suggesting that writing is deceitful because, unlike an orator, it cannot be questioned, and can travel indiscriminately, impervious to cross-examination and designed for massive misinterpretation.

I assigned Phaedrus to my hiphop studies students to get them in touch with the fact that writing is a technology, one far younger than oral literary culture, and to get them thinking about the fact that, when one is studying a rap song, the actual text is an auditory experience, and any written lyrics are just a partial representation of that ephemeral sonic text. As a writing educator by training, I find myself thinking about Theuth’s words and the worries about plagiarism that seem to have thought little about the problem, not of assessment, but of learning. How will a kid learn if they don’t write their own paper? How will people communicate when they no only don’t write, but probably don’t even read what they’re sending out? How will teaching and learning have to change so that people can engage critically with these tools while still mastering the skills of reading comprehension and writing that are already struggling?

While I have major disagreements with the AI boosters, I do agree with them that we are witnessing the deployment of a technology as important as the printing press and writing itself. And as Socrates knew 2000 years ago, a new literacy technology is not just “progress”; rather it changes the structure of human culture and intelligence, creating loss as well as gain.

And what I think we’re about to lose is authenticity, and authorship. Or, framed another way, the verification of authenticity and authorship is about to become a highly rare and contested process, as we lose the ability to easily trust that something was written or created by who we think made it; is a “genuine” image that represents an actual visual happening; or was even vocalized by the person we think vocalized it.

We are entering an era of profound fakery.

When I think about Germ growing into this era, I wonder about the role that authentication and verification will come to play on our platform. Germ is already designed to help users verify that they’re talking to who they think they’re talking to. Should we one day help them know whether their interlocutors had help composing their messages? Might we give users environments that help them discern when the words, images, videos, and sounds they’re receiving have been doctored? Will users of the future want that? What will authenticity even mean with five or ten years of these tools under our belts? What will these tools mean for authorship, authenticity, intimacy, and trust?

Only time will tell. In the meanwhile, we’ll keep building, keep critiquing, and keep making outstanding literary references.

At this time, generative text was just beginning to transform how we produced prose; the wholesale transformation of producing software was still, at least to the mainstream, yet on the horizon. Or perhaps that was my vantage point switching from one trade to the other. But I would extend my questions above, now as the cofounder of a secure communications software company, to ask—what might human-verified secure software mean to people in this new world?

*

What finally shook this post out of me was the story going around about how YA book editors are now habitually updating novels to include modern references.

Recall that this practice started with Mark Twain, racism and bias always these Trojan horses that smuggle authoritarianism into our culture. Because this is censorship and redaction. And then the method just spreads, like it does with everything.

But in a world where reading is becoming almost-always digital such that text is instantly transformable, what is textuality, what is a discrete edition of a text? As I wrote in 2023 without naming text/uality itself, what is an original, what is the authentic static text? In an all-digital textual world there is no original. Is there thus no authenticity?

No, because authenticity is a core practice of the human. Thus, how we negotiate it (because as hiphop studies teaches us, authenticity is a negotiation between performer and audience) is undergoing a complete renewal.

A friend’s startup is creating completely malleable UX for the agentic personal computer. In a world of agentic software that is always solving tasks for us in emergent, bespoke ways, software is no longer confined to static boxes and windows but instead becomes an evolving pixelated interface that responds interactively to the moment’s demands.

Working with characters as materials, we have invented software that processes information through characters but does not need us to engage anymore with characters. Returning to voice, we can now speak software into being using only our voices, never typing anything—a release from the embodied labor of producing text. But our voices are still our bodies, even if speech is more natural than writing. As voice becomes a tool of production, perhaps vocalization will acquire the technical specifications of other embodied labor?

As I type this on my laptop, an incredible thought: that the act of typing on a Macbook Air is becoming as antiquated as a scribe lettering a book after the invention of the printing press, or a town caller reciting an ancient text for hours after the invention of hand-written print.

We still memorize texts. And we still hand-letter them. But that is now called craft, a purposeful search after the revelations of technique. This is what’s happening to both writing prose and writing code.

In a world where every text is malleable and every digital representation of reality can be a spoof, how do we negotiate authenticity? What will come to be our shared understanding of the real?

When photographs were invented, people thought they stole their spirits. Today, people think their AI is alive. It’s not. It doesn't deserve respect the way a human, a river, or even an earthworm deserves respect. What deserves respect are the underpaid workers in the global South who labeled data and trained the models’ responses, the enslaved child miners who hacked the aluminum and coltan out of the ground to build the data centers to power the slopocalypse, and the people in Memphis getting emphysema and cancer from Colossus II.

But GenAI does deserve respect as a hyperpowerful technology that we create and shape and which is as transformative to human production and cognition as the invention of writing itself. And I still hold out hope we can build better models based on trained data that is grounded in the disciplines rather than undercleaned subreddits (please for God’s sake read Safiya Noble already), if we haven’t completely obliterated the arts and sciences in the next 15 years.

Unfortunately, we seem to be releasing hold of 3,000 years of desperately maintained civilizational knowledge just as we need it the most. MIT is closing 3 out of 4 campus libraries. American universities are shuttering humanities and social science departments at a rate that only matches that of the creation of women's and ethnic studies departments in the 60s and 70s—hundreds nationally a year. Of course, the parallel is not coincidence. We’re living through the end of the long Civil Rights movement.

Also recently there was coverage of scholars complaining against the new arXiv AI policy which will ban for a year with ongoing probation any author found to have any AI hallucinations or other infractions in their submitted work. The open complaints by working scholars is astonishing to me as a trained researcher because the whole purpose of research is verifying information to draw highly veracious conclusions. If you don’t verify the sources you’re using and the information you’re taking from them the entire logical proof of your argument falls apart. The whole university is based on the production of knowledge, literally a pile of proofs that have been painstakingly extended for three thousand years, since the great universities of Mali. And you just want to append synthetic data to that, it’s absolutely disqualifying.

So what we have here is a wholesale reorganization of humanity’s relationships to text, to information, to knowledge, and to authenticity.

This is beyond the transformation of our information environment because it describes the channels through which this corrupted information is broadcast to us.

In the 3 years since I wrote the piece above we have witnessed the cultural mainstreaming of dystopian sci-fi realities into near-term view, such fantasies as brain implants, synthetic friends, autonomous armed drones and robot armies, nonconsensual AR/VR, fetishized and nonconsensual body modification, genetic selection and engineering and genetic chimeras, cyborg realities and relationships, total climate apocalypse, the environmental catastrophicization of increasing numbers of large bioregions of planet earth.

In the same near term I can imagine a completely alphabetically illiterate technologist saying to a computer, “Show me our product’s growth this week,” never having in their life to have interacted with a character, only voice, shapes, and pictographs, in the same way a highly competent software developer today does not need to know how to build a computer from the mainframe out.

But in this world, who verifies that the inputs are correct and the calculations are correct? What happens when cuneiform is swept back into hieroglyphs? Is there ever verification? Do glyphs acquire new specificity of meaning? Does software just become sloppy, until it breaks? Will we have some kind of catastrophic global crisis where the software shuts down enough that we rededicate ourselves to quality? Will the built world just keep getting shittier and shittier, the elites continuing to recoil to their bunkers and penthouses like the Hunger Games, while the rest of us assume our places in the mines, the data centers, or the jails?

This is, for example, where UBI enters the chat.

But again, this is not where I’m going. And it might not go like that. Plato records that his teacher Socrates feared that by writing, we would forget. I fear that through emergent editing, we will inauthenticate. Maybe both did and will happen. But something new also emerged, a new way of being that those living through the shift never could have predicted, because it was and is wholly and utterly new.

What I wanted to unburden myself of here was my position, as someone not “supportive” of AI in the fetishistic sense that passes for technical literacy these days but as a literacy scholar trained in the history of writing, that this technology is neither ‘good’ nor ‘bad’ but rather a massive step-change in the nature of producing and circulating what was once called text and might still be called writing, or as I’ve used here even more elementally and to include the writing of code, characters.

In the way that writing—first pictographic, then phonetic and alphabetic—transformed trade and memory, and then that mechanized print transformed politics, creating nations and challenging the Church, I believe that emergent generative text will reorganize our society at the same scale. This is far beyond questions of upskilling or 20-year yields, but rather an epochal shift in what it means to record, to remember, to store, to verify, to articulate, to document, to reason, to represent, to interact, to query, to research, and ultimately, to be human.