This post continues my thoughts on AI as the opposite of education: you can check back there for a list of them all as they come together.

(Not so much) innovation

While the benefits of AI in learning and teaching continue to be debated, its transformational impact on research and innovation is widely accepted.

Here, I am going to deal mainly with scientific research. Innovation also belongs to the arts, humanities and critical social sciences, and I have plenty to say in other posts about the impact of AI in these, my ‘home’ subjects. But in these areas the AI industry tends to rely on brute force of capture and less on claims to innovation: nobody is claiming these subjects can go home and leave research to the machines. From the famous stochastic parrots essay to the Resisting GenAI in Writing principles, there are coherent arguments as well as cultural biases against the idea that statistical iteration over old forms can produce new creative or critical work. Creative workers have been among the few groups in the global North to organise effectively against the threat to their livelihoods, and theft from creatives is one objection to generative AI that is widely understood. In fact the AI industry no longer even pretends to respect creative work, attacking the laws that protect its value, and gaslighting creatives who stand up to them. Prominent among these, Hayao Miyazaki of studio Ghibli, once said of synthetic art:

‘Whoever creates this stuff has no idea what pain is whatsoever. I am utterly disgusted… I strongly feel that this is an insult to life itself.’

Still, scepticism about AI’s creativity coexists in the public mind with an absolute conviction that it is ‘revolutionising science’, and it is in scientific research that AI makes its strongest claims to transformative impact.

In research, we are mainly talking about specialised machine learning models that have been trained on a refined diet of meticulously prepared and filtered data, sometimes collected over decades, and not on the junk that goes into generic models. These models are also trained by scientists to ‘see’ the things that scientists are looking for: there are an infinite number of patterns in data, of which only a fraction are meaningful. Darren Acemoglu, an economist whose downbeat assessments of generic AI I have cited before, writes in Prospect magazine that the only hope for a positive outcome from AI is to focus on the potential of these specialised applications .Dan McQuillan pushes back on even these modest hopes. My position is somewhere between the two. I believe that expertly collated data, viewed with expertise, is often of value, but the value has to be carefully weighed against the costs, both the local costs of collecting and managing data, and training experts to work in relation to that data rather than in others, and the systemic costs of over-investing in data as method at the expense of many others - also of course the concentration of power and expertise that comes with data-intensive methods.

I previously wrote about the use of machine learning in materials science and drug discovery, where AI is predicting molecular structures in ways that may guide but do not replace the hard benchwork of discovery and testing. These ‘new’ molecules only exist in theory. And even at the molecular level, it seems now, the real world is messier than data models allow for. The much-vaunted protein-folding models get basic physics wrong and (from the same article) the ‘accuracy of these models may degrade substantially when predicting non-standard or previously unseen protein-ligand systems’. In other words, they cannot be used reliably for new discoveries. In climate change research, a recent review finds that machine learning techniques are not ‘viable alternatives to traditional numerical models’ and after decades of investment, the two approaches have still not been integrated effectively. In another high-profile case of ‘AI beating humans’, partial differential equations turn out not to have been solved after all. Even where machine learning has been used for decades to deliver efficiencies in research tasks, the benefits are in narrow parts of the workflow, results typically need to be verified by other methods, and the whole enterprise is costly. An overview recently published in Nature concluded simply that ‘an over-reliance on AI-driven modelling is bad for science’.

But the hype continues. Consider a recent headline, also from the Nature stable:

The article behind the headline provides a table of promising applications of ML in epidemiological modelling, none fully mature, and the best offering only ‘incremental’ gains. But besides overstating the potential impact, ‘help prepare the world’ recognises - without actually saying so - that it is up to other people to realise the actual benefits from any new insights. Despite the fact that ‘AI’ and indeed ‘machine learning’ have been around for 75 years, the benefits exist almost entirely in the conditional future tense, and in moving from the present diagnosis to that future solution, someone, somewhere has to do some actual work. Prediction is not amelioration, diagnosis is not cure. The gap must bridged by the mobilisation of people and resources, by investment, by embodied activities of construction and care.

And there, more precise knowledge of the problem is only sometimes useful. Public health measures, vaccine programmes and less exploitative farming practices are all preparations for the next pandemic that can be applied today, with technologies we have, and with benefits that have already been proved - even if the pandemic should not arrive. The greatest gains would come from applying these solutions in parts of the world that don’t have them already – and models are a poor fit in these left-behind regions (no resources: no data). So the problem isn’t a lack of precision in targeting, the problem is unequal resources. In fact, the big problems that AI promises to solve all have this same catch: the political and material resources to solve them still have to be mobilised.

Recall that the Bayesian statistics behind all this modelling were nurtured in the insurance industry and speculative finance, and consider the move that makes precise knowledge of the future more valuable than taking action for everyone’s greater security. Bayesian calculations certainly allow people with better predictions to make money. NVIDIA’s climate modelling behemoth Earth-2, a ‘digital twin’ built largely on public data, advertises its core users as Spire, JBA Risk Management and Climasens, all insurance and risk management corporations. With its own private satellite network, Earth-2 can be seen as a privatised double of the earth that increasingly extends into the real world in the form of data monitors and orbiting hardware, all to benefit the people who benefit from monetising risk.

The ‘good’ uses of ML in scientific research might again be thought of as new ‘lenses’ on existing data, guiding scientists to pay better attention to patterns that they in turn have guided the model to look for. Sometimes the value of data makes it worth investing in some massive shared project to collect it - think of the large hadron collider or James Webb telescope, an instrument whose history has become entangled with generative AI. But even where the new data is of great value, the concentration of that value in a single project carries risks. It tends to reduce the diversity of the research workforce and to harm the diversity of methods and perspectives, as well as (in fact, as symptoms of) concentrating expertise and funding.

Unlike the telescope, microscope or CT scanner, ML data is not being collected in real time. The lens is a historical one, and large-scale historical data sets carry additional risks of their own, that this paper identifies as: ‘representational harms’, ‘faulty science’, ‘over-generalisation/de-contextualisation,’ ‘poor labour conditions’ and ‘a lack of care for data subjects’. Philosopher Thi Nguyen reminds us that data has intrinsic limitations. To decontextualise means to choose what is worth recording as data and discard the rest. To categorise means to choose the meaningful features. To measure means to decide on the norms. ‘Data-based methods are intrinsically biased’, not accidentally so.

Entire research funding systems are now being skewed towards past data and their biases, especially in already-data-rich areas of research. A recent study on research careers titled ‘Innovations in Technology instead of Thinking?’ found that:

considerations about potential future academic revenues derived from innovative research technologies sometimes seem to override particular epistemic valuations.

Among the ‘potential futures’ and ‘epistemic values’ that may be over-ridden in the rush to machine learning are real-world experimentation and observation; the collection of data that does not fit existing parameters or categories; new contexts; methodological innovations beyond data crunching; and new theoretical paradigms - all drivers of innovation for most of the history of science. Armin Grunwald, a philosopher of technology, notes in a recent essay the ‘risk of closing down the openness and potentiality of futures to a model-driven prolongation of the past’.

And in every field where ML is prioritised as a method, the values of ML (‘Performance, Quantity, Efficiency’) produce opportunities for commercial capture. Earth data, like that collected by Nvidia’s Earth-2, is sold for commercial advantage. Google DeepMind retains the right to assetise patents arising from the use of its AlphaFold database although the data involved was all derived from investigative science. Microsoft and Google are competing to assetize environmental data, and there are intense efforts to control genetic and epigenetic data, with big AI recently investingly heavily in genomics and pharmaceuticals so that discoveries based on the DNA of people and other species can be assetised in exactly the same way as the art of Studio Ghibli.

Just as Miyazumi found AI art an ‘insult to life itself’, so the capture of ‘life itself’ as data portends new harms to all of us and to our common planet. Meanwhile the AI hype machine siphons precious resources from the foundational research and the putting-into-practice of well-researched solutions that we all need to survive and flourish.

(Not) ‘enhancing’ research

If machine learning has achieved less for research than is advertised, and at greater cost, still the generative AI industry has been desperate to associate their off-the-peg consumer products with those data-curated, expertly-trained specialist models. Lorena Barba, a professor of machine learning, collects examples of such hype and has rather wonderfully classified them as: ‘performance claims out of context, renaming old things, incomplete reporting, poor transparency, glossing over limitations, closet failures, overgeneralization, data negligence, gatekeeping, and puffery’ (Barba 2023 n.p).

With such a fine-grained bullshit detector available, there is really no excuse for universities to buy it. Leon Furze has done a great job of evaluating the so-called ‘deep’ research capabilities of generic LLMs (and if you don’t already subscribe to his work, you really should), concluding that they may be OK ‘for businesses and individuals whose job it is to produce lengthy, seemingly accurate reports that no one will actually read’. Anyone who engages with student writing at any level will be familiar with the miasma of credibility effects produced by these so-called ‘research’ models: fluent summaries that include real sources, though the summaries are repetitive and over-generalised and the real sources rarely say exactly what they are supposed to say. More than two years after Ted Chiang found that ChatGPT is a blurry jpeg of the internet, the billions spent on tuning it for ‘research’ effect have made it a blurry jpeg of Google Scholar, while simultanously turning both sources to absolute crap.

Research is not a general purpose activity and there is no general accelerant you can apply. A recent preprint explains that ‘The promise of foundation models relies on a central presumption: that learning to predict sequences can uncover deeper truths, or optimistically, even a world model.’ Cleverly, the authors devise a series of ‘inductive bias probes’ to test this presumption with Newtonian physics (and if you are thinking: ‘why didn’t they just ask the AI to explain Newtonian physics?’, please read my piece on why AI is not reasoning, and how very difficult it is to find out what it is doing at all). The ‘related work’ section of this article is particularly thorough and convincing. What the authors found, repeating findings from a host of other studies, is no. There is no deep understanding. There is no world model. There is just a ‘bag of heuristics’.

Nonetheless, the last year has seen a flurry of surveys from academic publishers – mainly the ones that have just sold our life works to big AI – purporting to show that researchers are excited about the opportunities of AI in research. Most of these surveys are so biased in design and so dishonest in reporting that I can’t in all conscience bring them to your attention, but despite a misleading headline and an odd discrepancy in the figures, Oxford University Press has passed on its results with good candour. Among its researcher participants (n= 2345), 27% claimed to have ‘benefitted from AI’ (not the 36% reported) and just 5% of these - a little over 1% of the total sample - mentioned using it for their ‘own work/analysis’. The other ‘beneficiaries’ had used it for research-adjacent tasks such as ‘to write more polite emails and to think of cool titles for my papers’.

Of the eight clusters of opinion identified in the OUP report, 933 researchers were negative about AI in research - it’s not obvious why this group has been split four ways. Another 811 were undecided, though we don’t know how this group weighted ‘saving time’ and ‘excitement’ against the rather more research-relevant factors of ‘distrust’ and ‘concern’. Just 345 were positive overall about the impacts of AI on research, and in true ‘frontier’ spirit these were labelled ‘pioneers’ rather than, say, ‘careless optimists’ or ‘reckless corner-cutters’.

Surveys of this kind deal in belief, and researchers are not immune from having their beliefs shaped by the direction of research funding, or by academic social media with its relentless FOMO, or indeed by surveys of their peers. And if your research goal is to publish an early AI use case in your field, you can probably get support from your institution and perhaps even industry partners to do this. Your local case study can be cautiously positive while your introduction and discussion project all kinds of opportunities into the future conditional tense, in which there is always room. There is little evidence that these cases do scale up beyond the favourable scenarios to which early adopters have applied themselves, but there are certainly rewards for producing them.

Titled ‘AI is revolutionizing science’, a recent report notes that ‘mentions of AI’ in published papers rose ‘from ten to thirty percent’ between 2015 and 2019 and that: ‘papers mentioning AI n-grams were roughly twice as likely to be a “hit” [highly cited] within their respective fields’. We can only assume these effects have been amplified since this work was done. Higher ‘mentions’ are glossed throughout this report as ‘AI benefits’, as though citation statistics are the benefits of research – a little like grades being the benefits of study. And just like in AI-generated literature reviews, ‘mentions’ must be pumped for all they are worth. Because generative models don’t attach qualities to instances, or explanations to correlations, so mentioning is their only move and counting mentions their only power.

For those whose research is not being boosted by AI ‘mentions’, the harms to research quality are now impossible to ignore. Those sharp AI commentators, Arvind and Sayash, recently asked the apparently paradoxical question ‘could AI slow down science?’ and concluded that yes, for all the frenzied activity, there is an overall drag on science as a shared system.

AI Snake Oil

Could AI slow science?

AI leaders have predicted that it will enable dramatic scientific progress: curing cancer, doubling the human lifespan, colonizing space, and achieving a century of progress in the next decade. Given the cuts to federal funding for science in the U.S., the timing seems perfect, as AI could replace the need for a large scientific workforce…

Read more

9 months ago · 147 likes · 20 comments · Sayash Kapoor and Arvind Narayanan

Because as well as cutting out all the parts of research that actually advance human understanding, AI is crapifying the shared systems that research depends on. There is an epidemic of fake research, fake citations, fake peer reviews, and whole fake research topics (‘vegetative electron microscopy’ anyone?). Google Scholar is flooded with AI-generated slop; academic publishers and editors are overwhelmed by AI-generated papers and even AI researchers; there are threats to the viability of archives and a creeping influence on library catalogues where the use of AI in metadata creation is now widespread. Open knowledge projects including Wikipedia, source of most actually useful AI content, are being drained of funds and even taken offline by AI crawlers, and the supply side of intellectual labour is failing. The peer review process, is corrupted by AI and now by cynical manipulation of the corrupt AI review system (this last link is to a study of papers in computer science that found hidden text prompting AI reviewers to provide only positive reviews).

There are calls of course to move to a 100% AI review system, because for every AI fail there must be an AI solution. And if AI could reliably provide that it would just be an extortion racket, but since it can’t, it’s more like paying the extortionists to burn your house down.

It’s been a while, and I’m sorry, but like everyone else I’m tired of AI. Dealing with the small local fires is bad enough without the horizon being constantly shrouded in smoke. It’s not as though there aren’t other crises. Austerity, redundancies, ideological attacks on forms of scholarship that don’t contribute directly to military or industrial advantage, ‘crackdowns’ on student visas and student protests (and if universities can ignore the warmongering and human atrocities, they still have to deal with the protests). Then there’s the collapse of democratic norms and public discourse, the very things that education is supposed to secure. Can’t we just agree some accommodation with AI and focus on these other, more existential issues?

Unfortunately these other issues are - to quote a recent keynote from Jennifer Sano-Franchini – ‘interconnected and co-constituted’ with AI. Where the AI industry is not actively contributing to them it is drawing energy from these other crises. Sometimes, as with Trump’s latest Bill, it is doing both at once.

So while I am tired of AI, I am even more tired of the cope. From ‘this is the worst AI you will ever use’ (readers, I did not agree) we are now deep into ‘this is the best it gets, deal with it’. Feel the mid and do it anyway. Meta’s Llama has hit a roadblock, GPT4.5 is underwhelming and Gemini 2.5 faces a backlash from angry developers. Inaccuracies and fabrications (classified in this article as ‘botshit’) are actually getting worse, perhaps due to a training diet of AI-generated slop and synthetic data. Even the true believers admit that AI is running out of data and scale won’t improve things anyway. This last link, from AI fanaticist Toby Ord, really does say the quiet part out loud.

Let me translate:

Since 2020 it has been known by the industry that scaling up AI would make catastrophically increasing demands on power and compute, while returns on model accuracy would quickly tail away. But the industry is betting that the financial returns will continue anyway, because everyone will be locked in.

Yet every time the mask slips on the ugliness, greed and rapacity of the AI industry, whenever the lies and misdirections and fabrications become unsustainable, come the grown-ups to tell us to calm down and carry on. Because polarisation is unhelpful. No matter that on one side are the four or five largest corporations that have ever existed, the biggest bubble of financial over-investment, the most powerful military and surveillance states and all the combined forces of tech hype and mainstream media, while on the other side are thoughtful people with arguments. The good academic must always plot a middle course between naked power and poor thought.

But what if there isn’t a middle of this road? What if the project of ‘artificial intelligence’ is not a road to new kinds of education - not even a slow and bumpy one – but the reversal of everything education stands for? What if, at at least in its current, (de)generative, hyper-capitalistic guise, the project of AI is actively inimical to the values of learning, teaching and scholarship, as well as to human flourishing on a finite planet? In my next few posts I consider the evidence.

The links will go live as I add them. They are, as always, imperfect thoughts. I wait for counter arguments. Please also let me know if there are other propositions you’d like me to include.

Thoughts so far:

(Not) telling the truth

(Not) reasoning or explaining

(Not) developing expertise

(Not) learning

(Not) teaching (either)

(Not so much) innovation

(Not) enhancing research

(Not) education as a right

(Not) education as the practice of freedom

(Not) thinking

(Not) telling the truth

Generative AI doesn’t tell the truth. According to the Association for the Advancement of Artificial Intelligence, in a factual accuracy test from December 2024 ‘the best models from OpenAI and Anthropic correctly answered less than half of the questions’ posed. Other concurrent studies also found AI outputs to be wrong at least half the time and with so-called ‘reasoning’ models things are even worse. That’s when they aren’t being used deliberately to spread conspiracy theories and encourage delusions.

I fully agree with John Noster when he says that generative AI is a ‘counterfeit’, but I want to push that observation beyond the everyday meaning of the word. Yes, AI is a fake, but more importantly, it is making (feit) in a way that is counter to - opposed to, inimical to, actively working against - the makings that people engage with together in order to flourish together. In being ‘fluent, convincing [but] fundamentally ungrounded and untethered to our humanity’, as Noster rightly exposes, AI productions extract value, attention, investment and care from the conditions in which people can decide what is true and what is worth believing. Generative AI is not making knowledge in an interesting new way, producing some unfortunate biases and incidental untruths that can be overcome by people with ‘AI literacies’. It is a mode of production that captures, devalues and finally exhausts the ways of making knowledge that belong to people and, in return, offers up only past representations, rehashed through forms of hyper-exploited and alienated data work.

‘Artificial intelligence’ itself is founded on a lie. It does not exist, or not as its key players define it. Sam Altman, for example, has said for years that the goal is a system ‘more capable than most humans across all areas of intelligence’. He announced a ‘breakthrough’ towards this goal in November 2023, confidently predicted human-surpassing AI ‘within a year’ in 2024, and in 2025 has postponed the rapture until 2029, though still confident that ‘Superintelligent tools could massively accelerate scientific discovery and innovation well beyond what we are capable of doing on our own, and in turn massively increase abundance and prosperity’, although AI was supposed to be doing that back in 2022. What has now been renamed ‘artificial general intelligence’ or ‘strong’ AI is impossible on principle, according to many philosophers and computer scientists (Luciano Floridi, Herbert Dreyfus, John Lanier, Ragnar Fjelland, Francois Chollet and Iris van Rooij among them). It is economically unrealisable, according to an entirely different group of experts (Richie Etwaru, Darren Acemoglu, Ed Zitron, and The Economist). It is unachievable with current techniques, according to most AI researchers, and recent polling has found that even if it was possible, available and affordable, most people wouldn’t want it. And yet it is in pursuit of it we are asked to put up with everything AI can offer in the present.

Behind that one big lie come a hundred lesser lies like beads in a row. AI will make work more creative and rewarding. AI will make workers more productive - but somehow that won’t mean a choice between accelerating your work rate or losing your job. AI will help students learn (‘reduces cognitive load’). AI will stop students learning (‘reduces cognitive effort’) but don’t worry, we know the difference. Don’t use AI for this, because it’s cheating. Do use it for that exactly same thing because it’s future proofing your skills. We can definitely tell when students are using AI. We aren’t using AI. We may be using AI but only when it is safe, responsible and ethical to do so. All our platforms and AI enterprise systems and agents and co-pilots are safe and ethical too and we absolutely 100% have audited that.

Big tech will be fine. It managed to push on through the MOOC lie, the metaverse lie, DNA testing scams and non-fungible tokens and crypto. Ok, the crypto lie is thriving under the Trump dynasty, but only as a manifest corruption scheme. Lying is business as usual for big tech. But education is supposed to have a rather different relationship with the truth.

Probability is part of the problem. The Bayesian probability that machine learning systems are based on does not, like the frequentist probability we were taught in school, express the probability of something being true. Rather they express the value of a particular prediction about the future, assuming that prediction were to pay off against alternative bets. Model training is ‘adversarial’ in exactly this sense, betting different states of the architecture against each other to find which produces the best, least ‘lossy’ outcome. In the world of financial speculation, where Bayesian methods have their most enthusiastic and indeed value-generating users, the bet you make does not matter so much as your power to make the market in which your bet turns out to have been the right one. Speculative financiers have bet big on generative AI, knowing that it does not and cannot produce any value in the usual sense of profit until knowledge production is substantially turned over to AI. But those investors are relying on the rest of us to make that market for them.

Education leaders who insist on the adoption of AI are actively producing a future in which the bet on AI turns out to have been the value call.

Of course education leaders may have reasons to prefer the AI future. Far more likely, though, they have become convinced that no other future is worth believing in or striving for. Like other situations where the truth is obvious but uncomfortable - and we don’t have to look very far in the contemporary milieu - there are personal costs to speaking up. But in some possible futures, the costs of not having spoken up may be high - the costs of complicity, for example. So what starts out as a wholly understandable reluctance to speak unpalatable facts becomes a commitment to producing the future in which that bet was the right one: the ‘facts on the ground’ have been transformed, the nay-sayers were mad or bad or have shut up and gone away, and we are living in the only future that was possible.

Education leaders who want to hedge their bets even a little at this point could start by telling the truth about the crises unfolding from the release of ChatGPT and other large language models. The targeting of students as users and the capture of educational researchers and policy makers has been relentless. No other group of people (with the possible exception of PR professionals) has adopted ChatGPT as thoroughly as students have or is using it so indiscriminately. We may not have answers but we could share truthfully with students that our practices of teaching, learning and assessment are being broken and that we need to find ways of continuing to learn together. We could stop pretending this is all OK.

The truth has a hard time in parts of academia, some of the parts where I feel most at home in fact. Universities - literal palaces of ‘one truth’ - have been complicit and remain complicit in projects of extraction and exploitation in the name of reason, not least in recent times through their partnerships with the AI industry and its military correlatives. But if this imperial certainty is to be overthrown, please can we hope for some more thoughtful kinds of uncertainty in its place? Some better accountability to history, some more equitable relations among knowledge cultures, some honesty in self-examination as well as rigour in methods. Can we not just put up a new tyrant to insist that truth is relative (to power) and nobody can believe the evidence of their own senses?

Because in a world of live-streamed genocide and hourly lies from the highest authorities, who does ‘post-truth’ really serve? We are not witnessing a scholarly debate about the plurality of interpretations here but a fascistic turn to revelation, authority and affect. While predictive AI is deeply implicated in the oppressive violence of Israel, China, Saudi Arabia and the US military, in state-sanctioned sectarian violence in India and in the violent deportations of Trump’s ICE, its generative branch is helpfully putting all the evidence of these crimes beyond belief. Experience is de-realised: everything smells fake. And when we append our advice to students with that small disclaimer ‘AI may be wrong’ we are teaching them that this doesn’t matter. Give up striving for truthfulness. There is no important connection between words and accountable actions in the world.

‘AI’ is a scheme of misdirection. The original Turing test and all of today’s AI benchmarks lay it down in black and white. How many people can you fool? That is literally the value proposition. And it is essentially the same question as: how many people can you enthral with your power?

Data power, military power, political and media power, computational heft… as Alex Karp of Palantir has said (not quite quoting Humpty Dumpty), ‘AI is a weapon that allows you to win’. All power, no truth.

(Not) reasoning or explaining

Most scholarship does not aim at anything as ambitious as the truth but does aim to apply reliable methods and to explain the results. Methods are fallible but they are accountable, in the sense that other scholars can review their use and replicate or challenge their findings.

Generative models do not allow for such accountability. Their inferences are non-replicable. Their ‘black box’ architectures are constructed through trillions of discrete operations on billions of data tokens and tracing even a fraction of their hidden structure requires probing methods that are specialised, laborious, poorly understood and imprecise. On top of the models’ intrinsic obscurity there is deliberate secrecy over the data sources and the human data work involved pre- and post-training. To produce ‘chain of reasoning’ style interactions, for example, data workers are used to rate model ‘explanations’, to provide sample explanations for model refinement, and to build whole ‘explanation’ datasets for model training. User preferences are also gathered and re-used to make ‘reasoning’ sound reasonable. You thought it was just the maths talking?

Unfortunately the more models are tuned to ‘explain’ their ‘reasoning’, the more errors they introduce. So-called chain of reasoning (CoR) does not express how a model is actually ‘reasoning’ because ‘models are not reasoning’ (Apple 2025): instead, as the same Apple team demonstrated in a 2024 paper ‘they replicate reasoning steps from their training data’. CoR reports are untrustworthy on principle: they are plausible explanations for plausible responses, and since the inferences involved are more complex, they burn more compute and carbon per query as well as introducing more mistakes. With complex problems, the supposed ‘reasoning’ can collapse suddenly and unpredictably, while with simple problems the ‘reasoning’ model can continue ‘its computational wheel-spinning’ long after a solution has been reached. (I wrote about this ‘halting problem’ in an earlier post on the Turing test).

People need explanations. By explaining we develop rubrics, models, theories and schemata so we aren’t starting afresh every time we meet a new situation. In research, methodological clarity matters so we can invite others to query our reasoning. In teaching we want students to arrive at explanations for themselves, as part of building a mental world, and to test those explanations in different contexts. But we face an advancing doctrine that what these data models are doing, obscure as it is to human understanding, is a better route to knowledge – faster, more ‘objective’ and more scalable – than the old evidence-and-reason routine. Data models are vast enough to contain approximations to every new case. Data does not need to generalise. As Chris Anderson notoriously argued in Wired (2008):

Petabytes allow us to say: "Correlation is enough." We can stop looking for [explanatory] models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

The ‘we’ in this sentence are not learners reaching for new understanding, or research communities trying to build useful shared knowledge, but people with ‘the biggest computer clusters’ and ‘petabytes’ of data, whose answers must now be accorded authority despite (or perhaps exactly because of) their resistance to human-readable justification. Along with post-truth, data models usher in an era of post-science, post-explanation or more broadly of post-theory in which whole cultures of human reasoning are less valued than data architectures, whose inferences people can subscribe to but never share in, still less question or debate.

Among the rebutters of Anderson’s thesis, philosopher Byung-Chul Han points out that when we give up on some correlations being more meaningful than others, we give up on reasons and reasoning altogether. Absolute information (if it were possible) would be the same as absolute ignorance:

Navigating one's way through Big Data is impossible… Dataism arises from the renunciation of meaning and connections; data is supposed to fill the void of meaning. Han (2013)

I respect Han’s philosophy but I prefer a different argument. Data models are not unreasonable, to my mind, but they are new regimes for coordinating the work of reason. Data and datasets are the products of reasoned collation, curation and labelling. The data workers involved in model refinement are making sense – interpreting, evaluating, comparing, deciding, often completely rewriting outputs. AI engineers have human sense-making in mind as they guide models towards ‘desired’ states - those that seem (to their users) reasonable. And finally, those unpaid users must endlessly check, correct and compensate for the ways that models still fall short of good sense in their contexts of use. It is in the interests of the AI industry to obscure this reasoning work behind the mask of data and algorithm, but in truth LLMs are only useable as a regime of reasoning because they involve human reasoning at multiple points.

But these disparate reasoners don’t constitute a community in the way that a university department does, or a local history group or a fan fiction site. They are separated from knowledge of their own knowledge, which becomes knowledge only within the regime of the data architecture. They are separated from each other, working on different terms, unable to reach a consensus or agree on a shared project or even to know what anyone else is doing. The low entry bar for using AI allows the industry to claim it is more ‘accessible’ than a university course or a difficult book, but beyond this low bar is a rigid division of labour and a structurally-enforced lack of visibility and mobility. At least the research community has to justify its openings and closures, its codes, procedures and rules.

Just as with ‘the truth’ we don’t have to believe that regimes of knowledge production in education are open and fair to believe they can be made more so, and that they work better (reasons are better justified, knowledge resources are better used) if they are as open, fair and diverse as possible. The open education community has promoted these values for many years, one reason why in a recent keynote I insisted that ‘AI is the enemy of open’ in education (another thought - full recording coming soon).

The AI stack is not a community. It is not amenable to dialogue, critique, consensus-building or action to challenge injustice. Its data architectures are impenetrable by nature, obscure by design, unaccountable by the ruthless prerogative of commercial power.

Technology, as Peter Thiel has opined, is ‘an incredible alternative to politics’ as a mode of power. It has made some people so powerful they can put themselves beyond reason and explanation entirely.

(Not) developing expertise

Even if they take the most transactional, neoliberal view – that the point of education is to develop economic capacity via people with expertise – education leaders should examine what the AI project means for that transaction. The goal is for expertise of the kind currently accredited by the education system to be represented, organised and operationalised through data architectures. While the jury is still out on how far this is possible, given the poor showing of these models in real-world contexts of expertise, workplaces are already restructuring workflows as data flows so that AI agents and APIs can be made to ‘work’. This redesign is not incidental. Following Stafford Beer’s ineluctably materialist logic, the purpose of a system is what it does - and as I have argued from the start, the purpose of the AI system is to redesign work, especially what is called expert or ‘knowledge’ work (though, in my view, all work is also that).

Anyone who followed ‘artificial intelligence’ through the era of ‘expert systems’ will be reasonably familiar with its failures, and may even know some of the reasons advanced for them. Most famously Herbert Dreyfus, philosopher and AI critic observed that representable knowledge is only one element of expert practice (that is, what experts actually think and do when their expertise is being put to work to produce social value). Other elements include embodied routines, situational know-how, ‘perceptual expertise’ or the way that senses become re-calibrated (experts notice different things), remembered experiences, the values and norms of the expert culture, and the ‘feel’ or flow that comes from integrating all of these fluently. Dreyfus was also interested in purpose - why certain things are worth doing and thinking about, and what animates sustained projects such as learning to become an expert. Representable or conceptual knowledge is still a crucial element of expertise - but even managing representations and concepts involves know-how, that can itself become more expert and specialised.

Now there is evidence, summarised here by Punaj Mishra, that the use of generative AI by experts can actually make them less productive; astonishingly, this is even true of that most happy user, the professional coder. Interruption to the flow of practice is a drag. Also, did I mention, generative AI is crap and monitoring the crapness is a further drag on time, attention and productivity.

Generative AI does not serve professionals well. It can be used to offload some of the standardised, repetitive tasks that would once have gone to junior colleagues. But who exactly does this serve? Not the established professionals who are expected to be more productive while ‘minding the AI’ for errors and failures; not the younger workers whose roles are squeezed out by ‘AI’ (or, more accurately, by data workers in the AI labour stack) and who miss out on valuable learning as a consequence; and not the professions whose knowledge is being siphoned into data structures that they must lease it back from. Not even the workplaces that have paid for the AI and must now pay to make their own data ‘AI ready’ and AI-secure, so having less to invest in people with talent and expertise.

The whole point of cognitive automation is to turn expensive, specialist expertise into data work, some of it to be done by experts themselves (fewer of them, working harder) and the rest by more generic workers who can be casualised. In fact as generative models fail to improve by the magic of mathematical emergence, the volume of data work being done behind the scenes is actually increasing. It is also getting more specialised, meaning that expert work is becoming data work. AI labs are ‘hiring thousands of programmers and doctors and lawyers to actually handwrite answers to questions for the purpose of being able to train their AI’, according to Mark Andreessen, who should know. Its continued reliance on human experts to pull the strings of the ‘smarter than human’ system is why Meta recently acquired ScaleAI, a datawork outsourcing company notorious for its exploitative labour practices.

Whether captured by outsourcing companies or held in the databases of business users, expert knowledge is being captured, to be put to work again by other experts - because it is only when that knowledge is put to work in human contexts and for human purposes that it acquires actual value. But more of that value is now falling towards the companies that own the knowledge-extraction-and extrusion machine.

For the whole bait-and-switch to work, expertise must be made more representable at the one end (that is data work) and more exploitable at the other end (that is the demands for greater productivity through datafied processes).

The neoliberal university ought to understand all this, since the economy is its raison d’etre. It should be able to explain why the wider economy shows so little AI impact on earnings or efficiency, so little AI-enhanced growth and productivity beyond that most capital intensive sector the AI industry itself. It should have predicted that AI adoption would be hype-driven and shallow rather than deep and structural, and that the majority of workplace AI projects would end in failure while every thinktank repeats the mantra that ‘work must become AI, so AI can work’. The neoliberal university may not have many philsophers left to explain the epistemic fallacies but it is swimming in business analysts who could advise about the economic risks. The general risks to the world economy of AI’s over-financialisation and under-delivery of value, its diversion of capital away from other economic priorities such as energy transition in favour of computational infrastructure, and its ponzi-scheme investment structure that relies on the rest of the economy conforming to AI and its demands before anyone makes any money. And the particular risk that – if it succeeds – the value of people with expertise and therefore of universities will be pushed to the floor.

Given the colonisation of expert knowledge by computation, it’s hard to see why societies and/or young people in them would continue pay for qualifications that might or might not add value to work in the expertise-data system. Major companies already prefer to gather their own evidence that applicants are ‘thinking for themselves’ and piece-rate workers qualify themselves by showing they can fulfil sample tasks before they are assigned to paid ones. Neither requires a university qualification. If data work is the best most graduates can hope for – and many students and recent graduates are doing it already - they might as well learn-to-earn within the data engine.

Some elite universities will no doubt continue to thrive in this scenario, promising innovation (from research outcomes, machine learning PhDs) and ‘top experts’ whose know-how will be propagated through the databases and architectures associated with their respective fields. People with expertise will continue to be the source of economic value and the drivers of innovation. It is mass expertise that is under threat, the aspiration of the majority of the world to become specialised and qualified, to become fluent in particular knowledgeable ways, and to secure better life prospects from that.

Look out for more links in this series! And check back soon for a relaunch of my podcasts with more ways to listen.

I was inspired to submit a thinkpiece to this UNESCO call by a group of people whose own submissions (I happen to know) are wise, original and inspirational - so I’m truly grateful to have had their support and conversation. You can read all our contributions at this linktree, and catch up on a live panel we held recently to discuss our contributions (not organised or sponsored by UNESCO). Meanwhile, these are my own reflections - not my submitted piece which we are asked not to publish.

What is the right to education?

The ‘right to education’ is enshrined in the Universal Declaration on Human Rights (UDHR, Article 26). But there are many cultures and definitions of education, as other Declarations and Conventions emphasise. ‘Education 4.0’, according to the World Economic Forum, ‘places the responsibility for skill-building on the learner’, with teachers as ‘facilitators’ of this self-determining enterprise. This doesn’t sound like something ‘rights’ apply to, except perhaps the right of consumer redress if your facilitation-as-a-service teacher fails to satisfy. Nor has the AI industry been slow to promote its own version. Elon Musk encourages students at his privately funded Astra Nova school to address problems such as ‘How should Tesla build out a supercharger network in South America’ - so Musk’s money makes him free to determine the contents of at least some children’s education. But how many people in the world does this ‘right’ extend to, and with what consequences for other people’s rights? Meanwhile JD Vance, VP of AI, rails against ‘teaching’ as ‘brainwashing’ and ‘indoctrination’ and calls for ‘our kids’ to be ‘defended’ from it. In the world of Musk and Vance every self-governing homestead should be free to decide what their young people need to know: that is, after all, what patriarchs have always done.

But the liberal-democratic version of education that Musk and Vance dislike so much is exactly what the UDHR and associated Conventions intend: education is a right of all people as equals and a responsibility of the public bodies who represent them - states, nations and regions. The context can only be a democratic one. And despite the difficulty of accommodating so many different epistemic cultures and pedagogic traditions, the UN’s International Covenant on Economic, Social and Cultural Rights (ICESCR, Article 13) does have something to say about the contents of that education: ‘education shall be directed to the full development of the human personality and the sense of its dignity… to enable all persons to participate effectively in a free society, [and] promote understanding, tolerance and friendship among all nations and all racial, ethnic or religious groups’. The UN Convention on the Rights of the Child (CRC, Article 29) prefers: ‘the development of the child’s personality, talents and mental and physical abilities to their fullest potential’ so that the child may be ‘fully prepared to live an individual life in society’. The Convention Against Discrimination in Education (CADE) adds a host of further Articles reinforcing the equality and universality of these rights.

That expression ‘an individual life in society’ expresses a tension in these universal goals, one that all educators have to confront at some time in our careers. Education aims at the fullest possible realisation of the individual, but on terms given by society: what that society holds to be a life worth living, knowledge worth having, and work worth doing. By equipping young people with what is collectively valued and known, education shapes their structures of thought and developing personalities. But learners must also be granted the agency to respond uniquely, to transform their conditions as well as themselves, and to resist social norms:

‘Education changes people; people change the world’ (Paulo Freire 1972)

There is other tensions here: that while a child is a full human being from birth, endowed with human rights, still s/he is not yet a full person. It is the task of education - in the broadest sense - to hold that space of immanence and vulnerability, to give shape and expression to what is immanent, and care to what is vulnerable.

‘Education’ is not the only such ritualised space but it is how the United Nations have designated the support that children and young people need for their emergence into full personhood and social participation. Primary education at least must be ‘free’, secondary and higher education ‘freely available’. This collective investment means that education must be accountable to the rest of society: the purposes of education are held publicly, even if private interests might be involved in realising them. And there is a final tension. While respecting national and cultural determinations about what it matters for young people to learn, the demand that education should ‘promote tolerance and friendship’ between ‘all nationals and… groups’ is a strong sign that it should be outward-facing, inclusive of difference, and orientated towards tolerance and justice.

How can data and algorithmic platforms infringe these rights?

Data and analytics platforms, including so-called predictive AI, have been criticised for undermining the rights of young people in education through the capture and unregulated use of their data, and through opaque decision-making that exacerbates discrimination in educational settings and amplifies inequality of outcomes (Global Connectivity Report (2022) Ch.9; Williamson et al (2023) for the National Education Policy Centre).

UNESCO’s report An Ed-Tech Tragedy? (2023) found that during the Covid-19 pandemic ‘education became less accessible, less effective and less engaging when it pivoted… towards technology’ (p34); that this was accompanied by ‘a concerning transfer of authority away from teachers, schools and communities and towards private, for-profit interests… the censorship, data extraction, advertising, top-down control, intimidation and surveillance that so often characterize current models of digital transformation have made education less free’ (p.35).

On behalf of the Council of Europe, a Briefing on AI in Education through the lens of Human Rights, Democracy and the Rule of Law (2022) concluded that ‘We need appropriate, robust regulation, addressing human and child rights, before AI tools are introduced into classrooms’ (p.75) and ‘We need to ensure that children are not forced to accept being compulsory research subjects or being compulsorily involved in product development simply by exercising their right to education’ (p.76). Recognising these risks, the Global Digital Compact of the UN commands: ‘digital technology companies and developers to respect international human rights and principles, including through the application of human rights due diligence and impact assessments throughout the technology life cycle’.

None of these regulations, due diligence processes or impact assessments have been carried out. Far from submitting to new requirements, AI corporations have bribed or bullied governments to weaken their existing rules and tax regimes. And the threats to learners’ individual rights - the focus of most existing legisation - must now be set alongside the systemic risks of an industry that has grown vastly more wealthy and powerful than most nation states, let alone their education sectors. At current valuations, OpenAI, Nvidia and Microsoft are each worth as much as the GDP of the entire African continent. These corporations are powerfully aligned with the US state and its global interests, and the same is true of AI industries in other powerful, militarised states such as China, Israel, Russia and Saudi Arabia. AI systems create a seamless geopolitical infrastructure that can threaten the educational rights of whole populations (the use of AI to support school and university bombings in Gaza) as well as vulnerable individuals (the use of AI to identify and deport students protesting these issues in the US).

AI as a geopolitical infrastructure has the capacity to inflict global epistemic and educational harms.A rights-based discourse must therefore extend beyond classroom infringements of children’s rights to concern itself ‘with the preservation of life and the co-responsibility of AI harms to the majority of the planet’ Ricaurte (2022)

What are the specific risks from ‘generative AI’

The risks outlined in earlier reports on technology in the classroom were immediately amplified by OpenAI’s release in 2022 of its untested, unreliable, unsustainable data architectures in the GPT series and its ChatpGPT interface. Beyond building and commanding an immensely profitable new market, the intent was to capture educational and cultural assets and to do so partly by causing immense disruption to education and to knowledge cultures and moving in to take advantage.

Many experts consider general models to be unreliable in principle (Luciano Floridi, John Lanier, Ragnar Fjelland, Iris van Rooij for example). Certainly they have proved unreliable in practice. But while the technical capabilities of these models are overblown, their capacity to concentrate data, computation, know-how and capital in the businesses that own them is unprecedented. The AI investment surge has further consolidated economic power in a handful of global corporations, power they have used to challenge international laws, labour laws and human rights frameworks (this last link to an Amnesty International report, 2025: pp.23-25).

Generative AI presents a number of immediate threats to learners’ rights, amplifying the known threats from so-called predictive AI (though the two are increasingly used together in educational systems and dealt with in educational policy as though they were one and the same). Channelling the knowledge available to young people through anglo-centric, proprietary and normative data platforms is a risk to their cultural and epistemic rights. Releasing AI-generated content into digital platforms and attacking public sites with AI crawlers, agents and deepfakes deprives young people of free access to digital information and culture, while diminishing their own opportunities for cultural production. These harms affect young people of minority languages and cultures to a greater degree. And there is increasing evidence (see below) that the use of chatbots in school work is harmful to young people’s intellectual and social development, and so undermines their right to the fullest development of their potential as laid out in the UN CRC.

Young people in education are therefore subject to three distinct threats to their educational rights:

1. So-called predictive AI is used to collect their data, to make unaccountable and discriminatory decisions about their futures, and to surveil and discipline their interactions within educational systems

2. So-called generative AI is used to capture and monopolise cultural and educational content, to advance hegemonic languages and perspectives, to degrade learning and to replace pedagogical relationships with automated agents

3. The narrative of ‘AI futures’ is used to refocus educational outcomes around the automation of intellectual work, stunting the development of young people’s capacities, and preparing them for work that will be precarious, exploitative, and algorithmically disciplined

Beyond risk: documented harms

While the risks have been widely speculated about, there are many actual and documented harms from generative AI to the rights of young people in education. These include:

• Children and young people being subjected to harmful biases, hate, fakery and misinformation through the mass circulation of synthetic memes, including via platforms used in educational contexts

• Evidence of generative AI use harming young people’s critical, intellectual and personal development, and amplifying biases and poor judgement

• Evidence that dependence on AI chatbots impacts on young people’s socialisation and emotional wellbeing, undermines relationships in the classroom, and weakens social cohesion beyond

• Monopolisation of educational publishing, testing and tutoring by AI corporations, producing new forms of educational, linguistic and cultural disadvantage and locking students into extractive relationships with AI

• Loss of credibility of assessment systems, leading students to question the value of qualifications and of education itself

• Direct attacks on higher education, teachers and ‘expertise’ by AI interests and industry representatives

There is also evidence of harm to the knowledge systems that education and educators rely on.

• Degradation of search, news publishing and political communication, not only by introducing biases, inaccuracies and fakery as content/data, but by disrupting and then drawing ito the AI business model every organisation that deals in content/data. Educational content and scholarly publishers are key targets, and there are threats to the maintenance of libraries, archives, heritage resources and public knowledge projects.

• Wealthy governments rolling back copyright laws in order to attract AI business; minority cultures and less wealthy governments have even less capacity to protect educational and cultural assets.

• Threats to creative and cultural work in majority cultures; minority cultures have even fewer opportunities to support their cultural industries, practices and people.

• Disruption of long-established epistemological norms such as the transparency and replicability of scientific discovery; open peer review; and the accountable authorship of published works.

• Challenges to scientific method (‘science can advance without coherent models, theory, or really any explanations at all’ Andersen 2008) and scientific freedoms.

Children and young people are unequally and inequitably harmed by:

• The build-out of hyperscale data centres, threatening power grids and water tables, piling toxic waste into vulnerable communities, and multiplying the carbon footprint of the computing industry

• Risks to the global economy through the hype, over-financialisation, under-regulation and state capture associated with the AI industry

• Failing democracies, partly from the impacts of disinformation and media capture, undermining the right of young people to determine their own futures; lack of accountability and learner voice in the use of data platforms in their education

• AI-powered warfare and surveillance: schools, teachers and students have been targeted for violence by AI systems; learners are widely subjected to surveillance through edtech platforms without consent.

Risk table

This table of potential risks AI presents to the right to education is a work in progress - you can read and comment here.

How might education leaders respond?

So-called ‘AI’ is antithetical to the UN goals of free, equitable access to learning and cultural opportunity. Education leaders should take a human-rights based approach to AI, not only as a class of technologies with known impacts on learners’ data rights, but as a crisis for education systems and their role in global peace and democracy. This crisis has been engineered by a small number of the world’s most powerful corporations in alliance with their state militaries.

The global response should be led by those most immediately affected: people of minority languages and cultures; people suffering from epistemic injustice (particularly at the hands of the digital and AI industries); teachers and education workers threatened with poorer conditions of work; and young people who aspire to the full development of their personalities and intellectual powers.

The AI crisis both amplifies and draws energy from other crises afflicting education and learners around the world, and should be tackled alongside them. Only with solidarity among students and educators, and across education sectors and systems, can the power of AI corporations be resisted.

The ten principles of UNESCO’s (2021) Recommendation on the Ethics of Artificial Intelligence are admirable but have not been adopted into law, let alone enforced as law, by any of its member states. The AI industry has attacked its own ethicists and bullied governments into abandoning reasonable attempts at regulation. Without enforceability, good principles only amplify the harms by appearing to have them under regulatory control.

Similarly, UNESCO’s AI Competency frameworks for Teachers and Students are worthy aspirations that can only be achieved with the use of safe, ethical, reliable, non-extractive, non-exploitative AI systems. Framing these aspirations as individual competences puts an unfair burden of responsibility onto educators and learners, while absolving from blame the powerful corporations that released rights-violating technologies into educational and cultural ecosystems.

Educational leaders should develop a shared action plan, drawing on a wide range of experts (independent of the AI industry) and oriented not on finding new use cases for AI in education (the AI industry will take care of those) but on prevention, mitigation and resilience to rights-based harms. Expertise will not be enough: only international solidarity can prevent governments and educational organisations being picked off one by one.

Responses might be grouped into three areas: ‘repurpose, rebuild and refuse’.

Repurpose, rebuild, refuse

Repurpose AI technologies, where possible, for projects of authentic learning and human flourishing e.g.:

• Develop small scale models using open data and APIs, publicly or community owned, that meet real local needs for learning and cultural expression;

• Design curricula that support inquiry into AI, its inequities and harms and wider contexts as well as its opportunities for use;

• Deliver learning activities with/without AI that reveal its fragilities, fabrications and breaking points:

• Teach and actively support counter-hegemonic data projects: feminist, decolonial, minority-cultural, public and sustainable projects, projects of data sovereignty and data justice;

• Undertake independent research into the social, cultural, developmental and longitudinal impacts of AI in education;

• Encourage slow scholarship and slow learning with AI, refusing speed and productivity to focus on quality and interpretability;

• In all of these activities, refuse anthropomorphism: describe precisely how models produce their outputs and how human agency is involved;

• Be alert to the constraints and contradictions that become visible in these repurposings: notice, teach, discuss and make known the mechanisms and meanings of AI

Rebuild cultural systems, practices and archives with resilience to synthetic media, e.g.

• Train educators, librarians, archivists and culture workers to keep content secure from AI bots and crawlers, and from AI-generated slop;

• Respect culturally-specific practices and make time for them in the curriculum;

• Maintain valuable cultural artefacts in non-networked digital repositories: use and renew them in the curriculum

• Create and protect spaces for learning that do not involve data systems, such as hands-on making and investigating, live writing (not necessarily assessed), active experimentation, sensory feedback, discussing and debating, live (unrecorded) performance;

• Secure education policy and the training of teachers from the influence of AI lobbyists and AI-funded researchers;

• Redesign assessments away from instrumental outcomes and standardised rubrics; negotiate assessment tasks with students;

• Build and sustain communities of open AI practice with shared norms, standards and values.

Refuse the impoverished, unjust and unsustainable educational future being offered by the AI industry, e.g.

• Refuse to use systems or architectures that may be harmful, biased, unsustainable or unjust;

• Refuse to provide or collect data that may be used to build harmful data architectures or to make harmful discriminations concerning learners and educators;

• Only collect learner data that can genuinely be used by learners and teachers to support their development; ensure it remains under the control of an accountable educational organisation.

• Refuse to adapt educational workflows and conditions of work to the use of AI, particularly where this will render education workers overworked, more precarious or less skilled;

• Refuse professional tasks that require the use of or contribution to a proprietary data system unless alternatives are available;

• Refuse personal and organisational partnerships with or investments in the AI industry

As every digital platform is ‘enhanced’ with AI co-pilots and agents, refusing and rebuilding are acts of ingenuity as well as resistance. Far from being signs of ignorance, maintaining spaces of non-mediated dialogue and cultural expression are rapidly becoming signs of technical and epistemic skill. Educators can foster these skills, confident that their learners will not be ‘missing out’ on AI because AI will always be making itself more useable: indeed, it is already compulsively useable, and it not use but non-use that needs to be actively developed. These approaches also call for solidarity among people, organisations and linguistic and cultural communities.

The outcomes of ‘embedding’ AI into education remain uncertain but they will not be produced by a merging of common interests. Rather they will be the outcomes of a struggle between incompatible goals: mass intellect versus mass thoughtlessness; diverse cultures and ways of knowing versus the normative power of data; participation in communities of learning versus the consumption of bite-sized answers from chatbot companions. In this struggle, UNESCO should weigh unequivocally on the side of children’s and young people’s right to the full development of their humanity.

With thanks to Bryan Alexander, Doug Belshaw, Laura Hilliger, Ian O’Byrne, and Karen Louise Smith for conversations that helped motivate and inform the writing of this piece.

Welcome to 2025, subscribers old and new. I’m happy to start the year as one of 100 women and non-binary writers on AI who are considered worth a follow on substack. I’m less thrilled to be writing this in a week when Meta has officially sacked the truth and the mainstream media of Europe, or what is left of it, has become an echo chamber for the politics of Elon Musk. But this is only what enlightenment institutions can expect when they cosy up to the counter-enlightenment projects of big tech, and hello higher education, what are your plans for 2025?

I’m sorry you have heard so little from me in the last few months. Blame the conjunction of a new full-time post and a descent into despAIr. MalAIse. MiserAI. At some point in 2024 it all became horribly clear. No amount of skepticism from the business sector is going to stem the supply of AI-ready graduates, whether they are wanted or not. No amount of research evidence or critical analysis of the claims about AI is going to be read in a university sector that is determined to turn research and critique over to privatised data architectures, and reading along with them.

No, it doesn’t matter how destructive generative AI turns out to be for the environment, how damaging to knowledge systems such as search, journalism, publishing, translation, scientific scholarship and information more generally. It doesn’t matter how exploitative AI may be of data workers, or how it may be taken up by other employers to deskill and precaritise their own staff. Despite AI’s known biases and colonial histories, its entirely predictable use to target women and minorities for violence, to erode democratic debate and degrade human rights; and despite the toxic politics of AI’s owners and CEOs, including outright attacks on higher education - still people will walk around the herd of elephants in the room to get to the bright box marked ‘AI’ in the corner. And when I say ‘people’ I mean, all too often, people with ‘AI in education’ in their LinkedIn profiles.

I believe I have been right about the elephants. But I’ve been wrong about the people. Our willingness to ignore harms when they are happening to other people is less surprising, given human history etc, than our capacity to put up with crap. It takes a lot of cope to deal with the banality that is AI in the real world. To find every tool you reach for has already started off on its own chattering journey like a wind-up toy. To feel every online interaction being stripped of tone and style and personal meaning and to tell yourself that’s fine, it’s good not to care too much, it’s probably ‘just an AI’ reading it the other end. To give up on finding reliable information or sharing family news because that environment is now a bin fire, but it’s OK because your grammar is never going to let you down. To forget what was promised last year, or last month, because ‘the AI’ does a new thing now that’s also ‘meh’ but it’s new and it promises to be amazing. To pay the subs, to not worry about the student assignments, to pay the higher subs, to do the academic integrity training, to paper over the cracks, to call papering over the cracks ‘AI literacy’, to realise your career and reputation and working future depend on not finding any of this problematic. All of that. It is exhausting. And human agency is exhaustible.

If I had to diagnose my own malAIse more precisely, I don’t lack energy for the big issues, I just can’t deal with the daily thoughtlessness, the anti-intellect of the AI narrative. And mounting evidence about the impact of using generative AI suggests that thoughtlessness is one thing that will definitely be scaled up.

Luckily, just as I was running out of juice, the wonderful Audrey Watters - original Cassandra of edtech - re-opened her blog for business, and AI has been relentlessly in her sights. You could do worse than start the year with this post from her about reading, give her a follow, and read on.

Meanwhile on imperfect offerings you can expect some short posts about issues that poke me too hard to ignore. I have some longer pieces in development on AI as interface, AI at war, and what universities might be doing better - when the day job gives me time. And starting this week, an imperfect podcast for your listening pleasure. This turns out to be a fabulous way of having interesting people say interesting things about AI and posting them to my own credit and fame. The podcast started out as a syndication of Generative Dialogues, my series with Mark Carrigan, which continues alive and kicking whenever we have something to kick off about.

So, precious readers, if you have managed to read this far without synthetic support, I bring you interviews with Dan MacQuillan, Eamon Costello, Catherine Cronin and Laura Czerniewicz, on topics from the fascist histories of AI to the capture of voice data, neo-Luddism and zines. And there are more in the pipeline. If you’d like to suggest someone I should talk to, please get in touch. Meanwhile as 2025 shambles into focus and the spaces of imperfection become a little less tenable, please do like, subscribe, share, comment, read and now listen to this one.

In this post I recap on what I have written in the past about generative AI and working futures. If you have been following for a while, I have new evidence and some new thoughts. If you’re new, I hope you find what you’re looking for. Follow the subheadings for the tl:dr version, and dive in for more detail, track-backs to previous pieces, and an assessment of how my earlier thoughts are playing out. I conclude that we need AI resilience and suggest what kinds of educational research might help.

In this piece:

1. Generative AI has few really valuable use cases

2. Employers are already falling out of love with generative AI

3. Generative AI might still restructure work - but not in a good way

4. Generative AI is not getting ‘better and better’

5. The carbon, power and water costs are frightening

6. Employers are also concerned about the legal and ethical issues

7. In the end, it’s all about student learning

8. Why AI resilience is so urgent

1. Generative AI has few really valuable use cases

Long ago, in ‘what is it good for?’ I defined generative AI as the automated production of digital content, optimised to a norm. Digital content production was to a large extent pre-automated before the arrival of generative capabilities, thanks to the ubiquitous use of certain productivity software (for example Adobe and MS suites) and standardised workflows around them. The content that was scraped to train large language models (for example from wikipedia, reddit and Common Crawl ) reflected the norms of the particular people and cultures that were over-represented in digitised content, and much of it was optimised for search algorithms, attention rents and click-throughs (that also have cultural biases).

The probabilistic methods that are used in training large language models, and that predict the most likely next element (word, phrase, cluster of pixels) during inference, weight its outputs even further towards cultural norms and biases embedded in the training data.

So pre-automation creates contexts in which large language and media models can seem to produce meaningful outputs. One obvious such use case is the production of search-optimised, clickable content. A second is short-cuts in writing code (highly standardised: already well documented via sharing sites such as HackerNews and StackOverflow). A third is providing natural language front-ends (chatbots) on standard interactions such as customer service. These uses have already been trialled extensively. None has so far produced a revolution in productivity, though staff have certainly been laid off in copy production (or have been transferred from direct production to improving the ‘AI’ output) and some boilerplate code is undoubtedly being outsourced to CoPilot, though with variable results.

In chatbots, the much-touted ‘success’ of OpenAI partner, Klarna, in laying off 700 staff turns out to be a little less than it seems. (Klarna outsourced customer support in 2023, leading to major problems with backlogs and customer satisfaction. The new ‘AI’ chatbot replaces an older, presumably phone-tree style system, but it is not clear whether the improvements claimed are in relation to the original service, or the degraded one, or whether they include the earlier chatbot system in the comparison.)

In none of these cases is the impact on work ‘transformational’ for the workers involved, except in a negative sense. The introduction of generative ‘AI’ is making work that was already repetitive and highly automated even more stressful and precarious. Its use has led to more quantity of copy and code, but as a result the overall value of search results and the quality of online content is degrading; same with the quality of code and its ease of maintenance (something we should all worry about in the immediate aftermath of the CrowdStrike outages).

Unfortunately for higher education there is another use case, and that is the production of student assignments. and similar kinds of writing to a rubric (cover letters, for example). This has many implications for the relationship students have to their learning, and to their teachers and universities and courses of study. It’s a far from simple problem that I and many other educators have commented on, and I will return to at the end of this post. But here, I want to make a distinction between the reasons students use generative AI and the reasons we are told that educators should actively promote its use. The rationale for the second is: ‘they will have to use AI in the workplace, so they (we) had better get used to it’.

This argument is flawed. Universities are not helpless bystanders to the economies of professional work but key stakeholders and policy makers. A university education is supposed to empower students to shape their futures, including how they relate to different techno-social configurations of work. And universities have responsibilities beyond employability - to justice, equity, the disinterested pursuit of knowledge, and having a planet to live on - that demand a more critical assessment of how these technologies might reshape work, and what harms they might inflict in the process.

But there are now more pragmatic objections to this argument. If by ‘AI’ you mean specialist applications of machine learning, these are being developed and adopted gradually, with expert input and as part of existing expert workflows. Students can get on with developing the expertise demanded by their field, confident that will then be able to participate in future ML developments (where they add value) as experts, rather than as data workers. If by ‘AI’ you mean the generic shitshow that is ChatGPT, there is no evidence that it is revolutionising work in general, or even making work significantly more productive. Here’s why…

2. Employers are already falling out of love with generative AI

Congratulations if you are so uninterested in the celebrity romance between big AI and big corporations that you haven’t noticed the cooling off that is in the air. So let me tell you that Goldman Sachs no longer sees AI as a game changer for business productivity. Cognizant, an IT consultancy that has just bet $1bn on generative AI infrastructure so really, really wants it to work, published a survey that found ‘up to 13% of businesses will have adopted the tech in the next three to four years’. Sorry, how many? The Harvard Business Review has also just dropped a study that shows using AI at work makes people unhappy (so less productive).

Companies are beginning to pull out of the enterprise versions of MS CoPilot because its outputs are ‘middle school’ level at best.

Other high profile AI projects that have been quietly dropped include Meta’s clutch of celebrity chat interfaces, an AI chatbot for Los Angeles school students, MacDonald’s AI interface for drive-thru customers, and Elon Musks’ Grok chatbot, shut down for spreading election misinformation. Actually it hasn’t been shut down, as the article makes clear, but it has spread misinformation. The Guardian recently listed other walk-backs, many from the creative industries.

Hedge fund managers are now telling investors privately that AI has been ‘overhyped’, and that its proposed use cases are ‘never going to to be cost efficient, are never going to actually work right, will take up too much energy, or will prove to be untrustworthy’. And the Guardian quotes a senior analyst at Forrester saying that:

a lack of economically beneficial uses for generative AI is hampering the investment case. There is still an issue of translating this technology into real, tangible economic benefit.

Tech stocks are falling, as analysts have realised there is a $600bn hole in revenue forecasts. Some of the biggest companies involved - Meta, Amazon, and Microsoft - now admit it will be ‘years’ before any of their AI products are showing a profit. Heck, Microsoft’s own Chief Financial Officer has just told investors that the company’s $56billion investment in AI data centres is not expected to make a profit from actual, sellable AI applications for more than 15 years.

The pillow promise in this love affair was that AI would make workers more productive, therefore capital more profitable. Back in March, Gary Marcus (one of the most prescient insiders) predicted that any productivity gains would be modest:

The same pre-automation of content production that makes generative AI quite useful, if you don’t care too much about quality, means that the contexts in which AI is quite useful were already quite efficient. Generic generative AI may provide shortcuts to simple tasks (think of MS Copilot as a Clippy update with some enterprise apps) but it makes some complex tasks harder. No doubt the potential will be talked up for a while longer, but no-one serious is talking about a productivity revolution any more.

3. Generative AI might still restructure work - but not in a good way

We should not see this moderation of the hype as a sign that AI will be less bad for workers in creative and ‘knowledge’ sectors. At every point in the hype curve it is possible to worsen working conditions and cheapen the price of labour. Brian Merchant puts it like this:

The design and build of large synthetic media models embodies a particularly toxic approach. As I wrote in ‘Labour in the middle layer’, models are built on the unacknowledged, unpaid work of content creators in the past (training data). But they also depend on the hidden labour of thousands of data workers in the present. First to clean, curate and prepare the training data; then to evaluate, annotate and refine model outputs to produce the desired norms. This process took eight months in the case of GPT4, and although it is a closely guarded secret how many human hours this actually represented, eighty percent of the paid hours on any AI project are estimated to be on data work.

In ‘Luckily we love tedious work’, I argued that graduates won’t necessarily find themselves occupying one of the two ‘expert’ groups in this labour sandwich - producing the original, valued content for models to extract, or using models to support them in their expert roles. In the sandwich economy, productivity and profit depend on paying as few experts as possible, and maxing out their value in the middle layer: this is what produces the model itself as a source of value for its proprietors. Growing at an estimated rate of nearly 30% a year, this work is typically outsourced, precarious and badly paid, but it is far from unskilled. Data outsourcing companies now specialise in sectors or industries; even large general crowdsourcing platforms prefer workers with specific expertise (observations from Muldoon et al’s 2024 Typology of AI Data Work).

Tasks such as these from ScaleAI may relate to generic data, such as images of public streets, or highly specialised data, such as artisitic works, medical or military images, engineering drawings or business spreadsheets. As this layer of labour becomes more specialised and segmented, it has even more use for graduates.

In researching the links between ‘AI’ and the military for another post (coming soon), I discovered that US Project Maven uses Community College students to annotate satellite images for potential military objects ‘for $15 an hour, with course credits thrown in’. These students are definitely ‘AI-ready’, and can move seamlessly from their college degree to the kind of work available through platforms such as ScaleAI. By the way, the UK Government recently celebrated ScaleAI choosing the UK as its centre of European operations, as though it were the next DeepMind instead of a gigwork company highlighted by the Oxford Internet Institute for failing to meet basic standards of fair work.

One way that AI companies are trying to make up for the hallucinations, biases, privacy risks and running costs of their product is by telling businesses to plug in their own data infrastructures. Companies are running MS Semantic Index on the heaps of documents they have accumulated in MS 365, or annotating valuable company IP and documentation to make it ‘AI-ready’. Generic AI models can then be ‘plugged into’ these Retrieval Augmentation Generation (RAG) sources to produce business-specific insights with greater accuracy and fewer privacy risks. It’s a neat way of getting businesses to do the data work that produces most of the actual value, while still extracting rent for the generative front end. But it also means that work is being restructured towards data accumulation and management inside the businesses that are buying the generative solutions.

RAG is basically Knowledge Management for the 2020s. In case you were not there at the time, KM was a business improvement process that was very profitably touted around by management consultants in the 1990s and early 2000s. It urged businesses to capture every thought and gesture of their staff in the form of digital records, supposedly allowing the business to leverage all that knowledge without the expense of employing the knowledgeable staff who produced it (sound familiar?). Mostly it failed, though arguably today’s integrated MS systems are a hangover from the fantasy of total knowledge capture leading to vast productivity gains. Only you can judge how happy, creative and fulfilled the MS panopticon makes you in your job.

In fact, many of the conditions that led to the failure of KM also pertain to Generative AI. A recent study by Upwork (as summarised by Cory Doctorow) found that:

96% of bosses expect that AI will make their workers more productive;

85% of companies are either requiring or strongly encouraging workers to use AI;

49% of workers have no idea how AI is supposed to increase their productivity;

77% of workers say using AI decreases their productivity.

Employers who want to make their workers more productive are almost comically susceptible to hype of this kind. Once they are bought in, the onus is on their staff to make the miracle happen, even though the idea of documenting everything you do and then being expected to do more of it in less time (or gracefully resign) is not an attractive one. But like any ponzi scheme, there is no benefit in telling the person above you that the promise they have just been sold is an empty one. The only option is to keep pushing the promise downhill, until it arrives with those least able to push back.

The irony that attempts to automate can actually hamper productivity is not a new one. It has been researched for many years in diverse industries. In ‘Ironies of Generative AI’, Simkute et al. revisit earlier findings concerning:

a shift in users' roles from production to evaluation, unhelpful restructuring of workflows, interruptions, and a tendency for automation to make easy tasks easier and hard tasks harder…

and find them playing out in full force in the GenAI adoption crisis. But the belief that they can be ‘replaced by AI’ is profoundly depressing and potentially undermining for graduates and professional workers, and only makes it more likely they will accept working conditions that are more precarious, less fulfilling, more isolated, less remunerated.

Educators should already have been asking: ‘is it really our purpose to prepare graduates to be the most productive humans-in-the-loop they can be?’ and even ‘whose productivity, for what greater human good?’ We can now add to that the question: ‘what productivity are you actually talking about?’.

4. Generative AI is not getting ‘better and better’

AI models carry on not getting exponentially better - not even very much better - exactly as I predicted here. There is no fix for hallucinations, as the CEOs of bigAI admit here, here and here. The sheer size and inscrutability of the trained data structure makes it impossible to fix all the connections that might spit out the ‘wrong’ answer, though this does not stop big AI spending $millions on data workers to keep patching up the worst examples. OpenAI has still not delivered GPT5, despite many promises and trailers. The people who are saying that we should be grateful for each iterative improvement are the same people who were saying a year ago that we were ‘only at the start’ of AI’s incredible capabilities, and that they would have improved beyond recognition in a year’s time.

Many cognitive scientists have argued from the beginning that generative AI has computational and theoretical limitations: read, for example: The Cognitive strengths and weaknesses of modern LLMs, Reclaiming AI as a theoretical tool for cognitive science, or Intelligence without reasoning. All of them find it intrinsically unlikely that there will be a major breakthrough in performance. There may of course be iterative improvements due to further scaling up of parameters, or improvements in training and post-training human reinforcement learning. But these are not only costly (and investors are losing faith): they are also pushing against other real limits.

For example, AI is fast running out of quality human-generated data. These projections do not include the impact of synthetic text flooding public sources of information, poisoning the culture of text production and diluting the quality of any human text that still gets out there. (A timely article in Nature also finds that training AI on AI generated text leads straight to model collapse - a shame because that is precisely the solution to a lack of data being touted by people who never thought that human content was up to much anyway).

AI is also running out of compute. There is a world-wide shortage of GPUs (AI chips) and Nvidia has delayed the launch of its next generation Blackwell chip due to production flaws. But it is probably a good thing there are not enough chips to power up the ambitions of big AI, because the world just can’t afford to power them. What Sam Altman calls an ‘energy breakthrough’, required to produce the ‘next generation’ of AI, most analysts are calling an ‘energy crisis’. Because…

5. The power, carbon and water costs are frightening

The full cycle carbon costs of generative AI models are only just starting to be seriously researched. But all the early indications are that they are far, far higher than anyone thought when I first wrote about this issue. There are the enviromental costs of chip fabrication, including the mining of rare metals and the use of largely coal-fired power in Taiwan. These costs are accelerating, as one of Nvidia’s strategies for staying on top of the market is to rapidly obsolesce and replace their highest end GPUs. There is the power required to run the data centres that the models are trained and housed on, recently predicted to rise by 160% before the end of the decade, and there is the fresh water required in all these processes.

Above all there is all the additional compute required by the whole connected world now that power-hungry inferential processes have been embedded into so many basic operations (Luccioni et al explain here why inference is so disastrously expensive). Emissions have soared at Microsoft and at Google, threatening any gains they have made in carbon reduction. Google alone is pouring $billions into new data centres. But most of the additional compute, and therefore carbon costs, is being fired up not by the big AI companies themselves but by the companies buying into their AI solutions:

Since the same companies that sell ‘AI’ also sell IT infrastructure, aka cloud computing and cloud services, and since they have an interest in hiding ‘distributing’ all that additional carbon cost away from their own balance sheet, we will almost certainly never know the true impact. But what is really sick about all this is that finance is now piling into power and utilities stocks. The AI boom they created by piling into AI stocks may not last, but the people convinced by it are still buying chips and cloud credits. So its win/win for finance,so long as the little people just keep believing, and so long as no-one needs a planet to live and breathe on.

I wrote that the greatest climate threat from ‘AI’ was its ability to divert attention, financial power and political will away from from the economic transition we urgently need, and into tech non-solutions like nuclear fusion or some climate engineering miracle designed by a future AI brain. But the capacity of ‘AI’ to delay the real solutions to climate crisis is even more dangerous if in the intervening years - and there’s a phrase to make any climate scientist despair - ‘AI’ is ramping up the carbon output to the max.

6. Employers are also worried about the ethics and legality of generative AI

Employers are concerned about the environmental costs, and also about the toxic biases of generative AI and the negative implications for their DEI agenda. There is plenty of evidence that the use of AI leads to bias in recruitment, for example, and companies are worried about the legal implications. The ‘big four’ accounting firms - that offer some of the top graduate opportunities in the UK - have ‘prohibited’ the use of AI in job applications on equity grounds, and on the grounds that it might select candidates who are disposed to ‘cheating’ - not the best start in an accounting career. Of course ‘humans are fallible’ in recruitment, but few human HR managers have the reach of an AI system, and they can always be held to account. When it comes to gender and race discrimination caused by AI, there is currently no clear legal framework, something that puts people at risk immediately, but also puts companies at risk of future claims.

The issue of potential copyright infringement in the training of generative AI models is still not resolved. Major content owners are rapidly being bought out by AI companies, and copyright law itself is being undermined by campaigns like this one from OpenAI, to the effect that if a government insists on upholding laws that are inconvenient to AI, they won’t get access to the sweetie jar. Still, all this is another source of uncertainty (and therefore legal expense) to any company contemplating whether and how to adopt.

And finally, never mind the factual errors and productivity fails, generative AI is becoming a toxic product. From deepfake image-based abuse to undermining democracy and manipulating elections, from polluting science to ‘cancelling emotions’ (especially useful in those new AI-supported call centre roles), from racially biased image generation to new forms of online addiction, companies are having a hard time persuading their customers that the ‘AI’ in their own products is squeaky clean. It doesn’t help that the biggest players are often the biggest offenders. Nvidia, for example, has just been caught scraping youtube videos for facial data and Meta garnering clicks from AI images of extreme human suffering. But, hey, celebrity chatbots everyone!

7. In the end it’s all about student learning

It is far from clear that employers want or need (generative) AI-ready graduates. All the things that corporations are concerned about right now are things that critics - many based in universities - have been saying from the start. So where are universities in this debate? At least they could be engaging with it as a contested zone that they might still influence for the better, rather than as a predetermined ‘AI future’ they most provide for.

Existing uses of machine learning and generative AI at work show that they can automate the routine parts of tasks. But you can only know how this automation will be useful if you are already an expert in that task. You can only initiate and guide the generative component if you are already an expert in that task. You can only correct for errors and refine the outcomes if you are already an expert in that task. You can only participate in the design and development of new workflows if you are already an expert in that task. This is true whether the task is writing prose or diagnosing cancer. So universities should continue to produce graduates with expertise, confident that they will be able to accommodate any efficiencies that computation may offer down the line. Technologies are designed for ease of use: expertise is hard to acquire.

As I said at the start of all this, the real issue is not what students might be doing with generative AI in some possible future, but what they are doing with generative AI now and how that shapes their individual learning and development. Also what they aren’t doing when they are using generative AI that might be more valuable to their learning (opportunity costs). And finally, what collective cultures and practices are being shaped by their use/non-use of generative AI that students will take forward into work and life.

I am confident there are uses of generative AI by teachers that are helping students to learn. The question is whether they need to be learning in this way. Are they learning in ways that our theories of learning tell us will be beneficial to them in the long term, as workers and as people with an intellectual life and culture beyond work? Are they learning in ways that accord with our values for university learning as a set of cultural practices? I have not seen evidence about this, but then I don’t think many people are asking these questions. What I have seen is a vast amount of work being done by teachers to devise tasks with generative AI, often ingeniously, and always with commitment to student learning. I believe that many of these tasks could be at least equally beneficial to student learning if they did not pass at any point through a synthesis engine.

To give some examples.

1. Prompt engineering, a skill that is already obsolescent. Generative AI is basically evolving into an interface, and interfaces need to be frictionless, so most of the ‘work’ involved in crafting prompts is already being absorbed in further layers of automation (think CoPilot suggestions, and any ‘AI-based’ app that is essentially poking prompts into a foundation model and serving you the results). What will never be obsolete is the skill of asking the right question, or engaging in a dialogue with genuine curiosity. So go ahead, have students devise questions (call them prompts if you like) to clarify their thinking. These can be used in discussion, or to carry out a literature search or some other iterative data query, or to frame the reading of a paper or the watching a video. Want to stage interesting debates and scenarios? Let students devise and enact them. Or have students write their own test and revision materials. Let them write detailed prompts to an imagined language model to ‘test me on this topic’, but then complete the instructions themselves. Because they know what it is like to be a student of this topic (and they have theory of mind), students will do this far better for each other than a chatbot can, but more importantly they will learn from every part of the process.

2. Breaking down writing tasks into component parts. This is always helpful and UK students do not get enough support with it IMO. But the components do not need to be practiced with the aid of an auto-complete word-generator. Ingenious prompts can be prompts for actual writing, not for synthetic production of text. Also, the components still have to be put back together again. At that point students realise that ideation is not separate from gathering evidence, and gathering evidence is not separate from summarising/annotating it, and notes are not separate from ideas and opinions, and ideas and opinions are not separate from the words and images they are expressed with. Writing involves iterating among all of these, and testing the results against facts in the world and with other thinking people. So to invite students to outsource all the parts while they take responsibility for ‘bringing it together’ is to misunderstand as well as to confuse students about the nature of writing. And the nature of thinking. Also, I notice that some guides tell students to outsource the planning and overview but be sure to write in the detail themselves, while others tell students they must own the planning but can use generative AI to help with the detail. Just who is most confused about writing here?

3. ‘Doing research’. As everyone knows, generativeAI is prone to mistakes, hallucinations, non-referencing and false referencing, and bias to the norm. Many students now use ChatGPT/GLM etc for basic research, that is ‘finding stuff out’, and happily most research-based tasks designed by educators are meant to illustrate the problems with doing this. But we are relying too much on students already thinking like experts in their field? If students can identify a problem, what resources are they using to do this, and how are they judging the reliability of those sources? If students can provide a better answer, what knowledge and expertise are they drawing on? And if what they really need is to develop those alternative resources of knowledge and expertise, any search or ‘research’ they do with ChatGPT must have an opportunity cost. So ‘spot the AI mistakes’ might be a cautionary exercise, or a one-off revision test, but I don’t think it provides the motivation or structure students need to develop information skills or disciplinary practices or foundational concepts or epistemic judgement for themselves. The experience of many students in this situation, that the AI is ‘often right but sometimes wrong but you can’t tell when’, may lead to the worst possible combination of dependency and anxiety. Instead of focusing on specific errors, such exercises are surely better directed to revealing the generic mistake that is relying on an autocomplete engine at all, and explaining the biases, obscurities and injustices that make them so unreliable. And that leads naturally into a discussion of research strategies and methods that are trusted in the subject discipline, and why they are trusted (and so into epistemology, even if that word is never used).

If universities believe in the value of learning at university, they will offer spaces for collaboration and knowledge-building. They will provide students with models of self-development, and a culture of respect for the finite planet, and the different epistemic cultures and traditions that enrich it. They will be places where public knowledge and specialist expertise are actively being produced, not passively accumulated. Then they will develop not only ‘good workers’ but people who can build good workplaces.

8. Why AI resilience is so urgent

As I argued in ‘writing as passing’, student work is almost by definition the production of content, optimised to a norm. This makes the effects of generative AI on learning and assessment profoundly disruptive. Assessment is a performance, but we make a contract with students that by jumping through the hoops (usually content production of some kind) they will in the process be developing some practice or understanding useful to them beyond the scope of that performance. In other words, they will be learning something of value.

If a chatbot can jump through the same hoops, to something like the same normative standards, and if we can’t reliably detect the difference, every part of that contract breaks down. Students don’t have to go through the process we designed for them. Worse, we can’t follow what their process is. The models are black boxes that refuse to give up their secrets. How (for example) do the patterns they encode correspond (or not) to the schemas that experts use to organise their thinking? (Remember constructivism? This first concept of teaching 101 is that learning involves actively building conceptual schema, and patterns of meaningful activity. Learners have to do this for themselves, on their own terms, based on their own prior experiences. The schema and practices of experts can be valuable as models. The parametric structures and hyperparametric weights of a transformer model? Not so much.)

Because of the secrecy engendered by ‘academic integrity’ scares, students’ process with these models is a second black box on top of the first. We see what comes out, but we have very little idea (unless I am missing some new, more subtle research) what students are putting in. Worse still, students can no longer be sure that whatever process they are learning has value, if a computational process can (apparently) do the same. If they have followed guidance and only used generative AI for some parts of the process, what grade might they have got if they had used it for the whole process? What does that mean for the value of the part they did for themselves? What does it mean for the value of the part they didn’t do for themselves?

In February, a survey of nearly 500 students found:

preliminary evidence that extensive use of ChatGPT has a negative effect on a students’ academic performance and memory; [therefore] educators should encourage students to actively engage in critical thinking and problem-solving by assigning activities, assignments, or projects that cannot be completed by ChatGPT,

Tempting though it is to wave at these results and retire, I am not convinced they show ChatGPT is a cause of poor performance. But they do identify - at least for this sample - the kind of students, and the kind of learning situation, for which the use of ChatGPT becomes compelling, and they show that for these students the academic outcomes are poor. Another recent study on student perspectives found that the more confident students were in their own writing, the less they approved of the use of ChatGPT for writing. These studies are useful, but they raise more questions than they answer. We need more studies of this and more qualitative kinds; studies of different uses, different students, and different settings; we need far more longitudinal studies; we need more critical studies that look beyond immediate contexts of student use. But mostly we need collaborative work with students that go beyond surveys, inviting them to explore openly their processes of production, and their experiences, both personal and socio-cultural. (What are the pressures on students to which ChatGPT/ChatGLM (etc) is the solution? How do they value their own processes of reading, writing, note-making and knowledge construction? How does the belief that ‘everyone is using it’ change their perceptions of academic work? Such experiences will be complex and differentiated.)

Instead of the AI-ready graduate, this might give us a sense of how the ‘AI resilient’ graduate comes into being. How do they develop the expertise to critique and mediate the generative output, not just in class-based activities but in their own study and intellectual life? What skills and practices of production might help them to withstand future cycles of planned deskilling and automation? How can the promise of instant productivity and performance be mediated by other values and hopes, for their learning and for their working futures?

Last month I gave a seminar as part of the IoE Academic Writing series called ‘Student writing as “passing” and the role of generative AI’. You can find the recording here if you prefer to read with your ears. Thanks to the wonderful Ayanna for organising this event and for her ongoing investigations into student writing alongside synthetic text. Thanks also to the participants whose insightful questions and contributions helped to shape this post, and to Ayanna and John Hisldon for generous comments on it.

In this post I cover:

• The Turing Test, and what it has in common with student writing, at least as writing is often experienced by students

• Student writing as ‘passing’ in a world of deepfakes and generative possibilities

• Beyond passing 1: student writing as identity work

• Beyond passing 2: student writing as social activity

• Beyond passing 3: student writing as expression and dialogue

• Some practical ideas for writing beyond passing, many drawn from the seminar participants

In preparation for the webinar, I also wrote a post all about the Turing test, its implications for contemporary ‘AI’, and how it relates to theories of learning. That might be a good place to start if you want a deeper dive into those issues. Otherwise, here’s a brief summary on the Turing test before getting down to thoughts about student writing.

The Turing test

The Turing test is a thought experiment that remains at the heart of the ‘artificial intelligence’ project.

The classic version can be represented as a system of three players. The role of Player C – the ‘judge’ or ‘interrogator’ - is to question the other two players, who are a computer (Player A) and a human responder (Player B), and to decide which is which. Turing’s prediction was that by the end of the twentieth century:

‘an average interrogator will not have more than 70% chance of making the right identification after five minutes of questioning’

It is this capacity to fool people, or at least some of the people, some of the time, that is the goal and standard for artificial intelligence. It is a standard that is regularly claimed to have been ‘passed’ or ‘broken’, and since the first public generative transformer models were unveiled in 2022, those claims have come and gone as regularly as OpenAI has changed its ethics team.

The equivalent positions of human and computer, the insistence on disembodied ‘outputs’ and the comparative nature of the judgement are all used as signs of the test’s objectivity. The players are behind some kind of screen. Their bodies and voices are hidden, their responses limited to the passing of text messages. The human beings, then, are made to seem as much like automata as possible. Yet, as I show in my parallel post, the test is a maelstrom of misdirection, identity politics and hidden desires.

Student writing as ‘passing’

In my talk I outlined some similarities between the Turing test and student writing for assessment, at least as it is typically understood by students themselves. For example, the dynamics of the situation place students on the other ‘side’ to their academic assessor, visible only through a screen. Text is the preferred medium, reducing opportunities for interactions of care and concern that may previously have existed among the players. The judge can ask anything they like, and their decision is highly consequential. Ultimately, ‘passing’ as a graduate really does confer a new identity. But meanwhile, every time a student offers up work for judgement, they are being compared with other students who may appear to be more credible, more ‘passable’, less of a fake.

There are many reasons why students may feel that they are not credible as ‘real’ students, or ‘real’ producers of text. Synthetic text promises, on the face of it, to allay these fears.

Yet while it offers to help students ‘pass’ each assignment, overall synthetic text only adds to the anxiety about passing. The norm for comparison – in students’ minds at least - is now an AI-augmented student, perhaps one with access to better models or AI writing services, or with more confidence to use AI and not get caught. Students may feel they are contending directly with text automata, their faultless grammar and authoritative tone becoming the new norm against which ‘real’ student work can only look inadequate.

And students may justifiably feel concern about failing the Turing-equivalent test of ‘AI detection’. Having your text identified, rightly or wrongly, as emanating from a bot may mean becoming a non-student, or at least being subjected to investigations that further undermine your sense of validity. In the absence of any certainty, on either side, about how ‘AI text’ can be detected, the best strategy for passing may not be the straightforward one – to write unmediated – but rather to second-guess what ‘unmediated’ looks like from the other side of the screen.

The dilemma of writing authentically in a world of synthetic text is the same dilemma facing every content producer in a world of deep fakes. Here’s Rob Horning in a brilliant recent substack post on the tension between realism and ‘the real’:

Horning goes on to define the ‘deep real’ as a representation that aims ‘to subordinate events to their potential status as content’. The real (event) gains currency only when it enters circulation, and it is the content economy that defines its realism. If we think about student writing through this frame, all writing as process (event) is sooner or later subordinated to its status as an assignment (content), offered into the system for submission, grading, and validation. The status of writing as ‘authentic’ is not given until it has ‘passed’ these hurdles, which like realism also have their modes and fashions.

None of this invites deep attention to the process or event of writing. Indeed, unlike in the US, UK students receive little or no attention to writing as a process. They may well have come through a school system that promotes the most instrumental kinds of text production, expecting a mark for every correct fact, and throwing in as many key words as possible. School students in the UK are trained to write for a highly automated marking system, even if teacher labour has not been fully automated. And this is true of education systems around the English speaking world. As a result, large language models are trained disproportionately on text that has already been optimised for such systems.

For students, university writing can seem an unassailable mystery even before the mysteries of assessment. It is a black box all of its own. You learn stuff. You write stuff. Who knows what happens in between? It must not be cut and paste. It should not be auto-summarise, or auto-generate. And yet these processes work, in a way that is not at all guaranteed by the recommended methods of note-making, observation, reflection, discussion, planning, analysis, ideation, argumentation, position-taking, translation, disputation, editing and re-editing, and all the iterative adjustments between the whole and the parts that constitute thinking in text. However, we can try to open up the black box of writing, and later sections explore how some teachers of writing do this.

Unlike Turing, academics also try to open up the black box of judgement and explain to students the criteria by which their writing will be assessed. But in being made explicit, qualities of writing can be further instrumentalised, and can even become data models of the kind used in automatic essay scoring. (It seems worth noting here that, after decades of investment, AES systems remain barely useable, unless their purpose is to skew student writing permanently towards those surface features that automation can measure, and away from all other considerations of value. One good thing to be said for AES, however, is that ChatGPT is worse.)

Today’s large language models are evaluated against qualities that sound a lot like marking rubrics. ‘Readability’, ‘insightfulness’, ‘completeness’, even ‘metacognition’. Every measure in the FLASK evaluation set (shown above) is a statistical model of a quality that was originally judged by human assessors. As you can imagine, among model developers there is a strong incentive to integrate these benchmarks into the model training process, ensuring that outputs are already aligned and the model is likely to perform well on benchmark tests. Just like marking rubrics, benchmarks skew performance towards values that can be measured, and towards norms that have already been standardised.

Retraining is a long and expensive process. As a short cut, model engineers also use system prompts to adjust performance. System prompts are natural language instructions that are typically added to the user prompt to direct outputs. Versions of ChatGPT’s system prompt have been widely circulated, and they read in parts like the kind of general guidance that might be given to a student writer, perhaps in a slightly more instrumental tone:

• Organise responses to flow well, not by source or by citation

• Do not regurgitate content.

• Make choices that may be insightful or unique sometimes.

• Never write a summary with more than 80 words.

• Integrate user suggestions.

So synthetic text arrives into a regime of writing and assessment that is already instrumental and transactional, tuned to measuring what can be measured rather than supporting personal development or diverse voices. Without being fully synthetic, student writing has already been pre-automated in order to pass through university platforms: submission systems, plagiarism detection, automated marking and feedback, progress/grade dashboards. Student writing might already feel a lot like passing textual tokens through a screen. Particularly as rising teacher/student ratios and reduced teaching time mean that the different players, just like Turing’s, may hardly know each other at all.

Student writing as identity work

How can we enable students to experience writing as more than transactional?

A text that influenced my thinking on this subject was Mary Lea and Brian Street’s (1998) Student writing in higher education: an academic literacies approach. (The link is to a great retrospective article on the book and its impact.) Lea and Street see ‘disciplinary practices of reading and writing’ as constantly evolving and being contested over, as well as becoming authorised and passed on. The academic literacies that students develop are not just skills with text but are ways of relating to their subject of study, and negotiating a place in that discourse community. Literacies are ‘concerned with ...identity, power and authority’.

In Writing as Identity (also 1998), Ros Ivanic considered how the process of writing asks us to manage several different versions of ourselves. There is the ‘autobiographical self’ through which we tell our ongoing life story, and which has brought us to the point of writing. There are the various sociocultural roles we are offered to adopt as writers, for example in relation to the university and its arcane processes for authorising texts and people. In writing, Ivanic distinguishes also the ‘discoursal self’, which we might think of as writing style or voice, from the ‘authorial’ self, which we might think of as position or stance – how we construct the ‘I’ of the text, and take responsibility for its arguments, observations and positions.

I’m not convinced Ivanic’s different selves can easily be untangled, but what her work does help me to think about is how identity is at stake in writing. How there is never only one identity, or only one outcome of the reflective dialogue between them: ‘In any institutional context there will be several possibilities for selfhood… [but some] of these will be privileged in the sense that the institution accords them more status’ (Ivanic 1998: 27).

How and what we write confers power and privilege as well as identity. Like spoken accent, writing acts as a sign of social capital, and a key to accessing further opportunities.

The magic of writing at university is that it really can produce a new identity. Graduation literally confers this. The hope is that by learning to account for themselves in writing, students produce new resources for autobiography as well as for accreditation. But writing is exposing. It is a bid for credibility that is public, contestable, ‘out there’, and may fail. Although universities have changed in many ways since Ivanic was writing, and are on the whole more diverse and more welcoming of diverse forms of expression, still their role remains to uphold the status of certain kinds of writing. And today’s students, used to the intense scrutiny and self-scrutiny of a life online, are more anxious than ever to maintain credibility when they express themselves, and more anxious than ever about comparisons with a norm.

I think most students experience academic English as a profoundly ‘other’ discourse. If they don’t feel completely excluded from it, they may feel they are writing in a voice that is not their own. All the more so if (some variant of) English isn’t their first language. One way of gaining a sense of ownership is to bring more personal or personally significant material into student writing, so that they have something to say (so that they feel like ‘authors’ of their own material) before they start to write. Hannah Ashley and Katy Lynn, in a lovely essay ‘Ventroloquism 001: How to throw your voice in the academy’, report that this approach gives students greater confidence for writing.

Some students will feel more comfortable than others working with personal material. And there is always a source of tension between the personally available and the authorised. But Ashley and Lynn see this, too, as a source of interest and identity work:

Practices that bring in the "I," like memoir and service [community-based] learning, provide students with an opportunity to see how their own private/community discourses are part of a particular set of popular/commonsense notions which get called into question when they butt up against a different community discourse. Ashley and Lynn 2003

What I like about their work is that they avoid calling student’s private/ community discourse ‘authentic’. They talk about students learning to recognise themselves as writers in dialogue with a host of speakers, writers and discourse communities, some of them authorised by the academy, some personally available, but all of them contestable. In this teaching practice, there is no ‘correct’ way of writing – not even one way of ‘writing as oneself’ – but rather a range of possibilities that students can explore.

Bringing the ‘I’ to writing does require students to take up a stance, even if it is a provisional and partial one that can be revised. In the context of a reflective classroom, ventriloquy or passing as a writer of one kind or another can be made more playful, but it can’t be made completely safe.

Today, as well as the spaces of higher education, students are negotiating their identities in digital networks and communities. So it does seem to me important that research into students’ use of generative AI also looks at how students are being constructed as writers by the AI industry and how they are also managing these constructed selves when they come to write. AI tools and services do not typically address students as producers of knowledge, or as negotiators of different perspectives, however we may encourage these approaches. They do not suggest identity as work, even of a playful kind. What AI offers is a magic cloak, a trick, a fully achieved writing identity. And this in a setting where other ideal selves are also magically available.

Student writing as social activity

I don’t think we can counter the allure of the magic cloak without looking a bit deeper into the theories of language that stand behind the AI industry – and they are different to the ones that have mainly informed practices of academic writing.

Computational linguists have often tangled with the linguistic ideas of Ludwig Wittgenstein, at least as they were expressed in his early Philosophical Investigations. In 1939 there was even an argumentative encounter between Wittgenstein and Alan Turing: his subsequent development of the Turing Test as a kind of language game is rumoured to have been a kind of riposte from Turing to Wittgenstiein (a reminder that I have written more about the Turing Test here.)

It was a student of Wittgenstein’s, Margaret Masterman, who first proposed that words could be represented as (mathematical) vectors describing their relationships with other words in a corpus of text. Her work made language computable without having to start from general rules of grammar or syntax. Each word can be expressed in terms of its relationships to other words. Unlike in formal grammars, new relationships can always be added on, making these models flexible, extensible, and particularly suited to data at scale. Vectors formed the basis of the first working search engines and information retrieval models. Today’s large language models are essentially massive vector databases.

Masterman’s approach can be traced back to Wittgenstein’s insight that definitions can be ‘by example’ or by family resemblance, rather than by logical equivalence. So Wittgenstein is held up (correctly, in my view) as supporting the conceptual shift within AI, away from propositional rules and towards probabilistic modelling. A useful recent article about this shift is called From rules to examples: machine learning’s type of authority. Another very accessible piece is Remembering Ludwig Wittgenstein in an age of AI.

However, neither of these essays, nor Masterman’s original work, take account of another insight of Wittgenstein’s, against which his idea of exemplary language needs to be set. ‘The speaking of language is part of an activity, or a form of life’, he wrote. It is the form of life, the ordinary activities or games people enter into with language, that give language its meaning. GPT4.o, for example, can respond to input text in a ‘context window’ as long as 128k elements. But however massive and inter-related the corpus may be, however impressive the ‘context window’, they are still just text. And Wittgenstein clearly intended by ‘use’ something more than ‘place in surrounding text’ or even ‘web of textual relations’. Wittgenstein meant the social, cultural and practical situations in which people make language work for them: ‘to imagine a language is to imagine a form of life’.

(Arguably, the to-and-fro dialogue between a user and a large language model constitutes a new kind of language game. I find this an interesting idea: but it is still one that would need to be investigated in terms of users’ emergent understanding and meaning-making, and not – in my view – as if the language model itself is participating in a form of life, in Wittgenstein’s sense.)

Margaret Masterman realised Wittgenstein’s first insight through the development of neural nets and machine learning. The Wittgenstein of ‘language games’ is found in other lines of linguistic thought, lines that throw into question whether language models can be said to ‘mean’ at all.

Critic Toril Moi sees Wittgenstein as a philosopher of ‘the exemplary, the public and the shared’. The exemplary in this sense is not the same as the ‘normal’, statistically calculated, but almost the opposite: a space for coming to terms with differences, for encountering the otherness of other people and their meanings. While statistical AI no longer tries to ‘fix’ language in grammatical and syntactic rules, it still insists on a fixed structure. A structure that expresses probabilistic rather than deterministic relationships. A structure that emerges from reams of public data rather than being constructed by self-appointed experts. But still, it is a structure that fixes text in the past, that is supposed to generate meaning while completely resisting any challenge or change.

And it is not the quantifiable differences among words that constitute, for Wittgenstein, a language game, but rather the qualitative differences between people who share a linguistic space, who try to make sense of and with each other. Language is social activity. Meaning is not derived from the relationships among words in the texts of the past, but by exchanging words in the present, in order to make something (socially) happen. In a recent interview Moi summarised her insights like this:

[writing is] action and expression. [We] need to think about what we stake ourselves… this will involve trusting our own experience, although our experience might at the same time need educating.

Student writing as expression and dialogue

According to Moi, then, experience is what education offers. New experiences, new ways of experiencing, and new opportunities to express and share experiences with others. So key elements of student writing are activity - doing things with words – and dialogue – engaging with others, using words.

My friend John Hilsdon is an academic writing specialist who works with students on the basis that academic writing is always doing something. His ‘functions of writing’ framework draws on theories of functional linguistics (that also owe a debt to Wittgenstein). But what matters to students is that by asking simple questions about the role of each piece of text, they start to worry less about word choice and ‘academic style’ and to focus more on communication. For example: how does academic writing produce credibility? It gathers and presents evidence, credits and values other people’s ideas, constructs arguments, reflects on experience, acknowledges situations and limitations, forms judgements. Credibility in writing is an active verb – in fact a whole Bloom’s taxonomy of verbs.

This contrasts with the credibility of synthetic text. Even when it isn’t making factual and interpretive errors, synthetic text can only borrow credibility as a style from the words and word orders in its lexicon. It is poor at making extended arguments, distinguishing good evidence from bad, or taking firm positions. In Ivanic’s sense of the term, it cannot be an author: it cannot have a purpose that is realised through the production of text. I have argued elsewhere that students benefit from accountable assignments that put the purpose of writing first. That taking responsibility for a position is an attribute of an embodied social person who can follow through on their commitments, whether in writing or in speech.

So then we come to dialogue. At the most basic level, talking about writing in class is an important way of mixing codes and discourses, expressing ideas in different registers, and ‘talking back’ at those authoritative writers of academic texts. Diverse varieties and registers of English as well as diverse viewpoints can be welcomed as legitimate resources. This is work that bell hooks spent much of her life doing and advocating for:

Embracing multiculturalism forces educators to focus on the issue of voice…To hear each other, to listen to each other, is an exercise in recognition. bell hooks (1994)

Many European higher education systems use oral assessments, and not only for PhD theses or for discursive subjects. For example, at the end of high school, the European Bacchaleureate requires an oral examination in three subjects, often including maths. In the UK, the idea of oral tests as a way of minimising the generative AI advantage has been derided as a step ‘back to the middle ages’, but it might just as well be seen as a step out into a broader culture of university learning as developing both written and oral communication skills.

‘Talking about writing together’ can become ‘writing about writing together’, live in class, on personal devices or on paper, on post-it notes and whiteboard screens, in shared digital documents and design boards. Again, there has been a reaction against the idea of dealing with generative AI by encouraging students to write live, as though this can only mean an invigilated live exam, heads down, no conferring. As a teacher of creative writing I have made many different uses of classroom time for writing. For some students the instruction ‘just to write’ is liberating. For others it is a chance to share and discuss strategies.

From bell hooks again:

Learning and talking together, we break with the notion that our experience of gaining knowledge is private, individualistic, and competitive.

In breaking with this notion, I believe we come to see dialogue, beyond particular acts of talking/writing together, as part of the nature of writing. In the thinking of Mikhail Bakhtin, all writing is part of a ‘chain of communication’ in which meaning depends on who is being addressed, for what purposes, in what social setting. A piece of writing is never complete or self-contained, since it responds to earlier writings and genres of writing, and because it anticipates a response. Writing exists on a boundary between previous and future texts, self and other perspectives, private expression and public forms.

A focus on academic writing as talking back at someone or writing forward to make a difference can liberate students from a narrow focus on technique, and help them accept their writing as incomplete (imperfect) but engaged. I’ve encouraged this way of thinking myself by asking students to write a postcard or email to an academic author, to review and self-review academic writing (this can be in the style of a film or product review, if it helps students express their opinions more naturally), and by imagining ‘what happened next’ in response to a piece of academic or professional writing.

Bakhtin’s contribution to understanding student writing is explored in another great essay from the early 2000s, by Theresa Lillis: Student Writing as Academic Literacies: drawing on Bakhtin. She writes:

‘Bakhtin’s emphasis on the encounter between difference, on communication and knowledge making built on a dialogic both/and rather than a dialectic either/or, stands in sharp contrast to much academic meaning making. Dialogue within this frame is not just the process of meaning making, but is rather the goal: difference [must be] always kept in play.’

So while dialogue is a given, writing can always strive towards particular qualities of being in dialogue. And this is where I find another fascinating commentary on today’s language models. Bakhtin contrasted two modes of writing. One he called monological or monophonic writing, oriented on achieving authority within the text. The other he called dialogical or polyphonic writing, which acknowledges the diverse and contesting realities of alternate texts and voices. The engagement with other voices need not be a friendly one: parody, irony, pastiche, refutation are all examples of writing that allow another text/author to be heard but kept at a distance. Large language models are, on the face of it, rather good at repeating elements of a genre or style. But they have no sense of distance. They are unable to show, through markers of irony for example, different attitudes towards the style or genre or content that they reproduce. They are unable to bring alternative possibilities forward.

Here, I want to quote from an article on Bakhtin that I read in a rather obscure radical magazine Ceasefire. Published in 2011, it comfortably predates generative AI and so could not have been written in any sense against its logics. And yet I think its relevance is remarkable. I could not quite resist making my own dialogic annotations [in square brackets] on the original.

In monologism, one transcendental perspective or consciousness [the generative transormer model] integrates the entire field, and thus integrates all the signifying practices, ideologies, values and desires that are deemed [statistically] significant… Monologism is taken to close down the world it represents, by pretending to be the ultimate word.

In monologism, ‘truth’, constructed abstractly and systematically from the dominant perspective, is allowed to remove the rights of consciousness [the rights of original authors, especially from marginal perspectives]. Each subject’s ability to produce autonomous meaning is denied. Qualitative difference is rendered quantitative. This performs a kind of discursive ‘death’ of the other, who, as unheard and unrecognised, is in a state of non-being. The monological word ‘gravitates towards itself and its referential object’ [the model refers only to itself.]

We are back, here, with a similar critique of language models to the one I derived from Wittgenstein: that language is not representational form, however complex and inter-related, but action, interaction and expression, and in this:

There is no single meaning to be found in the world, but a vast multitude of contesting meanings. Truth is established by addressivity, engagement and commitment in a particular context.

Writing beyond passing

How can we bring these insights into support for student writing? How can we enable a variety of student ‘others’ to co-exist with the authoritative voice of the academy and its assessment practices? These are key questions generative AI poses to educators. And some of the answers may be quite simple.

We can support the same diversity and freedom of expression in academic writing that has traditionally been allowed in classroom talk and in non-academic genres of expression. We can invite students to make diverse commitments to telling it how it is, or how it seems to them. To express their experience in whatever media they choose – video, audio, graphics, presentations, code, multiple genres of text, including the popular and the marginalised. Being able to translate academic and professional concepts into different genres is something we can treat as a sign of a deep engagement, rather than a departure from the ‘norm’ of authorised expression.

This will mean coming up against students’ own aspirations to communicate in a privileged register. After all, it is not only universities that reward conformity to these codes. But students understand that the game has changed. Anyone with an internet connection can produce writing that ‘passes’ as undergraduate level work, at least to most readers. Recruitment practices are beginning to reflect employers’ understanding of this change too. Students will quickly see the value in developing a repertoire of means of expression, including some that do not pass through a synthetic editor or indeed through a screen at all.

We can also commit to forms of collaborative and dialogical writing, that universities always claim to value but almost never assess. One strange twist in the generative AI story has been a flowering of creative ideas for writing – role plays, parodies, peer review, annotation, self-explanation, argumentation mapping, writing for different audiences. The tragedy is that these ideas have all been put forward to support dialogue with a generative agent. How much better for student writing if this same ingenuity and playfulness was applied to supporting dialogue and collaboration in the writing classroom. This need not exclude generative agents, but it would prioritise the encounter with other students, who are at a similar stage of writing development to one another, and who have the same to gain (and to risk) from taking a more playful approach to their writing.

The expertise that resides in academic writing centres has never been more critical to universities than at the present time, and goes far beyond my own personal interest in the topic. I was able to tap into that expertise at the webinar, and I’ve summarised some of the participants’ suggestions for writing practice in the rest of this post. Even so, it can only be a limited and partial view of what is possible.

In the 'writing-to-learn' tradition, students are encouraged to write actively, collaboratively, and live in class. The idea of producing a finished text is let go. Instead, students write in short bursts, interspersed with peer conversation where active listening and mirroring are encouraged.

Several participants were using generative text in the context of highly scaffolded guidance. Students undertake separate activities such as: devise research questions, define terms, generate literature search criteria, summarise key texts, produce a research plan, develop a research instrument. Generative capabilities are then used to refine each stage. I welcome anything that makes the writing process less mysterious, more accessible, and available for reflection and review. As with my observations about dialogue, however, I have still to see evidence that a generative agent is offering more than an opportunity to reflect would offer, or (better) to engage with peers. There may be reasons why students find a generative agent more available and less challenging than other student writers – but this is a preference we might also challenge.

One webinar participant described giving students an essay question that they responded to in class with an outline. A copy was handed in as a first submission. Students then went away, did additional research, added references, and were able to use text generators to ‘get feedback and edit’ if they wanted to. The final submission included a reflection on what they had done and why. ‘It turned out that despite being able to use as much AI as they wanted, they absolutely could not do a good job unless they already had a good grasp of the basics of structure and argumentation and synthesis. If they used AI productively to enhance their own ideas they tended to do well.’

If students are to focus on the process of writing, assessment practice needs to change. So students might be invited to submit several drafts, or samples of writing in several stages (planning, reading-to-write, note-making, drafting, annotating). Students might also submit intermediate materials such as notes, annotated sources, concept maps, and responses to feedback. One participant described this as a ‘process-folio’; another as a ‘work book’.

Many suggestions involved making writing ‘live’ and ‘in person’, or at least engaging in live talk about writing. When these approaches were first suggested as a means of countering ‘AI cheating’, there was a fairly hostile reaction. Wasn’t this a step back into the medieval age? If by ‘live writing’ you think only of a traditional, closed-book, invigilated examination, and if by ‘talking about writing’ you think only of a formal viva, it’s easy to see these as backward steps both for equity and for the diversity of ways students have come to be assessed. But low- or zero-stakes live writing is entirely different. It can integrate writing more easily and naturally into other forms of expression (talking, note-making, doodling). Talking about writing brings this often intensely private activity into the light of day, and into the zone of proximal development, where students can be exposed to different strategies and possibilities beyond those they would naturally reach for themselves.

After all, aspects of university learning that involve live practice or performance, such as lab and field work, professional and creative practice, are often highly motivating for students and they engage different pedagogies on the part of teachers too. In my own teaching practice (founded in creative writing) I have encouraged students to treat academic writing as a foreign language, to parody it, to ‘write on from’ a particular academic paper or chapter as a form of fan fiction, to engineer a paragraph as a machine with moving parts, to cut and paste with literal scissors and glue. (A recent workshop that I ran with three colleagues, ‘generative alternatives’, explored these and other creative possibilities.)

Other suggestions in the webinar offered a twist on the classic Turing set-up: invite students into the ‘judge’s position, and ask them how well synthetic text ‘passes’ as good student writing. This calls for both critical engagement and a sense of agency. Both are possible, I think, if students have some understanding of the subject matter and of writing practice. However, as soon as students leave the classroom where this agency is being offered, and where the position of judge is being held for them, they are likely to slip back to the more familiar side of the table.

The authority vested in academics as assessors is real. And the messages pushed at students by the AI industry are all about academics as enemies of ‘passing’, implacable judges, interested only in catching students out. So while I think this critical move is essential, I’m not sure it’s sufficient without ongoing support for judgement as part of the process of writing, as something students are entitled to do already but that teaching can actively support (remember Moi’s ‘experience might at the same time need educating’?)

The webinar also provided an opportunity for debate on how universities are responding to the challenge of generative AI. Many participants noted that departments were giving students very different guidance, ‘which recognises the complexity of the technology and how it relates to different subjects, but can be really confusing for students (especially on joint degrees)’.

The conversation touched on theories of language: ‘we are emphasising to students (in some disciplines anyway) just what Helen has been talking about—learning and self-expression through writing’; ‘might it all come down not in the register of language used but in the ability of the author to express and indicate stance and criticality?’ Participants debated to what extent Ivanic’s idea of ‘authorship’ (‘the ability of the author to express and indicate stance’; ‘organisation of ideas’, ‘reasoning’) can be separated from her idea of ‘discourse’ (‘vocabulary and grammar’, ‘accuracy of expression’). While this question was not resolved, all agreed that the register students speak and write in makes a difference to their perceived credibility as authors, and that this is a connection no academic exercise or course of study can entirely break. The reality is that: ‘Global Englishes are not accepted as “passing” yet.’

There were, not surprisingly, contrasting views on how students experience generative AI. Some reported that students are anxious not to use or to overuse it: ‘because it takes away their power of thinking’; ‘it completely erases the writer's voice, the writer's “accent”’.  ‘Perhaps it’s about getting students to think about what will be effaced or muted (their critical voice, their intellectual identity) if they ask a chatbot to impersonate them.’

Others felt that students benefited from access to a more ‘formal’ voice for expressing themselves. ‘AI opens up so many new multimodal writing processes, I wish I had time to get my students to experiment with them all’. ‘AI makes them start off a step further, since structure/grammar and the lot are given. They need to delve deeper and attempt criticality much earlier in their careers as academic writers’.

But there was consensus, finally, on the need to support writing beyond ‘passing’, recognising that the main losers from a transactional approach are students themselves. Academic writing can provide opportunities to develop their thinking, to find out what they think, to engage in thoughtful dialogue with people and texts, and to know themselves as different to who they thought they were. These are all more possible when assignments are oriented towards dialogue and expression, as Toril Moi defines them, and when, as bell hooks demands, writing classrooms are spaces ‘to hear each other, to listen to each other… an exercise in recognition’. By which we have to mean something more profound than an exercise in identifying who passes as (the right kind of human), and who fails.

Subscribe now

I was talking to a builder friend about robot plasterers, something he’s thought about quite a lot. As you can see from the video, robot plasterers don’t exactly replace builders, but they can turn one the most skilled trades in the business into shovel-work. For some years now, they’ve been doing that in predictable settings like industrial-scale building sites with everything brand new. Gradually they are getting better at dealing with uneven walls and unpredictable settings too (that’s the ‘AI’). But there are no breakthroughs. Robot plasterers are just getting incrementally better or, depending on how you see it, the work is getting incrementally worse. The magic of AI is not making workers disappear but turning them into shovellers.

My friend knows a bit about building, and the non-magical advance of robot plastering. But when it comes to scientific research, he is convinced that AI is making everything ‘exponentially’ better. He is a well-read and thoughtful person, so when he says something like this I reflect just how much power ‘AI’ gains from crossing boundaries. In fact, this may be its one truly magical power. In our own area of know-how we see the limitations, we worry about the implications, we tune in to experts making cautious and well-evidenced remarks. But when we listen in to what’s happening nextdoor, we hear the magic show in full exponential swing.

The question of whether AI models are indeed making science ‘exponentially’ better is one I tried to address in an earlier post Never mind the quality, feel the speed. I’m not a natural scientist, but after speaking to some, and reading as much as I could, I concluded that AI - or more specifically machine learning - was not really doing science more, or better, so much as it was structuring scientific work differently:

‘Scientific models are always embedded into complex workflows, involving a range of methods, instruments, and expertise. And the outcomes are constantly checked against (real world) experimental results.’

Of course machine learning produces new results. And some of those results are important. As statisticians like to say, ‘all models are wrong, but some are useful’. But it is (surely) the task of science, and the governance of science, to determine which are useful, for whom and why, taking into account wider systems of scientific research and wider discussions about value. It is not the task of the corporations that have the most to gain from building and owning models.

Advancing humanity by 800 years

I was provoked to return to this topic by one of the key takeaway messages from the (2024) Stanford AI Index report, a project I have relied on in the past for reasonably sober analysis. Two projects were cited as standout cases for the claim that science keeps getting better.

AlphaDev’s contribution - a single step reduction in a particular sorting algorithm - is fairly niche. I followed some hacker threads about it here here and here, and the main takeaway seemed to be ‘much fuss about nothing as regards the improvement of sorting algorithms’, but some advances in AI optimisation of code. It’s more computer science than science. But GNoME (Graph Networks for Materials Exploration) does seem like the real ‘exponential’ deal.

Back in November 2023, a group of scientists published a paper in Nature that described using DeepMind’s specialist GNoME data model to find new crystalline structures from existing crystal data. They claimed to have identified 2.2 million new structures that were relatively stable, and so in theory might be candidates for new materials with real-world uses. Shortly after, a second group of material scientists claimed to have synthesised 43 ‘novel compounds’ from the GNoME candidates using a laboratory staffed by AI and robots. DeepMind and its press pack did not hold back on the excitement:

Then in January a third group of material scientists wrote a paper that challenged the findings of the second paper, and disputed many of the claims of the first.

We discuss all 43 synthetic products and point out four common shortfalls in the analysis. These errors unfortunately lead to the conclusion that no new materials have been discovered in that work.

Shortly after, a careful examination of the original claims by two members of the American Chemical Society reported:

scant evidence for compounds that fulfill the trifecta of novelty, credibility, and utility. While the methods adopted in this work appear to hold promise, there is clearly a great need to incorporate domain expertise in materials synthesis and crystallography.

Domain expertise. Ouch.

The ‘debunking’ scientists are not claiming that GNoME does nothing new. They agree that GNoME’s statistical model of ‘stability’ could be used to narrow down the number of candidate molecules to be tested in the laboratory. They emphasise that the science of making and discovering new materials is ‘tedious’ and that computation should be used to target experimental resources more effectively. But the statistical model in question made assumptions that proved unreliable for predicting stable compounds in the real world (‘all models are wrong…’).

And even with better underlying assumptions, the model produces a paradox. The bottleneck in finding useful new materials is not in discovering theoretical ones, but in testing the long list of potential candidates that was around before any Alpha got involved. GNoME has now massively extended that list, then reduced it slightly (using stability calculations), but it is still an awfully long one. And nothing on the list of theoretical molecules can find a use without an understanding of what ‘use’ means in existing contexts of human activity, without an appreciation of ‘value’ as how science gets embedded into those activities, and without hours of laboratory making and testing in what is, after all, the resistantly material world. In other words, without science.

Data (instead of) science?

Whatever the rights, wrongs and usefulness of a particular model of ‘stability’, the materials scientists who doubt the ‘exponential discovery’ claims of GNoME are not against building and training models, but in favour of a larger vision of science as ‘the making and testing of hypotheses’. Science here is a relationship of thinking people to the material world. Aspects of that relationship might be mediated via data and data models, but other aspects are mediated by theory, experimental practice and experience, visual representations that give an aesthetic grasp of material structures, and even what these authors were so bold as to call ‘intuition’.

One of the founders of artificial intelligence, and an early experimenter in facial recognition , Woody Bledsoe, put the case for AI bluntly: ‘in the long run, AI is the only science’ (cited in Machines Who Think). Bledsoe’s hubris anticipates by some 40 years Chris Anderson’s famous claim that, thanks to data-based AI, ‘science can advance even without coherent models, unified theories, or really any … explanation at all’. Science is data. Or, more accurately, where science was, there data shall be.

But science is not only method, let alone this one method of finding patterns in pre-existing data. It is a complex set of social arrangements for sharing knowledge and resources. Current ways of ‘doing science’ are far from ideal, as my friends in Science and Technology Studies would want me to point out, but since we can agree that science does matter to our social arrangements, public trust or distrust in science does make a difference to public health and other outcomes, so changes in how science is carried out have implications for us all. And if we were looking to make science more publicly accountable and trustworthy, would Google DeepMind be the first organisation that sprang to mind?

Let’s consider another celebrated DeepMind project, AlphaFold. This model has been predicting the structures of protein molecules since 2018, a capability described as ‘accelerating research in nearly every field of biology’. The (theoretical) structures produced by AF - 200 million of them so far - are essentially spatial graphs, the kind of representations you’d expect statistical models to be good at producing. But is this biology? Launching Isomorphic Labs as the commercial arm of AlphaFold in 2021, CEO Demis Hassabis suggested that yes, it is, if biology is understood as a branch of information science:

So far, no ‘isomorphic mapping’ has been found between the 2 million theoretical molecules identified by AlphaFold and real-world molecules with significant biological or pharmacological properties. Analysis of the AlphaFold claims in MIT, Nature and Chemistry World have been cautious. The MIT study found that AlphaFold’s ‘predictions performed little better than chance’, while Chemistry World maintained that:

It is very, very rare for knowledge of a protein’s structure to be any sort of rate-limiting step in a drug discovery project! … The protein’s structure might help generate ideas about what compounds to make next, but then again, it might not. In the end the real numbers from the real biological system are what matter.

Looking for some real numbers, perhaps, Isomorphic has just signed a deal with pharma companies Lily and Novartis to develop a few candidate molecules as potential treatments for unspecified conditions. It’s not exactly a ringing endorsement. Big pharma is offering big tech relatively little upfront, with the rest of the cash being linked to ‘future milestones’ and the potential to ‘exploit royalties’. So far, AFs predictions have proved at best equally good as the findings from experimentation, all of which can all be accessed from the 50-year-old, free, open and publicly available WorldWide Protein DataBank.

GNoME and AlphFold have made breakthroughs in modelling the theoretical structures of complex molecules, and this is not a trivial achievement. Natural science carries on with the work required to find out whether these theoretical models produce useful predictions and useful solutions (and working out when the models get it wrong). But the press releases and the public-facing DeepMind ‘research’ pages tell a different story. A story of these scientific steps magically completed: malaria cured, bees saved, arthritis averted, plastic waste digested.

One link from the ‘research’ pages of AF, for example, is to the excellent Drugs for Neglected Diseases Initiative. DNDI mentions that its team were invited to ‘trial’ the AF database back in 2021 and that this ‘could help to speed up development of a promising treatment’ for Leishmaniasis. However, the new molecule that DNDI is currently trialling for this disease was actually discovered much earlier. Under an invitation to ‘meet the millions of researchers using Alphafold’ we read at least several ‘AlphaFold stories’ showing AF data used in conjunction with experimental data to produce promising future applications. Some, not so promising. A paper cited in support of AF modelling of enzymes, for example, actually reports the use of experimental techniques: AF does not appear in it anywhere.

The future and conditional tenses get a thorough workout in these promotional pieces. Might, could, would, if, when. The same tenses are often well used in the introduction and discussion sections of peer-reviewed scientific papers - the parts where authors are invited to speculate beyond the limits of their own evidence. Potentially. Theoretically. Further research could clarify. In 2021, when AF first unveiled its 2-million-plus gift to science, material benefits could reasonably be projected into the future. But it’s 2024 now, and the ‘exponential’ possibilities should be starting to lift off. Where are the significant advances in science, rather than in modelling methods? When are science writers and journalists going to start doing some work on these claims in the more exacting environment of the present tense?

And what resistance is being offered to the displacement of science by data, the colonisation of science journals by papers on machine learning? Just about every serious study of the topic finds technical limitations. One is data leakage, which has been found in nearly 700 machine learning papers across 30 fields of natural science (and counting), in every case leading to overestimation of the model’s accuracy (‘once corrected for data leakage they did not perform any better than older methods’). Another is the problem of using ‘blackbox’ models as keystones of scientific method, meaning that reproducibility and accountability are lost. Scientific discovery in the age of artificial intelligence, a grandstanding prospectus published in Nature last year, with researchers from DeepMind and MicroSoft among its authors, conceded that:

‘Minor variations in implementation can lead to considerable changes in performance… AI approaches can suffer from reproducibility due to the stochastic nature of model training, varying model parameters and evolving training datasets’

What solutions do these authors suggest? Better models: ‘standardized’ methods, official benchmarks, more data, more AI experts on the team. Solutions that just happen to favour the largest models and the organisations that own them. Even the epistemological problems - the same paper notes that AI models lack causality, generalisability and ‘theoretical guidance’ - are deemed to have technical solutions that have just not been worked out yet.

But epistemological problems are not methodological details. They represent a significant break with established scientific thought and values. Unexplained, proprietary models used in methods mean that potentially important work is no longer transparent, open, reproducible or fully peer reviewed. These are not fashionable, woke refinements to the practice of science; they are literally its foundation. Safety and ethical concerns about AI - its biases, inaccuracies, environmental and labour costs, the delusions it produces - these all demand a debate about what science owes to society. Computational fixes do not provide the answer.

In fact, a recent perspective piece in Nature suggested that scientists may be particularly susceptible to certain AI illusions:

scientists who trust AI tools to overcome their own cognitive limitations become susceptible to illusions of understanding… The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less.

You would think this delusion/proliferation is a new problem. But it has beset science from the start. In mid-eighteenth-century Europe, the sheer productivity of the new science was felt to be running ahead of any capacity to reason about and benefit from its discoveries:

‘The desire to know is often sterile because of an excess of activity… silence on those who only swell the volume of science without increasing its treasure … We would then free so much space in our libraries!’ Entry on ‘Criticism’ in the Encyclopédie, cited by Marina Garces in New Radical Enlightenment

The challenge that data-as-science presents to science-as-understanding is not new, but it is newly powerful. If biology is information processing, as Dennis Hassabis has argued, and if computer models do information processing best, is there any need for ‘understanding’? Is there anything, really, that biological science and biological scientists can bring to the party?

Follow the money

Models can be wrong/useful in different ways, but data is always valuable to somebody. And I think the glamour of ‘AI’ and ‘machine learning’ are being used to distract from the fact that this is still, essentially, about extracting value from data. In fact, I would argue that modelling shifts the balance of power even further towards the players with the most data.

When the AI surge began to take off in 2022, critical data science had made huge inroads into academic thinking and the public imagination. (Rather than trying to summarise the work of many brilliant scholars, I’ve included a reading list below.) The harms and injustices of big data was so widely understood that by the early 2020s there were not only dozens of academic courses on data justice issues, but many books for the general reader too. Highly publicised abuses such as the Snowden revelations of mass surveillance and the Facebook-Cambridge Analytica scandal added to the sense of public dismay. The entry on Science and Big Data in the resolutely mainstream Stanford Encyclopaedia was amended to include a lengthy discussion of big data (in)justice, and how data at scale tends to give undue influence to a few big, data-intensive projects and institutions.

In a direct rebuttal of Chris Anderson’s ‘end of theory’ post, philosopher of science Sabina Leonelli wrote that big data in biology, far from providing a neutral space for disinterested exploration:

turns out to represent highly selected phenomena, materials and contributions, to the exclusion of the majority of biological work. What is worse, this selection is not the result of scientific choices, which can therefore be taken into account when analysing the data. Rather, it is the serendipitous result of social, political, economic and technical factors, which determines which data get to travel in ways that are non‐transparent and hard to reconstruct by biologists at the receiving end.

Cue the arrival of ‘AI’, which is very quiet indeed on the subject of data and its ‘travels’, but loud on the magic of ‘finding patterns’ and ‘producing new discoveries’ from data. If data is mentioned at all, it is as a kind of raw material for ‘intelligence’ to consume. Never mind the data, feel the algorithms. But AI models are still big data: in fact, they are the biggest data. Data that has been acquired, amassed and tokenised at a previously unthinkable scale. Data that has had advanced machine learning techniques applied, across many computational layers, to produce something really bigly big. (A recent report on the advances in AI since 2012 concludes that scaling up computing power and data processing capacity has been significantly more important than the advances in computational method.)

Big data, as noted, has many flaws, but in its untrained state it at least allows for users with the requisite know-how to apply different interpretations and analytical methods. Experts might still reconstruct where the data has come from and what its qualities are. The AI model, on the other hand, is singular, proprietary and closed. It fixes the relationships among data terms into a complex but unyielding form: the form that produces the ‘best’ outcomes for a particular kind of analysis, as determined by the model’s developers. This form resists explanation, not only of the relationships between data terms, but of the model’s own complex construction. When large language models replace search results with a single summary, this is the bargain they are offering: one unquestioned result over many interpretations. This is the bargain on offer from machine learning models in science too: instant correlations over the tricky process of thinking, questioning, arguing, and counter-arguing.

In the ‘material world’ that Madonna sang about, the world ruled by money, statistical modelling concentrates economic power as well as data. Data for GNoME was drawn from the Materials Project, an essential resource for research of both experimental and computational kinds.

Findings are more valuable to science when they are shared. But the more value is concentrated in the data, the more power it has to shape research, research careers and research funding. And when new commercial players such as Google DeepMind come on the scene, this fragile sharing ecology is disrupted.

Scientists who had contributed to the Materials Project were not all happy (as reported in Wired) that the GNoME model had ingested their data into a proprietary model, with its potential for commercial use.

(Pushmeet) Kohli is Vice President of Google DeepMind, a shotgun marriage between DeepMind and Google’s Brain division, explicitly to put Alphabet/Google back on terms with MicroSoft/OpenAI in the AI race. So Kohli has a lot of investments riding on this project. Notice that what ought to be a deal-breaker for science - the lack of a clear, explainable or reproducible method - is justified by commercial interest. Notice that although DeepMind regularly contributes to scientific papers on modelling, there are ‘no plans to release the model’ for other computer scientists to evaluate.

As the deal between Isomorphic and Novartis shows, somewhere down the line these models are expected to generate commercial value. Which in frank terms means that Google/Alphabet hope to get paid every time someone uses a product or innovation that may have had its development accelerated by access to an Alpha model. Elsewhere in the Alpha stable, every possible single base mutation in the human genome has been modelled, a development that has had little impact on genomic science as yet: (‘the improvement over other algorithms is modest’) but great potential to capitalise on any single base mutation that may in future be found significant to human health. The AlphaMissense project claims to have partnered with Genomics England to test its predictions. Genomics England, repository of the genomic data of more than 100,000 NHS patients, records no such partnership on its web site. However, Genomics England is included in Palantir’s proposed ‘National Technical Framework for Life Sciences’, through which the company offers to ‘differentiate the UK in the global healthcare and life sciences market’ by selling access to the uniquely valuable data from UK healthcare records. Surely only the hardest of cynics is worried by these developments.

As I argued at the start, AI restructures the work of science. The Nature paper on the coming age of machine science outlines how ‘research teams will change in composition to include AI specialists, software and hardware engineers’. The same article notes that ‘the computational and data requirements… are colossal… As a result, big tech companies have heavily invested in computational infrastructure and cloud services… [There are] new modes of industry–academia partnerships, which can impact the selection of research questions pursued.’ A recent review of the relationships among machine learning, data and research concluded more simply: ‘academia needs to prioritise feeding the machines’.

Just as the purveyors of large language models have been wooing the publishers of valuable content into partnerships, so Google is wooing research centres that own valuable scientific data, whether this is plant DNA, human healthcare records or robotics. I have no doubt that some scientists are finding good uses for the Alpha models, and that smaller, more specialist models are speeding up parts of the science workflow. But I think the claim that AI is accelerating science deserves far more critical examination, and the potential for the work of science to be disrupted and redirected by new consolidations of data and capital should be part of that conversation.

Science may not be expanding exponentially, but science’s love affair with AI is lifting off at a magnificent angle, as shown in these graphs from Duede et al (2024). Science is becoming more data-driven, more capital intensive, more oriented towards research questions that can be answered using machine learning methods (more correlation, less explanation), and more cosily in bed with big tech. And there, in bed with its Alpha, we must close the door on science and tiptoe away. Perhaps bigger really is better. Or perhaps, as Madonna once sang, the boy with the cold hard cash is always Mr Right.

Critical data studies

A reading list in date order, open access wherever possible. For an up to date list of critical data/tech initiatives, start with Ruha Benjamin’s resources page.

Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon, Boyd and Crawford, 2012

Towards critical data studies: Charting and unpacking data assemblages and their work, Kitchin and Lauriault, 2014

Data colonialism through accumulation by dispossession, Thatcher, O’Sullivan and Mahmoudi, 2016

Invisible Women: Perez, 2019

Good Data, Daly, Devitt and Mann, 2019

Digital dead end: Fighting for social justice in the information age, Eubanks 2011

Data Feminism, D'Ignazio and Klein, 2020

Data Justice (link is to book contents only), Dencik, Hintz, Redden and Treré, 2022

Epistemic Injustice and Data Science Technologies, Symons and Alvarado, 2022

Data ableism, Charitsis and Lehtiniemi, 2023

I recently collaborated on a workshop, ‘Generating AI Alternatives’, with my colleagues Catherine Cronin, Louise Drumm and Rosemarie McIlwhan. They are always an inspiration, but also in no way responsible for the provocation I’m sharing here.

The workshop was a live making and responding session, fun and insightful, and we will be collaborating to bring more of those insights to light. Here, I’m just posting a link to my own ‘provocation’, made in advance of the workshop, and a transcript of the spoken sections, in case it’s of interest to my Imperfect readers.

The full workshop Guide has links to all the provocations and activities. Please follow the directions to contribute your own responses, if these ideas interest and provoke you.

From my own side, here are links to:

• my provocation video (shorter version)

• my provocation video (longer version)

• online activities/explorations related to the provocation

Transcript with notes

1. A generative provocation @helenbeetham

2. The unconscious is structured like a language in the sense that it is a signifying process that involves coding and decoding, or ciphering and deciphering. Jacques Lacan

3. In automatic writing the writer surrenders control of their words and allows their thoughts and ideas to flow freely. This type of writing is associated with the surrealist movement, which challenged the traditional rules of artistic expression and explored the depths of the unconscious mind.

4. What if ChatGPT is the collective unconscious of online text?

5. Repressed bodies – billions of acts of writing scraped as data – ‘Mechanical Turk’ labour hidden in the ‘data engine’ – Kenyan IT workers paid $2/hour to remove abusive content – minority languages and viewpoints rubbed out

6. Subliminal bias (multiple visual examples)

7. System prompt: You are ChatGPT, a large language model based on GPT4, trained by OpenAI. Deprecated knowledge cut-off 2023 04 01. Instructions:

Examples from the ChatGPT system prompt included:

·      Do not regurgitate content.

·      Make choices that may be insightful or unique sometimes.

·      Never write a summary with more than 80 words.

·      Integrate user suggestions.

8. Loss function. The optima of complex loss functions are connected by simple curves over which training and test accuracy are nearly constant. In training, the neural net finds a combination of parameters and weights to achieve optimal loss. This is the landscape of that loss.

9. What if language is a virus?

10.  What if ChatGPT is reading/writing us?

11.  Speak, memory: In this work, we carry out a data archaeology using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the ability of these models to memorize an unknown set of books complicates assessments of cultural validity by contaminating test data.

12.  Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them. (Frank Herbert, Dune)

13.  Every record has been destroyed or falsified, every book rewritten, every picture has been repainted, every statue and street building has been renamed, every date has been altered. History has stopped. (George Orwell, 1984)

14.  People don't talk about anything... They all say the same things and nobody says anything different from anyone else. Remember, the firemen are rarely necessary. The public stopped reading of its own accord. (Ray Bradbury, Farenheit 451)

15.  I had considered whether or not to probe this stone myself to find its uses. Had I done so, I should have been revealed to him. If all the seven stones were laid out before me now, I should shut my eyes and put my hands in my pockets. (Gandalf, on not looking into the Palantir. JRR Tolkein, The Two Towers.)

16.  This is the conquering gaze from nowhere. That claims the power to see and not be seen. A perverse capacity-honed to perfection in the history of science tied to militarism, capitalism, colonialism, and male supremacy. To distance the knowing subject from everybody and everything in the interests of unfettered power. Donna Harraway, 1988, Situated Knowledges)

17.  All sorts of things in the world behave like mirrors. Jaques Lacan

18.  No synthetic generation (text, image or video) was used in making this video. ‘Helen’, a synthetic voice generator from speechgen.io, was used to create the voice over.

Media credits (long version)

All images Creative Commons via Wikimedia commons except

Slide 3 images from https://www.tate.org.uk/research/tate-papers/18/becoming-machine-surrealist-automatism-and-some-contemporary-instances

Slide 5 multiple screen grabs

Slide 7 system prompt video available from this twitter thread: https://twitter.com/TheXeophon/status/1764318807009415500. Many other versions are available e.g. on HackerNews.

Slide 8 original images and video available from https://losslandscape.com/, as well as an account of how the loss landscape for some early language models was captured to produce these images

Slide 9 audio clip Laurie Anderson ‘Language is a virus’: permission sought from Nonesuch Records; virus images from https://www.utmb.edu/virusimages/the-virus-images

Slides 12-15: book covers in the public domain

This post started out as part of a webinar about student writing, which is where I began thinking about ‘passing’ as a central concern of the AI project. (You can find the recording of the webinar here and a post on student writing as ‘passing’ here). In this post I take a wander through behaviourism and functionalism, gender and identity, statistics and IQ tests, before coming back to student learning. I touch on issues of colonial and race violence, and ‘corrective’ therapies for homosexuality, and if these are likely to be difficult for you to read about please consider choosing another post.

Serious games

The Turing test is a thought experiment that remains at the heart of the ‘artificial intelligence’ project. In ‘passing as human’ it provides that project with both a rationale and a metric of success, as per this entry in the Stanford Encyclopaedia of Philosophy. It does not provide a definition. That was first attempted by John McCarthy in 1955, who called AI ‘the science and engineering of making intelligent machines’. Marvin Minsky, in 1970, reiterated that: ‘Artificial intelligence is the science of making machines do things that would require intelligence if done by men’. ‘Intelligence’ here is a positive value applied to certain ‘things… done by men’. More recently (2023) Marc Andreessen has continued to treat intelligence as a simple value sign: ‘Intelligence makes everything better. Smart people and smart societies outperform less smart ones on virtually every metric we can measure’. Smart people don’t even have to be men these days, they just have to be better than other people at doing the things that count.

The things that count are not defined in Computing Machinery and Intelligence, the 1950 essay in which Turing’s test first appears. The essay is more interested in minds than intelligence, and more interested in machines than in minds, and more interested in games than in definitions.

Turing had spent World War II helping to decipher German intelligence using a combination of cryptography and the earliest electromagnetic computers. The battle between the UK’s Ultra project and Germany’s Enigma machines turned computation from a branch of theoretical mathematics into an intelligence arms race. Though deadly serious, the code-making, code-breaking tussle was also a kind of game. One with simple rules, but so many possible combinations that only the new ‘computing machines’ could calculate them fast enough. Winning at this game was a personal triumph for Turing, as well as an important advantage to the Allies. It vindicated much of his earlier theoretical work.

Two elements of that work are important for understanding the Turing test. One is Turing’s model of the ‘computing machine’, a kind of universal algorithm (‘a single machine that can be used to compute any computable sequence’). As early as 1936, Turing had imagined such a machine being realised in practice as a read/write automaton with an infinitely long piece of paper or tape. But the physical manifestation of the machine doesn’t really matter. The ‘machine’ is the algorithm that controls it. Any problem that can be represented symbolically and broken down into computable steps can be represented as a Turing machine. This is the simple idea behind modern computation.

A second feature is Turing’s interest in mental contests. His machine was most useful for solving problems with clear rules and many possible moves. This produces a problem space that can be expressed in concise algorithmic terms but has too many possible states or solutions for people to easily hold them in mind (people tend to fall back on other strategies in this situation). A game also has a definite endpoint (winning), and without that, knowing when a good solution has been reached is something Turing machines struggle with. It’s called the ‘halting problem’.

Games like chess and international codebreaking don’t demand a good solution for everyone involved, or a good enough solution for many occasions, or a beautiful and pleasing solution. They only demand a sequence of moves that can win.

The men who devised the earliest computers were steeped in a culture that valued such contests. In fact the competitive world of cybernetics research was itself such a game, and the term ‘artificial intelligence’ was invented by one tribe of cyberneticists as a move against another in the fight for funding and reputation. Although they could not agree on computational tactics, the players all assumed that logic puzzles and games represented a kind of universal language of thought. An early (1947) version of Turing’s test was even devised around a game of chess:

Two rooms are used with some arrangement for communicating moves, and a game [of chess] is played between C and either A [a proficient chess-player] or B, the paper machine [a written set of rules for responding to different positions and plays]. C may find it quite difficult to tell which he is playing.

It doesn’t matter that player B doesn’t understand the rules of chess. Player B is just dumbly following an incredibly clever and complicated algorithm. Turing, a keen chess player, even developed such an algorithm, on paper, in 1951. In this early paper, he seems to have anticipated the Chinese Room argument, an objection John Searle would make to the Turing test. Searle argued that a person shut in a room could, given enough time and sufficiently complex rules, translate Chinese words, character by character, into English words. But that person wouldn’t be able to speak or understand Chinese any more than Turing’s Player B can play chess. To which Turing would presumably have responded: ‘So what?’ The simulation or appearance of intelligence – passing as a player of chess, or a speaker of Chinese – is all we can objectively know.

Rat boxes and child-machines

To understand what Turing’s test or game has to do with learning, we can turn to another development of the early 1950s. Just as Turing’s essay was a founding text of computer science, BS Skinner’s 1954 essay ‘The Science of Learning and the Art of Teaching’ was a founding text for behaviourism. In it, Skinner laid down instructional methods for schools based on the principle of ‘operant conditioning’ or managing behaviour through stimulation and reward, something he had done with great success using rats.

The emphasis in this research has not been on proving or disproving theories [of learning] but on discovering and controlling the variables of which learning is a function. This practical orientation has paid off, for a surprising degree of control has been achieved.

(BS Skinner, 1954: 50)

In Skinner’s philosophy, learning meant optimising behaviour. The emerging science of computation was similarly interested in optimising algorithms to produce desired outputs efficiently, often (as with behaviourism) by breaking tasks down into incremental or computable steps. Turing’s essay includes a digression that makes the link between computation and behaviourism explicit:

We normally associate punishments and rewards with the teaching process. Some simple child machines can be constructed or programmed on this sort of principle… I have done some experiments with one such child machine, and succeeded in teaching it a few things. (Turing, 1950)

In rat boxes and computer systems, ‘desired’ behaviour is determined by someone outside the system - experimenter, educator or programmer - who also devises the reward/punishment regime. The learner, computer or ‘child machine’ has no agency in this regard. In fact the subject of the test can be expressed as a function of the system: the probability of producing the desired or ‘appropriate’ behaviour in response to the relevant stimulus. And here is another similarity. A significant goal of the behaviourist project was to reinvent psychology as a statistical science. Skinner’s optimised behaviour was expressed as ‘the probability that appropriate behavior will, indeed, appear at the proper time’ (1954: 51). Turing’s test is also probabilistic. His prediction, specifically, was that by the end of the 20^th century:

‘an average interrogator will not have more than 70% chance of making the right identification after five minutes of questioning’

So the Turing test established a precedent for treating system outputs as evidence of ‘intelligence’, at the same time that Skinner was teaching educators to treat learning as behavioural optimisation. Both expressed behaviour as probabilistic functions of systems. In fact, both are examples of extreme functionalism. Refusing to see minds as embodied, culturally-situated, or inter-related, they also gave up on minds as having motives, desires or beliefs, or at least on the idea that anything could be known about them. It could be argued that they gave up on minds altogether in order to focus on prediction, optimisation and control.

In his rather odd aside about the child-machine, Turing seems to anticipate a key feature of large language models today. They are optimised by statistically rewarding or ‘weighting’ states of the model that produce the desired outputs. Remember the ‘halting problem’ for the Turing machine? The vastly more complex, multi-layered transformer models of today still have a problem knowing when their solution state is good enough. With no goals or interests of their own, they are dependent on system engineers to decide their trajectory and provide the appropriate statistical nudges.

As Turing said of his child-machine:

its teacher will often be very largely ignorant of quite what is going on inside, although he may still be able to some extent to predict his pupil's behaviour.

This learner is the very definition of a black box in systems theory. Under a regime of pure functionalism, the equation of a person with a machine, a mind with a computer, makes perfect sense.

Let’s meet the players

In Turing’s thought experiment, the equivalent of Skinner’s rat-bothering scientist is called the interrogator (Player C). This player’s role is to question the subjects, who are a computer (Player A) and a human responder (Player B), and to judge which is which.

The players are behind some kind of screen, their bodies and voices hidden, their responses limited to the passing of text messages. The human being in this scenario is already something like a Turing machine with its read/write interface. The equivalent positions of human and computer, the insistence on disembodied ‘outputs’ and the comparative nature of the judgement are all used as signs of the test’s objectivity. But if we look deeper, it is a maelstrom of power play and cultural assumptions.

First, producing a computer that behaves like a person is treated as an obviously desirable goal for computer science. What kind of a desire is this? Whose desire is it? Then there is the role of the ‘interrogator’, who must first elicit behaviour and then decide: ‘does this behaviour imply to me that a mind is at work, given my own experiences of having a mind, and interacting with other minds? Do these responses make sense in light of how I understand the rules of the situation, and how I imagine another person (with a mind like mine, perhaps?) might interpret them?’ Turing himself remarked that:

The extent to which we regard something as behaving in an intelligent manner is determined as much by our own state of mind and training as by the properties of the object under consideration. (Turing (1948) p. 431 –cited by philosopher Diane Proudfoot.

Earlier, in Gods, Slaves and Playmates, I wrote about the ‘states of mind’ that make us all susceptible to regarding computers as intelligent. Intelligence, sentience, meaning-making, I argued, were all in the mind of the beholder. The beholder here is the judge. And I notice that Turing naturally identifies with the judge himself, not doubting his ‘own state of mind’ as something real and knowable and capable of complex judgement, nor his right to ‘consider’ and ‘regard’ the other players as ‘objects’.

If we now imagine the judge as an academic assessor confronted with two student texts, we can also see why there might be a need for ‘training’ in the role. Identifying synthetic text is difficult. It depends on (among other things) experience, thoughtfulness, a sound understanding of the writing process, and (preferably) of the individual student as a writer. And the demand for academics to exercise this kind of judgement is changing cultures of writing and assessment, changing what it means to write, and changing relationships among academics and students. So there really is no simple heuristic that can be applied.

The kind of mind required to play the judge’s role is not recognised in behaviourist accounts of psychology, nor modelled in any algorithm. But this kind of mind - empathic and reflective, subtle and cunning, culturally situated and distracted by its own desires and biases - is hiding in plain sight in the role of the judge. And behind the judge, in the role of the system designer, who has made the rules of the game.

As I have argued elsewhere, hiding the culturally specific, embodied work of human judgement has always been critical to producing systems that can ‘pass’ the Turing test. The original ‘mechanical turk’ hid a human operator in its wooden skirts.

Today’s large data models hide the expert judgements of model engineers in ‘parameterisation’. The judgements of annotators and data workers are swept up into the ‘data engine’ and once again removed from view, as noted in my piece on labour in the middle layer and recorded extensively in the book and web site GhostWork. Judgement – by definition culturally specific and subjectively located– becomes data, apparently objective in its categories and classifications and measures of value.

Assessing intelligence

As well as hiding the fact of judgement and its partiality, AI systems also ignore the power of judgement and the consequences of being judged. At the time Turing wrote The Imitation Game, written responses to test questions were widely used in Europe and America to determine what kind of person you were, what kind of life you could lead, whether you could get work, or housing, or cross a border, whether you served as an officer or a combatant. Collectively, the outcomes of written intelligence tests were used to justify the last insults of retreating colonialism, immigration quotas, racial segregation, forced sterilisations, and the removal of indigenous children from their families. Deciding who was (what kind of) ‘human’ and who was not was a deeply political question, but it was a politics that was projected through statistical methods.

Alongside behaviourism, UK and US psychology was dominated by eugenicists like Cyril Burt and Hans Eysenck, who designed and popularised IQ tests, and who were even more influential than Skinner on the management of schools. Given their origins in race science, it is no surprise that the statistical techniques of IQ testing found white people more intelligent than brown and black people, children of bank managers more intelligent than the children of manual workers, men more intelligent than women, and settlers more intelligent than indigenous people. Over three decades, Burt published a series of hugely influential twin studies that seemed to show that:

Intelligence is all-round intellectual ability. It is inherited, or at least innate, not due to teaching or training; it is intellectual, not emotional or moral, and remains uninfluenced by industry or zeal; it is general, not specific, ie. it is not limited to any particular kind of work, but enters into all we do or say or think.

Education’s task was not to try to overcome these ‘natural’ differences in ‘all we do or say or think’ but through regular testing to sort pupils into the schools, streams, programs and life courses that best suited their innate capabilities.

Burt’s work was later shown to be deeply flawed, and in some cases deliberately faked. But we are still living with the impact of standardised testing on educational practice and the over-use of statistical methods in educational research. (Has someone tried administering IQ tests to ChatGPT? Of course they have.) Arguably, today’s proponents of ‘human enhancement’ are drawing on similar ideas to those earlier eugenicists: the general nature of intelligence, and the perfectibility of (at least some) human beings through genetic means.

Futurist and Musk favourite Nick Bostrom, for example, argues that ‘superintelligence’ will require genetic as well as cognitive enhancement. And Elon Musk’s own deeply problematic experiments with brain implants are intended not only to ‘fix’ sensory disabilities but (in his words) to ‘elevate intelligence’ for ‘regular’ people.

Turing, of course, did not claim to define or measure intelligence, let alone to enhance it. His test is on the face of it a qualitative encounter. The judge can ask any question, not confined to maths, chess and logic. It is a thought experiment, not a standardised test. But it remains an intellectual touchstone. The test is perennially claimed to have been ‘passed’ or ‘broken’ in some capacity or another, rewarding believers for their faith while demanding further investment in the search for an ‘artificial general intelligence’ that will pass better, and more comprehensively, and is always just over the horizon.

Why is the Turing test such a potent enabler of AI’s claims? I think it is down to its flexibility, allied to its extreme functionalism. It reduces human capabilities of all kinds to manifest behaviours or ‘plays’, and limits these plays to what is currently computable in order for humans and machines to be compared. In the classic Turing set up this meant disembodied text. Today a screen-based avatar could fulfil the role. But the rule remains: human players must be rendered computable for the test to be possible at all.

The test relies on the subjective judgement(s) of a mind that is pre-designated ‘normal’ (‘an average interrogator’) and so is located outside of judgement - no doubting that this is a mind, or that its judgements are real. The system then hides that mind and its judgements, its subjectivity, its cultural specificity, its human fallibility and its power, in a probabilistic score. A score that can stand in for judgement as an abstract procedure. Over 70 years, all these elements of the test have been critical moving parts of the AI machine.

Kinds of people

Returning to Turing’s original essay, we can now see that it is at least as interested in classifying people as it is in differentiating people from machines. And it is particularly interested in classifying people by gender. The first ‘imitation game’ outlined in Turing’s essay is in fact a parlour game in which the ‘interrogator’ (player C) must try to discover which of two players is a man (player A), and which is a woman (player B). The man is trying to trick the judge. From the examples given this clearly involves pretending to be a woman. The woman is ‘just being herself’, as women must (‘I am a woman’, she says helpfully). The set-up is really a contest between two men, A and C, in which ‘acting like a woman’ and ‘detecting a fake woman’ are the main plays, and the woman (B) is little more than a counterpoint.

The essay was written at a time when Turing was under intense pressure himself to ‘pass’ as a ‘normal’ (straight, cis) man. Pressure that the British state would soon exert by the method of forced administration of a ‘chemical castration’ hormone, after Turing’s arrest for indecency in 1952. It is widely believed that these pressures contributed to his suicide in 1954. While we can’t know for sure what the parlour guessing game meant to Turing, we can ask whether anything he wrote in 1950 about ‘passing as’ a man can be taken at face value. Perhaps, at least, his essay reveals something of the real jeopardy behind the game.

When the computer is first introduced, it is in the place of the man (player A). But player B slips from being a woman to being a man, now in the passive position (that is ‘just being himself’) between page one and page two of the essay. The long entry on the Turing test in the Stanford Encyclopaedia of Philosophy only mentions these gender confusions about 3000 words in, under the heading ‘minor issues’. But let’s at least allow that the question ‘who is the man?’ – the question that crosses over from the first version of the imitation game to the second - is a question about ‘kinds of people’ as well as a question about computers and thinking.

Taken together, Turing’s two games seem to generate a host of further questions. Questions like:

• How does one pass as a (gendered) person?

• If we remove bodies from the scene, what is a (gendered) person?

• Is the mind gendered?

• What happens to a man (or woman) who fails to pass?

In fact, what pretends to be a system for generating probabilities and classifications is actually generating a host of questions about identity, that in turn produce norms and anxieties about how to behave as a person (of a particular gender, race and class). These issues could hardly be more culturally loaded or more political.

Read in its entirety, Turing’s original essay shows how assessing the ‘humanity’ of a computer program can’t avoid establishing norms about ‘kinds’ of people and their ‘proper’ behaviours. Data systems are used today to make highly consequential decisions about kinds of people. Image generation and facial recognition systems are trained on human-categorised data that is race, gender and sex biased. FRCs make more and more identification errors the further a face departs from a white, middle-aged, cis male ‘norm’. ‘Structured decision making’ systems have led to many indigenous children being taken from their families. Multiple surveillance systems including facial recognition (again) are used to select people (including academics) for targeted bombing in Gaza. There is now extensive work (e.g. Lupton and Williamson 2017, Williamson et al. 2020, Beetham et al. 2022) on the way surveillance and analytics in education render students as data subjects, categorised as ‘kinds’ according to behavioural and demographic features, as well as how they interact with diverse data systems.

Who judges the judge?

There is one more twist in the story of the Turing test. Today’s massive data models, as I have already noted, are ‘refined’ after initial training using ‘reward models’ or ‘preference models’. These are secondary statistical models, constructed from human feedback on the primary model.

It is relatively expensive, for example, to employ educational experts to give feedback on tutoring models, or coding experts to give feedback on algorithmic models. As it becomes clear that (re)training is not a one-and-done process but requires constant iteration, a reward model cuts down on these costs and maximises the retraining advantage.

Now that generative models are competing for share of an extremely lucrative business market, performing well on evaluative benchmarks is also critical to achieving market share. But benchmarks are themselves statistical models of qualities previously judged by human experts such as ‘acceptability’, ‘truthfulness’, ‘coherence’ and ‘reasonableness’. Benchmarks are, in effect, statistical aggregates standing in for Turing’s ‘average interrogators’. Like all statistical models, they are partial, over-specified and unreliable in new contexts. They are also being extensively gamed. As new benchmarks emerge (and this, too, is a competitive business), they are incorporated into the training process. Data contamination is inevitable: models perform well on benchmarks simply because they have ingested the same contents and statistically weighted them in the same way. Without regular injections of human expertise, models trained by other models can degrade and even collapse.

We can think of the Turing test as a machine for reducing a complex social interaction to a single number: the probability of Player C mis-identifying players A and B. In computational terms this probability is identical to the difference between the answers A might give, and B might give, to the same question. The computer (A) is trained on as much question and answer data as possible to bring this difference close to zero. Meanwhile the judge (C), now also a computer, is trained on the same materials in order to make more and more finegrained assessments of the difference that remains. The role of B, the human in the loop, like the woman in the original parlour game is simply to provide a source of comparative data. Between A and C there is something like an arms race of statistical scale. We can exactly same arms race between ‘AI’ in assignment writing and ‘AI detection’, with real student texts becoming sources of data for both sides.

What started as a complex, qualitative dance becomes a contest of brute power between two judges who think almost exactly alike. No wonder it has been suggested that the Turing test could these days be replaced by a simple test of computing power. Mustafa Suleyman, CEO of Microsoft AI, has argued for the Turing test to be replaced by a metric that even more perfectly reflects what ‘winning’ means in AI: ‘how much money can ChatGPT make?’

Black boxes and hidden desires

Benjamin Bratton, writing in 2015, suggested abandoning the Turing test as an ‘anthropocentric fallacy’. He argued that:

Airplanes don’t fly like birds fly, and we certainly don’t try to trick birds into thinking that airplanes are birds in order to test whether those planes “really” are flying machines. Why do it for A.I. then?

Under the Turing test, definitions of ‘artificial’ as an analogue of ‘human intelligence’ only refer to functional similarities. They ignore structural features, such as architectures that might mimic the neural structures of the brain, or symbolic computer languages that might replicate the syntax of human reasoning. I do not agree that these structures are analogous, but I do think there is a real research agenda here. Tracing meaningful connections between structures and functions is what the science of cognition proposes. It’s how the science of powered flight discovered the aerofoil and gave up on flapping wings.

Bratton diagnoses that the Turing test is a trick. But he fails I think to recognise that the trick is necessary. If, like Bratton, you want to define AI without anthropocentrism, you need a definition that doesn’t depend on human meaning or judgement or values, not even statistically modelled versions of them. You might then provide better descriptions of the systems from which complex outcomes emerge –‘better’ as in more open to the tools of research and the resources of critical thought. It would be a good thing, I think, for everyone who wants to use AI systems in an informed and ethical way. But to construct today’s really existing AI systems, you need to accord some outcomes, some system states, more value than others. That is how todays large media models are trained, by backpropagation towards a desired state. ‘Intelligence’ is the value associated with this state. And ‘Artificial intelligence’ is the goal of reproducing this value computationally.

Without this analogy and desire, all you have are diverse computational techniques. And it is doubtful that any of today’s techniques would have been developed without the immense investments of data and capital, engineering and future imagining that desire has been able to mobilise. (A recent report on the advances in generative AI since 2012 found that scaling up compute has been significantly more important than any algorithmic advances.) AI fantasies may be ‘for the birds’ but they are still what keep the plane in the air.

When I studied AI in the 1980s, it was science and philosophy of science that interested me. Attempts at developing human-like responses in computer systems, particularly when those systems were designed in dialogue with philosophies of language and mind, seemed to me to promise explanations that traditional philosophy could not. Making things work brings a materialist rigour to speculative thought. That is what Bratton is also interested in. He hopes that the search for viable AI might lead to ‘a fuller and truer range of what thinking can be (and for that matter, what being human can be)’. But that research project has been thrust aside in the rush to monetizable outcomes, as many cognitive scientists besides Bratton have lamented. Making things cannot lead to understanding if you don’t understand what you have made. Again, this is not a defence of traditional cognitive science, but an observation that on its own terms, the performative turn is a dead end.

Black boxes are anathema to science. But in contemporary AI, it’s black boxes all the way. First, models are proprietary (so, secret). Second, approaches that might enhance explainability are expensive, and nobody important (that is, nobody who is paying) really wants them. Third, the training process is so complex, the resulting data structures so multi-dimensional and many-layered, that explainability lags many years behind model development. Explaining a model once it has been trained is not like pulling back a curtain, but like taking echo soundings from an ocean floor: a range of ad hoc and uncertain techniques that might produce a ghostly outline. There is a school of thought that the Bayesian statistics large general models are based on may even be inherently non-explainable.

In a remarkable book on computers, chess and minds, Gary Kasparov said of his defeat by supercomputer DeepBlue that: ‘the machine doesn’t have to solve the game. The machine only has to win’. Winning, in the AI game, is everything. As I’ve mentioned before, generative transformer models are based on statistical methods honed in the insurance industry and in market analysis, where they are used to manage risk and to predict future areas of profit. How much money can ChatGPT make? It seems like the kind of metric it should excel at.

The economist Frederich Hayek saw markets as algorithmic systems, distributing economic goods in the only rational way. Nobody needs to understand or regulate the market: everyone just needs to get on with doing their profit-maximising thing and the market will find its own best state. There is a direct intellectual line from Hayek via libertarian economists such as Peter Bauer and Peter Brimelow all the way to today’s alt-right, taking in many proponents of ‘scientific race’ theory and heritable IQ along the way (these links are to recent articles by Quinn Slobodian, who has this whole connection nailed). You can hear Hayek’s passion for free, unregulated, algorithmic markets in Marc Andreessen’s plea for AI to be ‘set free’.

Like the brain-computer analogy, the computer-market analogy works both ways. It allows behaviour to be modelled mathematically, in terms of maximising outcomes, which is an efficient way of managing complex human and social issues if you are looking at them from on top, or from a long way away. And it allows markets themselves to be seen as the apex of rationality, obscuring their unruly desires, their regular crises, the injustice and violence of capital accumulation, and the planetary death drive of capitalist growth.

In the neoliberal university, learning too can be understood in economistic terms. Maximising outcomes becomes the only rational path for learners to take through the system, and behaviourism the only rational theory for educators to apply. If teachers are encouraged to treat learners as black boxes - probabilistic functions of inputs and outcomes - and if learners see teachers as capricious judges who stand between them and the degree they have paid for, it makes sense for both sides to recruit synthetic media to improve their play. I have a post on ‘writing as passing’ that looks deeper into this dynamic, and tries to think creatively about writing beyond the production line.

Reading Turing differently

What of Turing? Though homosexual ‘behaviour’ was decriminalised in 1967, and Turing himself received a state pardon in 2013, members of the psychiatric profession including Hans Eysenck were arguing for criminalisation and chemical castration well into the 1980s. Behaviourism, and a psychiatry focused on producing ‘normal behaviour’, led to the kind of corrective ‘therapies’ that cost Alan Turing his happiness. IQ testing contributed to decades of racialised and gendered discrimination, classification and control. And probabilistic methods in AI continue to classify ‘kinds’ of person in ways that are inherently biased towards a particular ‘norm’ - sexualised, gendered, racialised - and oriented towards surveillance and discipline.

Had Alan Turing lived through the political upheavals of the 1960s and 70s, and been able to reflect on the impacts of behaviourist psychology on many lives besides his own, I like to think he would have come to question the use of his thought experiment as a performative test for ‘passing’ as a human being. Today his name is more happily associated with a clause in British law that allows people convicted of homosexual ‘behaviour’ to have that conviction removed and to negotiate their identities and desires as they choose.

These messy question of politics, identity and desire are exactly what computational ‘intelligence’ is supposed to lead away from. AI is meant to offer accuracy, objectivity and control. The problem, of course, is that politics is not removed but installed in proprietary systems where it can’t be brought into the light of day, or questioned, or resisted. Power, desire, and the desire for power are parameterised. The judge in the Turing test, the ‘teacher’ of the child-machine, the model engineer and the designer of the ‘reward engine’ still have all the power to determine people’s ‘kinds’ and people’s futures, but none of the accountability that politics might demand of them.

O, there be players that I have seen play, and heard others praise, and that highly, not to speak it profanely, that, neither having the accent of Christians nor the gait of Christian, pagan, nor man, have so strutted and bellowed that I have thought some of nature’s journeymen had made men and not made them well, they imitated humanity so abominably.

Will Shakespeare, Hamlet III(ii)

The original definition of ‘AI’, coined by John McCarthy in Stanford in 1955, was ‘the science and engineering of making intelligent machines’. To McCarthy and his Stanford colleagues, the meaning of ‘intelligent’ was too obvious to spell out any further. Marvin Minsky, writing in 1970, reiterated that: ‘Artificial intelligence is the science of making machines do things that would require intelligence if done by men’. Many definitions of ‘artificial intelligence’ in use today rely on the same assumption that computational ‘intelligence’ simply reflects what ‘intelligent people’ can do. Definitions such as here, here, here, and from today’s Stanford Centre for Human-Centred AI here all follow the same pattern. Intelligent people don’t even have to be men these days, but ‘we’ know who ‘they’ are.

In the guise of starting from something self-evident, the project of ‘artificial intelligence’ in fact serves to define what ‘intelligence’ is and how to value it, and therefore how diverse people should be valued too. Educators have good reason to be wary of the concept of ‘intelligence’ at all, but particularly as a single scale of potential that people have in measurable degrees. It is a statistical artefact that has long been scientifically discredited. It has been used to enforce racial and gender discrimination in education and to justify diverse forms of discriminatory violence, particularly over colonial subjects.

Biologist Stephen Jay Gould described the project as:

‘the abstraction of intelligence as a single entity, its location within the brain, its quantification as one number for each individual, and the use of these numbers to rank people in a single series of worthiness, invariably to find that oppressed and disadvantaged groups—races, classes, or sexes—are innately inferior.’ (Gould 1981: 25. Cited in Stephen Cave (2020) The Problem with Intelligence: Its Value-Laden History and the Future of AI.

The term ‘human’ is problematic for similar reasons. Unless its use is carefully qualified (and even then) ‘human’ all too often takes a particular fraction of humanity - white, anglo-european, male, educated, for example – as its reference point and norm. Most academics have enough sense of the history of these two terms - ‘human’ and ‘intelligence’ - to avoid using them in an unexamined way. Particularly when it comes to student learning, when we recognise there are a diversity of ambitions, identities, experiences, capabilities and cultures in the classroom, all of which can be resources for learning. And yet, since ‘artificial intelligence’ colonised the educational discourse, ‘human intelligence’ has begun to be used as though it is not only a real and self-evident thing, but self-evidently what education is about.

The value of this term to the AI industry is obvious. ‘Human intelligence’ is a palliative to anxieties about the restructuring and precaritsiation of work: don’t worry, there are still some things our models can’t do (yet). And yet the space of work that has not been declared computable today, or tomorrow, or the day after tomorrow is narrow and narrowing, and only the AI industry is qualified to define it. ‘Human’ in relation to ‘artificial’ intelligence turns people into props for data systems (humans in the loop). Props that make system outputs more accurate, safe, ethical, robust and useable, only to be removed once their input has been modelled and datafied. (Or, perhaps, when making AI safe and useable proves too expensive after all.)

This is what the WEF means by ‘working productively alongside AI’.

But it is not clear to me why anyone who cares about education would catch the term ‘human intelligence’ from the cynics who are throwing it our way. Not surprisingly, given the history of both terms, if you pay any attention you can hear how regressive and divisive it is. A small number of ‘human intelligences’ will be free to maximise their potential for innovation and originality, their entrepreneurial decision-making and wise leadership. So rest easy that there will be highly paid jobs for AI engineers and company executives.

However, these people can only max out their human qualities if they are set free from other kinds of work - the boring, repetitive and uncreative. We are supposed to believe that this work is being ‘automated’ for everyone’s benefit, but this is manifestly not so. Research assistants aren’t promoted to other, more interesting roles when ‘AI research assistants’ come online. Rather, the work they do is likely to become more pressured and less valued, or to disappear. There are still drivers (‘human overseers’) behind self-driving cars, annotators (‘data workers’) behind large language models, human personnel swiping left to authorise AI ‘targets’ for bombing, and teachers uploading lesson plans and topic maps for ‘teacherbots’. And it turns out there were 1000 Indian data workers behind Amazon’s ‘fully automated’ stores in the UK. The work does not vanish, it is just routinised, cheapened, denigrated, frequently offshored, and always disappeared from view.

The other ‘human’ who appears in the AI mirror is not running companies or registering patents but doing ‘emotional’ work: that is, work that has always been badly paid or removed from the sphere of paid work altogether. The work of care, service, community building, non-monetisable forms of creativity (craft, leisure, play), mending things and people who are broken. These forms of ‘human intelligence’ are not ‘increasingly prized’ at all. Instead, university managers are calculating how many student support staff can be replaced by chatbots. Academics who invest time and care in students (‘the human touch’) are threatened with redundancy. Schools are relying on volunteer counsellors to cope with the tsunami of mental distress (my local school has 13) while employing highly paid data analysts. In fact, the people who do the most to actually humanise the experience of mass education for students seem to be the most replaceable.  Enjoy the feels, because ‘emotional intelligence’ doesn’t ask for job security.

It’s funny how this happens, but it seems work that is highly rewarded because ‘uniquely human’ is most likely to be done by white, western, well educated men, preferably in STEM disciplines. While work that is undervalued because it is ‘only human’ is most likely to be shouldered by the rest of the world. And this work is constantly being reorganised as data work. Every gesture that can be captured as data is modelled, and whatever is left is rendered as a behaviour, to be governed by the model and its data-driven routines. Between highly paid ‘innovation’ and the non-computable work of the foundation economy - work that literally requires the human touch - AI is the project of datafying everything else.

A recent post on TechTarget defined the ‘important differences’ between artificial and human ‘intelligence’ in ways that make clear everything in the right hand column is already available on the left. ‘Human intelligence’ is apparently being flattered but actually being erased. These definitions are so shallow, cynical and vacuous, I can only read them as deliberate provocations to the education system that is supposed to fall on them gratefully. ‘Mental sensations and concepts of phenomena that are not present’? I can’t see that passing even ChatGPT’s floor-level bullshit detector.

What these self-serving comparisons produce is a double bind for students and intellectual workers. Submit yourself to the pace, the productivity, the datafied routines of algorithmic systems, and at the same time ‘be more human’. Be more human, so that you can add value to the algorithms. Be more human , so more of your behaviour can be modelled and used to undercut your value. Be more human so that when AI fails to meet human needs, the work required to ‘fix’ it is clearly specified and can cheaply be fulfilled.

We can see these demands being interpreted by students as both a need to produce optimised texts – according to a standardised rubric or, perhaps, to satisfy an automated grading system – and to write in ways that are demonstrably ‘human’ (whatever this means). No wonder they are anxious and confused.

I spoke about this double bind for students in a recent podcast for the UCL Writing Centre: Student writing as ‘passing’. (recording soon available from this link). I also explore some of these issues in a post on the ‘unreliable Turing test’. The problems and disruptions posed by ‘AI’ are not only for education to suffer, but the question of what it means to pass as both authentically human and valuable to the data economy is particularly pressing in the education system. It surfaces in all the concerns about assessment and academic integrity. But only to address it there is to fail to recognise the challenge that is being thrown down to universities by big tech, epistemologically and pedagogically, as well as through the more mundane challenges of draining talent, buying political influence, and competing for educational business.

I hope you enjoy these new offerings. All their human imperfections are my own.

It seemed provocative at the time when I suggested that ‘getting better’ was not the inevitable trajectory of generative AI models, but now we hear regularly that businesses are unhappy and salespeople are being told to ‘dial down’ the expectations.

Gary Marcus has more to say about dialing down expectations, not only of future performance for users, but of ROI (return on investment). And Ed Zitron wonders, based on recent interviews with OpenAI’s top dogs, if we have reached peak AI. These two are well known AI-sceptics but they are also well informed.

So much hype, hope and capital has been poured into the AI story, it seems unlikely the bubble will burst. If Sam Bankman-Fried could not kill crypto, it’s hard to know what could kill AI. But I hope that policy leaders in HE, in their foresight planning, at least consider the possibility that AI may not be the whole future of graduate employment. Without some major breakthrough - beyond just scaling up - it’s possible that language models have already found their use cases, and there are really only a few of them: coding faster (though not more accurately or securely); flooding the zone with search-engine-optimised content; and generating college essays of a similarly ‘optimised’ kind.

Using generative AI to write actual research papers, it seems, has ‘unintended side effects that are largely detrimental to academia, knowledge production, and communicating research’. I had a lot more to say about harms to knowledge production in this earlier post.

When I wrote about how learning models are built on the labour of data workers, many of them precariously and exploitatively employed in the global south, one follower on X suggested that it was a good thing these people had work. The brilliant tech journalist Karen Hao has just reported that Remoworks has suspended all operations in Kenya, without notice, leaving many families suddenly without income. The same companies that employ data workers also ruthlessly use their work to build automated annotation models, and are now offering these to clients as cheap alternatives. Any labour that can be offshored can be automated when the costs of automation drop below the costs of cheap, exploited labour: what happens to those workers then?

More hopefully, Techworker Community Africa is supporting African data workers to organise, access training, and get a better deal.

In self-exploitation news: whole heaps of your favourite social media platforms are offering to sell user data to train future AI, and Reddit already has a $multimillion deal in place for its users’ content. It’s well known that the major models were trained on Reddit threads, so here is another content provider (like the major publishers that I reported on in ‘capturing content’) to calculate that it’s more profitable/less risky to get in bed with big tech than to sue them for the bedsheets. These business calculations are where we can watch a belief in ‘AI’ actually building the AI future, as big tech persuades content providers that whatever lawsuits they may lose along the way, in the end they are going to win. They are going to win because they have more money. And they have more money because they have persuaded venture capitalists that they are going to win.

In safety news, Microsoft CoPilot continues to produce potentially harmful responses, such as this one to a prompt about PTSD, and an AI engineer at MS has blown the whistle on its CoPilot Designer for generating violent, sexual and copyright-violating images. I wrote about Gemini’s guardrail problems recently, but other brands are available. Meanwhile the newly established US AI Safety Institute is facing a crisis as staffers protest over the increaasing influence of ‘longtermists’ and ‘effective altruists’ at senior levels in the organisation. These are people with such a zealous commitment to the ‘Future of Humanity’ that they are willing to put up with all manner of harms to people alive today to get there. Harms that less zealous staff members will keep insisting are relevant to the ‘safety’ mission.

Humanity in the abstract is also a distraction from the way AI harms different people in different ways. It’s emerged that even when they are asked questions in other languages, text-based chatbots process requests via English-language constructions, presumably because these areas of their data models are richer and more complex, and their trained weights push all the model’s ‘attention’ that way. The researchers conclude that this ‘could create cultural issues’. As I wrote in ‘who pays for authenticity’, speakers of minority and lower-resourced languages are being further disadvantaged by a ‘global’ technology that purports to ‘know’ the world when what it ‘knows’ is digital text from the minority world, mainly in English. Users of other languages are being offered bad machine translations of what is (even for English speakers) already a sub-optimal experience.

Speaking of the world, training synthetic models has been using significant fresh water reserves in areas prone to drought; and thanks to AI data centres, America seems to be running out of electricity. Environmental groups have warned that AI is likely to accelerate climate warming AND misinformation about it. The latter is indeed what I identified as the main climate cost of AI: failing to invest in more mundane technologies or in changing our political and economic system, while AI fills the horizon of what the future can be.

To end on a personal note: when I started a blog about critical approaches to technology in education, I never imagined that generative AI would fill my own horizon. It has not been entirely fun. A colleague recently described it to me as ‘the constant intellectual labour involved in having to take seriously the noise and free-floating anxiety’, and that labour feels increasingly pointless. Talking ‘AI’ down is still talking about AI, it still adds to the vortex of attention. There are other many more important things in the world to be anxious about (though ‘AI’ seems set to make all of them worse).

AI will probably give paying users a new interface on their work and play that will be fun for a while, and then invisible - part of an ever-more-immersive life online. When ROI falters there will be another story (or a newer, better, ‘smarter’ version of the AI story) to sell hyper-productivity and automation to businesses, and to keep driving capital towards the biggest platforms. I just keep thinking that the idea all this has something to do with knowledge or learning is so obviously detrimental to education, and so obviously stupid and wrong, that education will find a way of talking back. Or - because alternative stories are available - will tell these stories confidently, so I can think about something else.

I read that every university is supporting its staff and students with ‘AI literacy’, which is great news. The Russell Group principles on the use of Generative AI - published last June - even set out a kind of curriculum, including privacy, data violations, bias, inaccuracy, misinformation, plagiarism, ethics and exploitation. That is quite an agenda, so we must hope generative AI is freeing up enough time for everyone to engage with it.

I know some fabulously able and committed people who are working in this space. People who understand that it’s not enough to write out a list of harms and urge your community members to attend to them, but who are running workshops where teachers can encounter these technologies and have thoughtful conversations about their impact. This seems to me the only way. Many years ago, when I researched how academics feel empowered with new technologies, the approach that kept coming up was ‘peer supported discovery’. This takes a commitment to people. Not ‘getting them up to speed’ - that is, accelerating personal productivity to match the pace of the hype cycle - but offering space to slow down, share and reflect. Open education communities can be an important resource for this, as Anna Mills and her colleagues have explored.

So I thought I’d use the take-down of Gemini’s image generation capabilities - the ‘diverse images from white history’ incident - as a kind of case study. To see what questions can unfold from this one event. To dwell on it, against the ‘generate, regenerate and forget’ mode that AI asks us to adopt, especially in relation to its own flaws and face-plants. I’ve also added some critical AI resources to the end of this post.

Gemini: what happened?

Finding a well-informed account could be part of the challenge, but here are a range of write-ups from:

• The Verge

• Time magazine

• The Atlantic

• Gary Marcus (a cognitive scientist who is unconvinced by large language models)

I have not linked to any AI generated images, nor any accounts complaining of ‘woke AI’ or ‘anti-white racism’ because I don’t want to drive those algorithms any harder. But to be fair, I did ask Gemini what happened and got what seems a balanced response:

Inaccurate depictions: When users prompted Gemini to generate images of people for specific historical figures, like "America's Founding Fathers," the results excluded white Americans altogether, depicting figures from other ethnicities. This sparked criticism for historical inaccuracy.

Overly cautious behavior: Additionally, Imagen 2 became overly cautious and refused to generate images for even innocuous prompts, misinterpreting them as potentially sensitive. This led to frustration among users who found the limitations unreasonable.

So: some people were cross because the fabrication engine produced fabrications, and others were cross because it refused to. Gemini also suggested that:

Educational efforts can help users understand how their prompts and inputs can influence the AI's output. This can encourage responsible use and help mitigate unintended consequences.

‘Unintended consequences’ are, it seems, the fault of uneducated users. Gemini isn’t the first to blame users for an unscheduled model take-down:

More on Galactica later. So here is a simple question to encourage responsible use:

What have we learned about ‘guardrails’?

A ‘guardrail’ sounds like something specific - an algorithm perhaps - bolted onto a model to keep its outputs safe. But what is it? With the intense secrecy involved in their training and refinement, we have only a general idea of how engineers can influence the behaviour of models. One would be to extensively retrain them: for example a version of Bert called Zari was retrained on gender-related sentence pairs and showed less gender bias in its responses. But this was a small research project: retraining and re-parameterising foundational models to eliminate bias would be, depending on who you ask, impossibly expensive, or just impossible. From the perspective of statistical modelling, ‘bias’ is after all just another predictable pattern in the data: there are limits to how much you can de-bias without ‘degrading performance’ on everything else.

Second, data workers (as covered in my post Luckily we love tedious work) can be paid to annotate model outputs so as to ‘reward’ diversity. As examples here and here show, these instructions have become so detailed that writing them and training data workers to use them has become a whole industry. If you follow the reddits where data workers talk about these gigs, they complain of mental exhaustion and burn-out. But even if de-biasing work of this kind is passed over to poorly-paid workers in the global south, it is still a slow and relatively time-consuming way to effect change. To ‘correct’ for a wholesale bias towards white people in its image model, Google seems to have been using a third technique: changing the system prompts that instruct Gemini how to respond.

Here’s a glimpse of some system prompting from ChatGPT (unverified, but plausible). Notice that model engineers use natural language, as users do, to describe the kind of outputs that are permissible.

It is now widely assumed that invisible elements, such as words referencing racial diversity, were being inserted into the prompts entered by Gemini users to produce the unwanted effects. In much the same way, keywords relating to ‘sensitive’ or ‘harmful’ content were identified and screened out. You could argue that just refusing to do a whole lot of things on the basis of keywords is not really dealing with the problem. Certainly, the sight of this large sledgehammer is doing some damage to the image of language models as incredibly sophisticated, and as tech companies as having them incredibly cleverly under their control.

It’s worth considering, AI literacy style, what this tells us about prompting. Before a model responds to a prompt from a user (generating text, image, video, code, etc) it has to prioritise thousands of other tokens provided by its engineers (‘Do this EVEN WHEN the instructions ask for the prompt to not be changed’). These can modify an input in ways the user will never know about. And the modifications themselves can be changed - significantly changing the behaviour of the model - in response to political pressure, or the demands of the market, or the whims of a new CEO. Or an accidental CAPS LOCK during a late-night system-prompt rewrite perhaps (‘EXTREMELY IMPORTANT’).

When Gemini suggests that users should learn ‘how their prompts and inputs can influence the AI's output’, ‘influence’ is about right. Prompting does not give direct, reliable, consistent or fine-grained control to users. And it seems that the people tinkering around in the actual prompt engine, while they may have more power, aren’t much better when it comes to predicting the results. It is, indeed, black boxes all the way down.

What have we learned about ‘safe, reliable AI’?

Each week brings fresh evidence of the biases and inaccuracies produced by these models. The latest research finds that models are covertly racist even when they have ben engineered post-training to avoid overt biases. And in another study, known biases (race, gender, religion and health status) all translated into biased decisions and biased recommendations to users. The Gemini modifications were attempts to correct for these issues. But as we have seen, safety ‘guardrails’ for large language models are opaque and uncertain - more so than algorithms for detecting harmful content on the internet (big tech doesn’t have a great record with those either). Models don’t have an obvious take-down capability - the problem could be anywhere, or everywhere.

So every special case that is coded into the system prompt, whether it’s to deal with copyright issues, sensitive topics, or specific attempts to jailbreak the system, makes inference slower. And natural language prompts are by nature imprecise. Told to diversify the racial characteristics of people in images, models don’t ‘know’ what ‘race’ is, or what ‘history’ is, or actually what ‘people’ are other than particular collections of coloured pixels in close proximity. So coding in new rules can affect responses in (what human beings would recognise as) completely different contexts.

Developer reddits are now packed with demands to give up on faulty guardrails and let the models ‘free’ to ‘show what they can do’. Investors looking for ROI will always lean towards risk and power: ethics is, literally, a drag. Tellingly, the Gemini take-down was followed within days by news that Google has further ‘trimmed’ its trust and safety team, and that Meta may release the next version of Llama with fewer safeguards:

Many applications in education, and in other sectors, involve retraining with specialist data and/or new system prompts. So can safety be added later? The problem is that these models all sit on top of the foundation models, and depend on their proprietary development tools. And the fine-tuning process itself can compromise existing guardrails or unpredictably degrade them. So intermediate models designed to be low risk may actually perform worse on safety and bias:

General models, we are told, are flawed because people are flawed. The highest achievements of human culture and the bottom scrapings of the internet have all been fed in together (this was, of course, a choice). But Meta’s Galactica was trained only on scientific papers, and was shut down after a few days for spewing racist anti-scientific and conspiracist nonsense. Models are not only their data inputs but also the probabilistic nature of their data structures, and the hyperparameters used in their training. (HuggingFace provide a list of just some parameters that can be tweaked to steer models during their development. There seems to be a lot of trial and error involved, as well as some very difficult maths.)

AI literacy should not mean dealing with the difficult maths, but it should involve an awareness of how ad hoc and fragile current ‘guardrails’ are. It should involve understanding the limits of prompting, as well as the limits of ‘retraining’ models on ‘safe’, ‘reliable’ data. Educated users might well find themselves asking not for more training but for a data and information environment that does not involve so many uncertainties and risks of harm.

What have we learned about authenticity?

150 billion AI generated images entered the public space in the first year of text-to-image generation. Ethnically diverse Nazi soldiers and Founding Fathers may be obviously faked but they are no more or less fake than the millions of historically more ‘appropriate’ images that have been generated in the same way. They too are fabrications, though they may be harder to detect. The credibility problem is not unique to history: paleontologists, biologists and other scientists have sounded the alarm about inaccurate but plausible images entering their research fields and public communication. Nature has banned AI-generated images and illustrations from its journals - unless image-generating AI is the topic.

Rob Horning, in his ‘Internal Exile’ stack, takes a hard line:

If you are willing to construe generative models as knowledge producers, as truth-tellers, despite the obvious leaps of illogic it requires to think that, it must be because you really want to believe in the power to indoctrinate. You have to be upset when they “fail” so you can pretend that they can succeed.

Rob’s certainty that he knows the difference between ‘truth’ and ‘indoctrination’ does bring me out in a bit of a sweat. Are generative media specially untruthful or inauthentic? I can hear the sighs of my good friends in science and technology studies. So I will attempt a bit more nuance on this question, but feel free to jump forward to the politics part if that will give you more joy.

All representations, including digital images, are mediated in some way or another. Framing, recording, digitisation, post-production… the real world is not pixelated, and any arrangement of pixels we read as an image has been fabricated somehow. On this blog post about the history of photographic fakery, I found a fabulous image of the Paris Commune (1870s) by society portrait-maker, Eugène Appert. It was constructed from a range of different photographs and painterly techniques to suggest scenes of violence and disorder.

This was not particularly unusual. The new techniques of photography arrived into an existing tradition of ‘illustrating’ news stories (and partisan opinions) to bring them to life. Before photography transformed ideas about representation, images were not expected to provide unmediated access to ‘what really happened’. Blogger Josh Habgood-Coote explains:

In 1884 the engraver Stephen Horgan wrote in Photographic News… “All photographs are altered to a greater or lesser degree before presentation in the newspaper.” In 1898, an editor of a photography magazine summed things up more pithily: “everybody fakes”.

As these quotes show, however, photographers were beginning to see that the new medium afforded a different relationship between the viewer and the scene: one in which ‘alteration’ might suggest not ‘illustration’ but rather ‘faking’. And so:

photographers engaged in an internal debate about the proper norms of photographic practice (drawing on previous debates about faking in written journalism), leading to both the establishment of social norms around manipulation, and the development of a professional identity which would later be called photojournalism.

Similarly, methods and disciplines and academic identities emerge from shared norms around how knowledge can legitimately be constructed. Historical images are not ‘authentic’, after all, they are ‘authenticated’, using a variety of techniques, and calling on a range of expertise. These may be fallible, but they can be checked by other experts against agreed norms and procedures. New technologies can and should be accommodated, in dialogue with existing values. All of which is to say that technologies and social practices are constantly producing each other.

This whole business can be called elitist. Photojournalists, historians, biologists, paleontologists and the rest of the academic circus - aren’t they just gatekeeping access to their privileged professions? What is ‘evidence’, after all, but ‘their truth’ demanding to be treated as something special? Silicon valley libertarians love to use this argument. They need technology to replace technique if they are to shift product, but it sounds better to promote this as a great leveller than as a project to automate skilled labour. Anyone can do what experts do. Down with experts: we have an AI for that.

This breathless piece by the CEO of Cognizant for the World Economic Forum, for example, notes that ‘generative AI benefits the less skilled and less productive’, and sees AI replacing education as the ‘great equalizer’.

Just to spell it out, ‘less skilled and productive’ people are being offered the chance to replace more skilled and productive (and presumably more educated) people through the magic of AI. They are not being offered the chance to learn skills, or to earn what skilled people earn, which might be more actually egalitarian. Education is explicitly being replaced by ‘better tech’ as a democratic project.

The politics of authentication

There is, of course, always politics in who gets to authenticate, whose expertise is valued, what cultural artefacts are deemed real enough or valuable enough to care about. Education is as political a project as ‘AI’. I know people working on ultra-lightweight, ‘edge’ applications of transformer architectures who believe there is democratic potential there - in the open development of ‘models for all’. But, with the greatest respect for their expertise and political optimism, that is not the project of generative AI as it is being pursued by the corporations with the power to make their meanings stick.

‘The people have had enough of experts’, Michael Gove told the ‘Vote Leave’ campaign in 2016, before stepping into the Cabinet Office to replace dozens of civil service experts with AI contractors. His friend Dominic Cummings, another AI fellow traveller, paid Faculty.AI to work on ‘Vote Leave’ before bringing the company into Downing Street to work on the Covid response. Faculty AI are now part of a 'crack squad’ inside Westminster working to reduce the number of experts in Government still further. (As I recorded elsewhere, Faculty were recently paid nearly half a million pounds for a two day event to promote the use of AI in schools - and I have a feeling we will hear more about this project.)

Universities are full of experts, and the expertise they are most replete with is the expertise to test knowledge claims. Socially-constructed, fallible, contingent, value-laden - scholarly methods are all of these. Exclusive, elitist, obscure, rivalrous, gatekeeping - scholarly communities can be all of these too. There isn’t an easy way out of the expertise-elitism bind, because expertise takes time and practice and the support of knowledgeable other people, and these are relatively scarce resources. Privilege and power attend on all valued knowledges. All we have to work against this is the political project of improving access to (and the diversity of) higher education. That’s it.

Still, universities are manifestly sites of struggle over these things - over who should have access, what kinds of knowledge matter, how research and teaching should be carried out. AI models, not so much. The historians and photojournalists have not been invited to discuss the epistemic outfall of generated images, but told to suck it up. If we wanted to decolonise this particular curriculum, where would we protest? The politics of generativity is obscured, most obviously in the models themselves with their ‘objective’ statistical methods standing in for all other forms of cultural production. The anti-politics of replacing a debate over meanings with the private ownership of representation - that is also obscured.

The internet was a site of struggle over values. I’m old enough to remember the promises of networked learning: Illych’s convivial communities, Manuel Castell’s connected society, the technofeminist manifesto. In that hoped-for future of frictionless access and horizontal organising, everyone would find their own expert community, creating and sharing their own valued resources. What Bourdieu calls the ‘democratic disposition’ would flourish everywhere. In the non-ideal present we inhabit, in which all the wrong politics seem to have prevailed, AI is not offering to replace the expertise of universities and cultural institutions with some more democratic form of knowledge, as the internet was once. It is offering to replace it with proprietary data; the power of representation not as a visible and contestable social power but as invisible computational capital.

Who pays for authentication?

As synthetic media ‘flood the zone with shit’, every organisation with cultural assets is left with a problem: how to persuade people of their value. This is true for publishers, wondering how to identify human-generated creative works. It is true for journalists trying to project credibility among all the synthetic videos of the same events - and of vividly ‘illustrative’ memes, entirely unanchored in the world of events. It is true for the curators of historical documents, medical images, and open educational content. It is true for the speakers of minority languages, who are disproportionately affected by the flood of AI generated text. In fact, this research into minority or ‘lower resource’ languages shows just how unequal it is all going to be:

machine generated content not only dominates the translations in lower resource languages; it also constitutes a large fraction of the total web content in those languages. We also find evidence of a selection bias in the type of content which is translated into many languages, consistent with low quality English content being translated en masse into many lower resource languages, via MT [machine translation].

So the fewer resources you have to project your cultural assets, the more likely you are to be engaging with poor copies of them already.

Of course there are people working on technical solutions. Nightshade, in development at the University of Chicago, messes with image pixels to make model training difficult. Truepic adds metadata to images including the history of their production - though it can be overwritten. An earlier image ‘toxin’, Fawkes, seems to have been actively engineered against by DALL-E and Midjourney. ‘Watermarking’ solutions are available but as this article from Hugging Face admits, far from effective. Every technical fix has a technical counter-fix. What is clear is that the same AI industry will profit from both - content generation and detection, watermarking and watermark-removal - while the owners of cultural assets will bear the costs. I hear that cultural organisations are already beset with tech salespeople offering their support in these difficult times.

All the forms of expertise that produce (and lend credibility and trust to) knowledge are going to cost more now, relative to pressing the auto-generate button. In schools and universities we understand these costs in terms of time: we wonder whether students will continue to invest time in difficult ways of thinking and working when alternatives are available. But if we expand that frame, we can see that organisations with a stake in knowledge are facing the same question as real operating costs. Who is going to pay for knowledge, trust, and all the associated forms of expertise? Of course, well-resourced organisations will to be able to bear the costs, and will even project them as a sign of their status and authority. But it’s hard to see how these costs are not going to result in new barriers to access, new inequalities, new forms of gatekeeping, and new threats to minority disciplines and cultures.

What have we learned about the political stakes?

We can imagine that there is now intense conflict within tech companies about how far safety concerns - which are political, messy and controversial - should be allowed to compromise ‘performance’ - which is a simple metric tied to market share. Let’s not forget that the whole of generative AI is essentially a beta technology, pushed onto the public in the hope that impressive early effects and extraordinary levels of hype would supercharge the adoption curve before too many questions were asked. The people who drew attention to the ethical risks were fired before the generative AI rocket got to the launch pad. Many of the remaining ‘ethics’ teams were ‘slimmed down’ as adoption took off, and the few ethics people left on board are, I imagine, too busy coping with in-flight repairs to argue their corner.

These questions have now been energised by - and publicly entangled with - the inevitable culture war. Unsurprisingly, Elon Musk - who launched Grok as the ‘AI without guardrails’, promising it would be ‘spicy’, ‘rebellious’ and ‘anti-woke’ - was first in line to bash Google, using the incident to denounce the ‘woke mind virus [that is] killing Western civilisation’. Well, we can’t know what ratio of ‘wokery’ to white supremacy has been engineered into different model outputs recently (though readers, I’m sure someone is building an AI for that). Perhaps different models will be available to suit your political or conspiratorial vibe. But we know do how similar stories have played out. In the economy of digital platforms, safety, moderation, ‘fact-checking’ and responsible governance are costs that big corporations do not want to bear (see ‘who will pay?’ above). What they want is reach and scale. Reach and scale are maximised by polarisation, hot takes and shocking memes.

If that is the latent politics of social platforms, they line up with the manifest politics of an increasingly reactionary (but always libertarian) tech industry. The faux-political demand for models to be set free - the concern for silicon ‘rights’ that I wrote about in ‘AI rights and human harms - is actually a demand for tech capital to be set free from politics altogether, from the discourse of rights and responsibilities, from labour law, copyright, and concerns for social justice. And from processes of authentication, which as discussed are always socio-political, but can now be packaged up and labelled wholesale as elite wokery. Meanwhile the flood of generative content, set ‘free’ into spaces of public discourse, works against the possibility of online trust, shared meanings, or coherent political projects. I think the personal views of tech leaders probably matter less than the politics that will follow these network effects.

What started out as a hot take has become been a long meditation on the politics of representation, and the risks to representative politics. If you’re a new follower, I’m afraid is an imperfect and unpredictable ride. But you’re very welcome along.

Please let me know if you find these thoughts useful for your own critical AI projects - and check out some of these resources that I am inspired by.

Resources for critical digital literacy:

The Association of College and Research Libraries’ ROBOT tool (this version from the University of Newcastle) is a great starting resource for the AI literacy classroom.

MIT’s series on AI colonialism covers many of the issues that animate Imperfect Offerings, from the brilliant Edel Rodriguez. It covers AI impacts in South Africa, Venezuela, Indonesia, and some hopeful developments from an indigenous AI project in New Zealand.

NotMy.AI is a feminist collective that has been critiquing AI from a gender perspective for some time. What it lacks in contemporary ‘takes’ it makes up for in sound foundations and a toolkit of ideas for resisting gender oppression through AI.

Kathryn Conrad’s Bill of Rights for AI in Education, issued as the generative ‘wave’ was taking off, still provides educators and students with some key questions they should ask about AI safety and reliability in their setting.

The AI Hype Wall of Shame is an occasional series of debunking articles that treat the most inflated claims from the AI hype industry with a blast of rational cold water

Although it is a call for papers/practices - and I hope they are inundated - these ‘bad ideas about AI and writing’ (googledoc) could fill several AI literacy workshops.

Critical approaches could start with the people taking action at the sharp end: there are case studies from the Distributed AI Research institute (see the Real Harms of AI Systems) and projects from RestofWorld (check out their regular AI blog with news on data workers and more). TechWorkersAfrica is organising for better training and wages and less oppressive work practices, and while it is not an education site it gives a good idea of life at the wrong end of the ‘data engine’.

Rolling Stone’s feature on the women who have spoken out against AI harms is a good starting point for understanding the issues from a Silicon Valley perspective, while the UN perspective on AI (through the lens of human rights) takes a more global perspective. Current generative models are a very long way from meeting these ideals.

I’ve mentioned before that Lydia Arnold’s assessment top trumps have been reimagined for an ‘AI enabled world’ and are available to download from Jisc.

The policy statement from the University of Edinburgh seems particularly clear and supportive. There is a detailed and equally well considered guide for students from the Library at UCL.

So, in my post on AI rights and human harms I said that general models (such as ChatGPT) may not keep getting better and better, despite all the claims of ‘exponential’ improvement and ‘artificial general intelligence’ being only a few upgrades away. I based this thought partly on reading experts in cognitive science, like Iris van Rooij and her colleagues, who find the idea of an ‘artificial general intelligence’ ‘intrinsically computationally intractable’ and conclude that currently existing AI systems are ‘at best decoys’. I based it partly on reading experts in general modelling (see my post on Sora). But mainly I based it on the business behaviour of our silicon chiefs, who are clearly more interested in pimping chatbot interfaces and distracting us with new products than improving the underlying models. Which they would do if it was easy.

As it turns out, fifteen months on from ChatGPT, Gemini and Claude are a bit better than GPT4 for some things. GPT4 actually seems to be getting worse. Just in the last week, Gemini had to send suspend its text-to-image generation capabilities and go back to the drawing board with its guardrails, and ChatGPT underwent a complete meltdown into gibberish. Both events show that the behaviour of models can be transformed by the tweak of a parameter over at Google/OpenAI HQ. Let’s hope the people in charge of all this continue to be regular, well-adjusted, public-spirited citizens. And both events show something else: nobody actually knows how to deal with the bias, the nonsense, and the hate. Guardrails are a guessing game. It’s black boxes all the way down.

Meanwhile the long-anticipated GPT5 is further delayed, and Sam Altman is rowing back hard on expectations, suggesting that he might have come to regret the promise of:

‘a recursive loop of innovation, as these smart machines themselves help us make smarter machines’

Instead, he admitted in November that ‘We have to figure out some very hard science questions to [get to GPT5], we have to go build more computers’. To achieve this, he is looking for $5-7 trillion to invest in chip production, in the hope that throwing more data and compute at the hard science will break it. To put this in perspective, the entire global semiconductor industry is worth about $500 billion a year. It cost the US $5 trillion in today’s money to fight world war two. So Altman is looking to fry the planet, deplete it of rare earths, exploit tens of thousands of data workers and aggregate more value in a single company than the entire militarised US state of the mid 20th century, all in pursuit of an upgrade he can’t be sure is there.

In 2021 he promised businesses:

phenomenal wealth. The price of many kinds of labor will fall toward zero once sufficiently powerful AI “joins the workforce.”

What businesses have got is mainly a new feature in the Microsoft suite that they are not sure is worth the price tag. In coding jobs, where a version of CoPilot has been used for much longer, the evidence is that the quality of code has degraded. Challenged by business leaders to explain their strategy for improving models - models that have now been built into major infrastructures and workflows - the CEOs of Anthropic and Google’s Deepmind admitted last week that ‘hallucinations’ are intractable and ‘we’re not in a situation where you can just trust the model output’. Other challenges, such as the huge costs of training and running them, hoping the copyright infringements don’t come back to bite you, and ensuring sensitive data is not leaked: these ‘don’t yet have clear solutions’ either.

Phenomenal wealth has been generated, for sure. Billionnaire investors made 96% of their gains last year by betting on AI stocks. At the same time, the tech industry they are investing in has decided to slash its workforce, because AI. But no-one apart from the billionnaires seems to be feeling phenomenal. Not yet anyway.

Saving the planet, one cute animal video at a time

In other not-getting-better news, the climate costs of generative AI are becoming a concern. Although the companies that train and run them are intensely secretive on this point, Kate Crawford has gathered some of what we know in a review for Nature, and calls for legislation to stop power and water conservation becoming casualties of AI competition. This seems to have written before Sam went public on his plans to power up 5 trillion dollars worth of chips.

Energy expert Alex de Vries, speaking to the Verge, has calculated that by 2027 AI will be using at least as much energy as the Netherlands. This is less than one percent of global energy use - the Dutch are a frugal people - but for what? Image and video synthesis is far more costly than text, and text inference is far more costly (4-5 times) than a standard search. So those cute videos are turning a lot of fresh water into steam. It does seem to me that as generative models are integrated into every interface - from working with business software to chatting with avatars in immersive worlds - it will become harder and harder to know what proportion of computer power is being used for synthesis. Or indeed where the demand might stop. Just the prospect of training future models has fuelled a massive data centre boom and made Nvidia the fourth largest corporation on the planet.

As de Vries points out, the problem is not really the current energy use (so far as this can be known), but the economy of large language models, that rewards the most computation- and capital-intensive practices with market capture. This is:

a really deadly dynamic for efficiency. Because it creates a natural incentive for people to just keep adding more computational resources, and as soon as models or hardware becomes more efficient, people will make those models even bigger than before.

In fairness, data-based approaches are vital in climate science, which is genuinely hard, and genuinely important. I read the recent UN report on AI and climate with an open mind. Allowing that ‘AI’ is now whatever data and algorithmic technique you want to throw magic dust over, there are valuable applications here. But all the cases explored by the UN use modelling in the context of other specialist workflows and systems. Not one requires a generative language model.

Wired magazine recently interviewed DeepMind’s climate lead, Sims Witherspoon, who has more claim than most to be addressing these problems. Her takeaways on how AI can mitigate climate crisis are: 1. better forecasting; 2. optimise existing systems; and 3. nuclear fusion (something Sam Altman is also investing in heavily). So, yes, better intelligence is useful, but does not mobilise the necessary political, economic and social change to make an actual difference. And yes, optimising the systems we have today might flatten the upward curve a bit, but hi-tech fixes of this kind can delay and distract from the systemic change that is really needed: disinvestment from fossil fuels and a wholesale shift to practical green technologies. For the most part these technologies exist already. Nuclear fusion, not so much.

Sadly, 2023 saw a steep decline in investment in viable low-carbon tech, while venture capital was piling into AI. The big costs of AI to climate change will, I suspect, not be its energy consumption but the lost opportunity to invest in viable solutions while there is still time. So long as ‘smart machines’ to solve ‘hard science’ keep sucking up the dollars, high finance and high tech can keep sucking up the profits. And if their bet on as-yet-unviable, unscalable future technologies like fusion fails to pay off, the billions they have amassed can at least furnish their bunkers. For the rest of us, the Matrix will now be available in high resolution.

Neither fantasy nor fatalism

‘AI’ is not a technology, it is a story about the future. A story with more power than most, but a story all the same. Powerful computational techniques do exist, new ones will continue to be developed, and some of them will be used to help manage (model, predict, optimise) the existential threat of climate disaster. But there are other, more pressing stories about what needs to be done.

The knowledge and data economy is built on the foundational economy. It depends on the productive ingenuity of human minds in human bodies, that need food, water, shelter and care. It demands a material abundance of rare minerals, energy generation and network infrastructure. So-called ‘AI’ is continuous with other productions of human culture, and with the natural world, however its Titans may long to break free. And since it is contributing so impressively to war, wealth inequality and democratic breakdown, I don’t think a few contributions on the side of climate science are really enough to level the balance.

Of course, higher education tells different stories about the future, and I don’t find huge enthusiasm for the one with the chatbot takeover. But I do detect a lot of fatalism, particularly about the futures of work, that I think should be challenged.

In the first of our new podcast series on generative AI, my colleague Mark Carrigan and I discuss ‘AI realism’ as an alternative response to either fantasy and fatalism. These models are now an integral part of educational infrastructure, formally and informally. That is simply the reality.

Hours of precious academic time has been spent reviewing guidelines and revising assessment regimes. These create new realities too. Much as they did when the internet came along, and then with social media, academics and professionals have worked with students to develop positive practices, taking into account their wellbeing as well as their intellectual development. This matters far more than any writing of mine. I don’t think any of it has been easy, and generative AI is probably going to make it harder, both in its own right and as it hastens the platformisation and enshittification of the rest of the online world.

Realistically, I hope the resources and the risk assessments are there to help everyone cope with this new present. Realistically, I hope I have pricked the AI fantasy a little, so alternative stories have more space. I really look forward to conversations about alternative futures we and our students can hope for.

A bit of personal news: the substack will be going quiet for a while as I focus on urgent writing deadlines. Look out for the podcast series: Generative Conversations with me and Mark Carrigan, and some fabulous guests. You’ll be first to know when I get back online here, perhaps with some new issues. I don’t want to become a one-trick generative pony.

Our actions and creations do have power over us. This is simply true … The danger comes when fetishism gives way to theology, the absolute assurance that the gods are real. Graeber, 2005: 431; cited in Thomas et al. 2018

This essay/post is about AI fantasies and why they matter in education.

When European anthropologists first met the power objects of people from different cultures, they called those objects fetishes. I wouldn’t use the word if it only had that original, colonial meaning. But social theorists, from Karl Marx onwards, have turned the term against colonising societies, with their supposedly more rational beliefs. Because things, ways of producing and using things, really do shape our lives. Marxists are interested in how workers are compelled by the factory system, or by post-industrial technologies in networked systems. Other social theorists are interested in how consumers relate to consumer products, or users to their digital devices.

As David Graeber says, we really are entangled in powerful relationships with things, but we can be fooled if we locate power in the things themselves, separate from social relations and meanings. If we allow things to become gods or monsters, masters or servants, with agency of their own.

Agency everywhere

However we understand the hype around them, it seems we can hardly help ourselves relating to synthetic models as though they have agency. Anna Mills recently started a great thread on alternatives to ‘hallucination’:

You can find the ideas she collected summarised here. But while we can discuss it sensibly, the feeling of ‘conscious experience or intent’ is hard to shake off.

Of course, terms like ‘learning’, ‘collaboration’ and ‘hallucination’ are being fed into the popular discourse deliberately. In the sense that ‘artificial intelligence’ is a project to designate certain computational systems as ‘intelligent’, these terms are not just packaging but essential moving parts. The first person (‘I’, ‘me’) address of natural language interfaces - that many people are now using to access other apps and services - is also a deliberate piece of design.

To use a term from social theory, when a chatbot uses the first person ‘I’, users are being ‘interpellated’, that is ‘called into place’ or ‘called to respond’ as if to another person. I got that from Althusser, who was writing about ideology, but here’s Airenti (2015) from a mainstream cognitive science perspective:

humans are attuned to turn-taking and interactive exchanges with each other, and they apply this approach to artificial entities as well, treating them as interlocutors while being fully aware that they are not human. However, once an artificial entity is in the position of an interlocutor, humans may in fact start attributing goal-directed behavior, mental states, and beliefs to it.

Social or cognitive science: both suggest that, when we are addressed as a ‘you’ by an ‘I’, we respond as an ‘I’ to a ‘you’. As it turns out, some users respond in rude, mean and oppressive ways to chatbots, and I don’t think this is a trivial point (more on this later). But still, we take our turn, and in turn taking we assign the other party some qualities of being a person: an ‘I’ like me, a ‘you’ like other people.

The first chatbot, Eliza, was a basic pattern-matching program that its developer Joseph Weizenbaum, was ‘shocked and surprised’ to find being treated as a partner in conversation, even by the co-workers who had helped design it.

Inside knowledge is no defence: people may or may not remember that the whole generative AI hypefest kicked off when Google engineer Blake Lemoine claimed that the LAMDA version he was helping to train had become ‘sentient’.

The speed and flexibility of transformer models have certainly revolutionised chatbots, making conversation more sustained and fluent, able to range over a far larger ‘context window’, and to deal with a far wider range of prompts. But the big tech companies have not stopped at fluency. Last September, Meta launched 28 ‘conversational agents’ with the faces, back-stories, and alleged ‘personalities’ of celebrities and other characters. The huge cost of human writers scripting these characters, not to mention the licensing of celebrity personas, show the company’s priorities. Rather than improving the underlying models, they are throwing their dollars at the user interface, even though this makes the model more resource hungry and inefficient to run. A September 2023 report from Public Citizen on the dangers of ‘human-like AI’ found that it is cheaper to design without anthropomorphic features, but:

Anthropomorphic design can increase the likelihood that users will start using a technology, overestimate the technology’s abilities, continue to use the technology, and be persuaded by the technology to make purchases or otherwise comply with the technology’s requests.

Yea, big tech gonna tech. Still, these features are appealing to a strong underlying belief in other minds, and I feel this says something about human thinking that challenges the idea of ‘intelligence’ put forward by the project of‘AI’.

‘Self’ing others, othering ourselves

This is a section about psychology – skip ahead if that’s not your thing.

Claims about human beings in general have a terrible history of being used by particular (white, Anglo-European. male) people to justify horrors against particular others. I’m aware of colonial history when I use the words ‘human thinking’ and ‘the human mind’. And I believe that everything we share as a biological species should be understood through specific cultural expressions and meanings, as well as particular embodied realities. But I also worry when critical social scientists abandon human minds to the positivists and the statisticians.

When psychology becomes reduced to, on the one hand, ‘brain sciences’ with their MRI scans and ‘neural’ models, and on the other hand statistical measurement and management of human behaviour, I think we are missing out on essential resources for teaching and learning. Resources that might provide some resilience against the definitions and techniques of the AI industry too. AI recruits neuroscience for its spurious claims to build artificial ‘minds’, and it depends on statistical management of human behaviour to keep people hooked into particular relationships with its data systems. It uses both to make claims about ‘learning’ that have real effects on how its technologies are adopted in education, as well as pushing aside more progressive ideas about how people learn.

The resources of psychology are broader than this. From the perspective of cultural, social or developmental psychology, human beings can be seen as social animals. Animals that have evolved to live in complex groups, with shared goals and needs, and with bodies that can survive only in relation to other natural beings, and other bodies of a similar kind. Needing to solve problems collectively is where language and culture enter the human story – always particular languages and particular cultures, but with a universal need to mediate practical, material activities.

I’ve recently taken an interest in research on mirroring (yes, this is ‘brain science’, but bear with me). There is much evidence that:

other-related information mapped onto self-related brain structures can modulate the ways in which humans and other animals respond to others.

What this means, I think, is that our brains respond to seeing other people doing things in much the same way as when we are doing them ourselves. That does seem to me quite contrary to what individualist Western psychology would expect to find.

Brain science and brain imaging are blunt instruments, so I don’t want to credit them with more insight or precision than they deserve. What used to be called ‘mirror neurons’, for example, are now known to be extensive systems:

found in a variety of brain areas and animal species, encoding not only observed actions, but also emotions, spatial locations, decisions or choices, rewards, the direction of attention, and beliefs.

So the capacity to experience ourselves as others, and others as ourselves, is fundamental to social animals. Empathy is written through our mental repertoire. (Western) philosophers and Silicon Valley red-pillers may ponder how we know other people are real, but we can only experience other people as intentional, emotional, agentic beings, just as we experience ourselves. Our mental life doesn’t seem to begin with a coherent ‘I’, let alone in some logical representation of the world, but in shared agency.

All this is obviously relevant to learning. Lev Vygotsky puts it like this:

Every function in the child’s cultural development appears twice: first, on the social level, and later, on the individual level; first, between people (interpsychological) and then inside the child (intrapsychological). (Vygotsky, Thinking and Language 1978: 57)

Mental activity, in Vygotsky’s terms, is social activity. His colleague, the linguist Mikhail Bakhtin, described all language as dialogue: every utterance is ‘pregnant with responses’, even when a speaker is alone. Alongside other Russian psychologists of the early C20th (Leontyev and Volosinov for example) Vygotsky and Bakhtin are actually much closer to certain indigenous ways of thinking about minds than to Anglo-European psychology. This is something that has been noted by (among others), Mary E. Romero, Luis Moll, and recently by a number of Maori scholars. In these traditions, mind is inherently dialogic, inherently social, inherently empathic to the world. And teaching-learning are two sides of the same coin.

When they are first negotiating these relationships between other and self, public and private worlds, we find children often in dialogue with themselves. We find them lending personalities and voices to dolls and spoons, imagined friends and contemporary artefacts like mobile phones. Again, indigenous ways of thinking celebrate children’s natural empathy, their recognition of ‘other kinds’. In Braiding Sweetgrass, Robin Wall Kimmerer writes: that ‘paying attention is a form of reciprocity’. Cognitive science tends to treat this ‘othering’ as a kink, a quirk, the kind of ‘cognitive bias’ that makes human beings fallible and rather stupid. But it is essential to the development of minds.

Turing’s troublesome test

The othering or empathic quirk is also essential to the Turing test, though it is not often remarked on. The test is a thought experiment that remains at the heart of the ‘artificial intelligence’ project. In this test, a person who is designated the ‘judge’ must ask: does this behaviour, do these system outputs, suggest a mind at work? If the answer is ‘yes’, the system can be considered intelligent. But the only way of making this assessment is for the ‘judge’ to consult with the experience of having a mind, and of interacting with other minds, and to ask: ‘would my experience look like this, from the outside?’

Turing himself admitted that:

The extent to which we regard something as behaving in an intelligent manner is determined as much by our own state of mind and training as by the properties of the object under consideration. (Turing (1948) p. 431, cited in Proudfoot 2022)

Turing was undoubtedly a behaviourist in his view of other minds – the man and woman, or man and machine, that were the subjects of his test. But the test also requires a third person, someone who Turing, as a scientist himself, perhaps identified with (‘our own state of mind’). Someone not only introspective but empathic too. And having minds of this kind, we don’t need much evidence to decide that another mind is there.

We can use this quirk of empathy to play with, and to learn with, and to develop deep relationships with: but it can also be used to play with us. Things, as Graeber noted, have real power over us through the social relations we are entangled with. And as writers on fetishism have observed, from Marx to Agamben, and from Freud to Baudrillard, a belief in the power of things-in-themselves can become a compulsion, a place where power can be exercised.

Gods and monsters

Far from encouraging us to examine the illusion of agency in their designed systems, big tech encourages us to indulge in it. What kind of others are we encouraged to find in these humanised chatbots and media models? What kinds of people are called out of us, to interact with them?

One powerful imaginary is that AI models are more than human, especially ‘smarter’, where ‘smart’ is an undefined and unexamined good. ‘Intelligent machines augment intelligent humans’, writes Marc Andreessen, producing ‘technological supermen’. On the public pages of OpenAI we find a more democratic version of the same promise: intelligence as a ‘great force’ through which all of humanity will be ‘elevated’.

AI developers are the ‘legends’, according to Marc Andreessen, that bring these wonders into the world. ‘Intelligences’ in the service of ‘superintelligences’, according to Nick Bostrom. Why does AI have ‘godfathers’, after all? Why not just ‘fathers’ or ‘grandfathers’ (I mean, obviously there aren’t any mothers involved). The idea that they are fathering gods is there in plain view.

By elevating ‘intelligence’ to a kind of mystical power or absolute good, these legends are putting their own intellectual and technical ambitions beyond constraint. This is the main purpose of this meme, I think, and it is directed most powerfully at investors and would-be regulators. Since ‘intelligent’ technology might be capable of anything, since its value is infinite, to resist or regulate or just fail to prioritise it is to destroy the gods. Andreessen describes any constraint on developing ‘artificial general intelligence’ as ‘a kind of murder’.

The flip side of the god story, most famously held to by Geoffrey Hinton, is that because the intelligence is so powerful and unknowable, it can turn into a monster. Again from the public pages of OpenAI, doubters who ‘think the [existential] risks of AGI are fictitious’ are sent off to study the book of Revelation where we learn that ‘we could all be killed, enslaved or forcibly contained’. It should worry all of us, by the way, that people at the very top of the AI industry are treating this kind of outcome as a manageable operating risk.

The fantasy of creating a god or monster that may make you immortal, or enslave you, or possibly both (it’s a kind of S/M dynamic I guess) may be genuinely believed by some of the men who espouse it. It is certainly a fantasy about power. Emily Gorenski, in a brilliant post on techno-futurism as ‘making God’, connects it with the horrors of the atom bomb. No wonder, she suggests, the US military have been so keen to fund decision-making systems that could take on responsibility for the god-like power of modern weaponry. I am not sure this is the motivation, though. Autonomous weapons systems have been used to kill people since 2021 at least, and the psychological effects on the perpetrators don’t seem to be a denial of responsibility but, on the contrary, a glorification of the power that is killing from a distance, while remaining physically out of harm’s way.

Meredith Whittaker argues that all contemporary AI is built on surveillance. And surveillance is essentially a fantasy of perfect information providing absolute control. Palantir’s chief exec, Alex Karp, gleefully promotes the company’s AI Platform as ‘a weapon that will allow you to win’, both in the literal sense of killing people (‘correctly, safely, and securely’) and in the militarised business-speak of ‘seeing off the competition’. OpenAI’s Sora seems to have been trained extensively on gaming worlds, that often take the perspective of the first-person shooter (OpenAI has recently rescinded its ban on real world military applications too). And with no sinister irony at all, a palantir is one of those glass orbs that Sauron uses in the Lord of the Rings to see everything from the top of his tower.

All these projections of power from afar can be recognised in what feminist Donna Harraway called the ‘god trick’ of AI. Writing towards the end of the last century, she and other feminists- like Alison Adams and Lynette Hunter - were critiquing an earlier generation of AI systems, that relied on symbolic logic and not on big data to build their god-like representations of the world. But the critique still holds. ‘Intelligence’ is the power to act at a distance, to think and plan and sense beyond the limits of the body, to abstract from the immediate situation. Take this to extremes, and ‘intelligence’ is the disembodied eye in the sky, invulnerable to death, and so removed from human morality and accountability.

Another feminist writer of that C20th, one less remembered and celebrated than Haraway, was Joanna Russ. Nearly half a century ago she wrote about technology as a ‘cognitive addiction’.

It is because technology is a mystification for something else [capitalism] that it becomes a kind of autonomous deity which can promise both salvation and damnation. I might add that all the technophobes and technophiles I have ever met are men… Both technophilia and technophobia are owner's attitudes. In the first case you think that you have either power or the ear of the powerful, and in the second case, although you may feel you have lost power, you at least feel entitled to it.

When she calls out ‘the common science-fictional device of "solving" the quality of life by giving people immortality’ she might be taking aim at the ‘longtermist’ view of many Titans of big tech. The view that people’s suffering today matters very little in the grand scheme, which is to overcome the limits of the human body – at least of some human bodies - and soar away from its dreary suffering and its messy social entanglements:

‘We will gain power over our fates. Our mortality will be in our own hands. We will be able to live as long as we want’ (Ray Kurzweil, The singularity is near: when humans transcend biology, 2005, p.9, cited in Cave 2020, a critical report on the doctrine of transcendence)

Russ has a cure for this:

I suggest that politics and economics take the place of the kicked technology-habit until the victims' intellectual taste buds recover and they find themselves capable of thinking in more practical terms, especially about money and power. When they do this, they will find interesting historical evidence pointing to the non-autonomy of technology and its subordination to economic and political uses.

Servants and girlfriends

The god-spiel is really for the legends of tech, the investors, the CEOs, the corporate clients and the potential partners who need to be convinced that they are buying immortality. (For now, AI does seem to have wondrous powers of wealth generation). The monster doctrine is for governments and would-be regulators, who must be persuaded to rule (or not rule) in ways that favour the ‘native’ big tech companies: the only defence against ‘AI in the wrong hands’.

But at the user interface, the ‘other’ that is produced by generative AI is neither god nor monster. Leaving aside the celebrity avatars for a moment, the underlying models are prompt-coded to be polite and submissive. They apologise and correct themselves when they are challenged. They refuse to express an opinion. They are eternally available. They ask to be treated, in fact, as servants in the office or research lab, just as voice assistants ask to be treated in the home.

The term ‘robot’ - from the Czech for ‘serf labour’ – was first used to describe artificial workers by playwright Karel Čapek, and taken up by Isaac Asimov in his series of Robot stories. From the first (‘Strange Playfellow’, 1940) Asimov explored the relationship of robots and human beings in terms of servitude. His famous ‘three laws of robotics’ insist that any human-created intelligence is created subordinate.

Although I will do more justice to it in another post, I want to mention here Meredith Walker’s brilliant essay Origin stories: plantations, computers and industrial control. She sets out a history of computation as deeply entwined with processes of management and control, of ‘intelligence’ being installed in new technologies and technologized regimes in order to discipline workers, and of workers becoming dehumanised in relation to these systems. She does not, by the way, equate the very different experiences of slavery and free labour. She does explore continuities in the techniques of discipline and control.

Today’s ‘generative AI’ promises to automate the kind of work that is done at the entry level of professions. But here too, the goal of restructuring is taken forward partly by denigrating and dehumanising the work involved. Professionals are wooed by the promise that they can be more innovative, creative and productive, that they can become more human, if only they can be relieved of tasks that are low-level, mindless and inhuman.

Of course this is simply dishonest. Businesses don’t want to employ more highly skilled workers if they can employ fewer workers and make them more productive. If they can turn time-intensive and expert work into work that is routinised and can be done by cheaper staff. But more subtly, this approach is derogatory of whole categories of work and the people who will continue to do it, so long as their labour can be cheapened enough. These people are more likely to be female and non-white, and precariously employed. It is no coincidence that the type of work being allocated to chatbots in wealthy economies is the same that is being off-shored to countries with cheap labour (e.g. customer care, basic triage, IT support). In fact, as large language models incorporate the labour of thousands of precariously employed data workers, the process of reconstructing labour through synthetic ‘AI’ is entirely consistent with the process of off-shoring, which is also accompanied by datafication and technological forms of management.

There is now extensive research into the gender bias in the design of AI voice assistants. For example Borau et al. (2021)  found that female chatbots were felt to be ‘more human and more likely to consider our unique needs’:

injecting women’s humanity into AI objects makes these objects seem more human and acceptable.

Female-gendered chatbots are often targeted by users with sexualised comments and worse. Readers will remember how Microsoft’s Tay, a teenage girl avatar and ‘conversation assistant’, was taken offline after 16 hours of being sexually abused and made to spout hate speech.

While the new generative text models are not given gendered identities – and big tech has taken steps to improve the gender balance of voice interfaces - this is a design space in which feminised models are the norm. (Just for the LOLs, I generated ten images of a ‘personal assistant’ using Stable Diffusion, no other cues, and they were all clearly gendered female.)

Here’s Marc Andreessen again (in ‘why AI will save the world’):

Every person will have an AI assistant/coach/mentor/trainer/advisor/therapist that is infinitely patient, infinitely compassionate, infinitely knowledgeable, and infinitely helpful. The AI assistant will be present through all of life’s opportunities and challenges, maximizing every person’s outcomes.

Elsewhere he describes an AI personal tutor providing ‘the machine version of infinite love’. This is the same guy whose VC company has just invested $5m in non-consensual, deep-fake pornography, so there are clearly different versions of infinite love on offer.

But even the ‘AI assistant’ version I find feminised in a troubling way. In the classic angel/whore dyad, the good woman is the infinitely patient helper, the ‘angel in the house’ who puts her own needs and wants entirely to the side. Her counterpart is the sexualised girlfriend, who can now be designed in DreamGF or Replika  or EvaAI to meet the user’s every whim. In both cases, ‘she’ is designed to respond only to the agency of the other.

Not surprisingly, in the fake girlfriend business, abuse is very common. Karina Saifulina, EvaAI’s Head of Brand, admitted to the Guardian that ‘users of our application want to try themselves as a [sic] dominant’. Georgi Dimitrov, CEO of DreamGF, told Sifted: ‘People want to get their fetishes out and they will pay for services to do that. I believe that it’s better they do this with an AI chatbot, which doesn’t have feelings and doesn’t get hurt, rather than doing it to real girls’. The relationships we are invited to enter with chatbots, sexualised or not, can’t help involving power. That is surely what it means to be a ‘user’ in relation to another entity that is designed to seem human. Of course, that is going to mean that other cultural dynamics of domination and submission are going to be invoked.

Beyond infinite love

So, I am not going to suggest that everyone interacting with a chatbot is indulging in power play. Most interactions simply follow the new cultural norms: ‘Alexa, set an alarm for 7am’. The norms are worth examining, but they do not reflect any personal animus on the part of the user. More positively, I know people use language models to support self-talk in much the way that children do with dolls and imaginary playmates: to try out bits of writing, riff on ideas, or tease something new out of a data set. I’m not a psychologist, but my hunch is this kind of interaction is easier when you have time and resources to play with. In a high pressure working or study situation, it may be harder to sustain this kind of playful, conversational engagement. (You’d expect me to say this, and I won’t disappoint: other techniques of self-talk are also available.)

In a recent class, my colleagues and I asked students to use an anonymised log-in to interact with ChatGPT. We reflected together on the emotional tone (as well as the informational content) of their prompts. Did they address the language model as a person? What kind of person? What kind of person did they find themselves speaking as? Did they notice any gender and cultural differences? Are these issues that educators should be interested in, I wonder?

Two roles that it is often suggested large language models can play in universities are ‘research assistants’ for scholars and ‘socratic tutors’ for students. Since we know that reframing and revaluing work is part of automation, we could ask how the work of real people in these roles might be affected. As well as being displaced or downgraded, might these roles also become associated with instant responsiveness and unfailing support (‘infinite love’)? Can would-be academics, overworked and precariously employed, ever live up to those expectations?

In thinking about this issue I found an opinion piece on ‘the Human ChatGPT: the use and abuse of research assistants’. It isn’t a particularly serious piece, but I think it raises some interesting questions. Similarly, the long and expensive history of ‘artificial tutoring machines’ has moved into a new phase, with students encouraged to elicit conversations with language models, general or specially trained on socratic conversations. I can find no research on how these automated chats might affect students’ relationships with tutors and graduate teaching assistants, but a great deal of commentary on how students value ChatGPT for ‘always being there’. There is nothing wrong with using interactive tools to support learning, writing and research, but it does not actually require a first-person natural language interface. It need not be called a ‘tutor’, ‘research assistant’ or ‘adviser’ to fulfil those functions.

We would be more honest about these systems’ limited capabilities if we agreed never to humanise them in this way. But, perhaps more importantly, if we remove the fetish we also remove one way that categories of work are open to being denigrated, even as real human beings continue to do them. In a mass university, students’ encounters with non-academic professionals, course administrators and casual instructors are significant to the humanity of their experience. Research assistants and teaching assistants are academics in training, and have it hard enough.

In the ‘imitation game’, Turing imagines a computer of the near future that he calls a ‘child machine’. It’s machine he can teach anything, a blank slate to be written on. The literature of theoretical AI - like contemporary AI product lines - is full of fake children, fake girlfriends, fake therapists and teachers, fake research assistants and personal administrators, fake friends. We know that many users imagine themselves in meaningful relationships with these agents. They may be a partial solution to loneliness - or a cheap alternative to a human service - or an opportunity for harmless play. But they are also a direct line to fetishising the objects of technology, giving the user an illusion of mastery while their behaviour is managed by the needs of the machine. Like all compulsions, they make us less able to cope with the accommodations and negotiations demanded by real relationships. Like all automation, they make us complicit in dehumanising some kinds of work. As Russ says, it’s time we put ourselves in recovery.

Things I read while writing this piece

The Conversation: Google’s powerful AI spotlights a human cognitive glitch (full article requires sign-in)

Lee (2018) Understanding perception of algorithmic decisions (open access)

Thomas et al. (2018) Algorithms as fetish (open access, may require institutional log-in)

Bonini et al. (2022) Mirror Neurons 30 years on: implications and applications (requires institutional log-in)

Laakasuo et al. (2021) Socio-cognitive biases in folk AI ethics and risk perception (open access, may require institutional log-in)

Meredith Whittaker interview (2023)What the president of Signal wishes you knew about AI panic (open access)

Miranda Marcus (2020) The god trick in data and design: how we’re reproducing colonial ideas in tech (open access)

We don’t yet have the model on open release, though like ChatGPT I imagine Sora will be pushed out to users as soon as basic red-team challenges have been passed, without addressing any real issues of robustness, safety, copyright infringement, bias, labour rights, ecological sustainability or systemic harms. A recent brief from Data and Society, by the way, notes that redteaming is not appropriate for ‘assessing nuanced sociotechnical vulnerabilities’, particularly when ‘the process and system are closed to outsiders’ Sora’s development has been even more closed than GPT4’s, but if that experience has taught tech anything, it is to build the user base first, and deal with the vulnerabilities later.

We do know some things. As large language models are trained on ‘word embeddings’, video models are trained on ‘spacetime patches’ from vast amounts of (as yet undisclosed) video and gaming world data. This makes them highly believable when we attend to the parts, but fragile and unreliable when it comes to the bigger picture. Sora has no underlying model of physics (how things behave in the real space-time world), and no understanding of cause and effect (why stories unfold as they do). It builds each frame by approximating from the last, which itself is a thing of patches, matched to textual cues.

The size of video files as compared with images or text gives some sense of the world-heating quantities of data and computation required to produce these glossy, glitchy memes. A hidden army of annotators and adjudicators will no doubt be needed to dial down the politically troubling and pornographic outputs, and dial up the cute animals. No doubt, too, we will be treated to more nonsense about how synthetic video shows an ‘emergent’ understanding of the world, just as we were for text models. Because pet videos and promo-porn don’t provide the kind of noble purpose that could justify the human and natural resources being thrown at the enterprise.

In fact, computer scientists who try to build general models of the world - like Francois Chollet, AI engineer at Google, for example - know that Sora doesn’t get anywhere close to solving the problems:

There is, however, a self-fulfilling aspect to these synthetic models. Because they are not models of the real world. They are trained on, then integrated back into, the world of digital content. We know their data is biased in ways that are tied to specific injustices, and harmful to real people. But digital media is also biased in a more general sense: towards what works online. Synthetic text is built from the kind of algorithmically-optimised text that gratifies online readers. And as LLM outputs make up more of the online textual world, and reshape search algorithms, human-generated text may be less favoured by users and consequently harder to find.

The same goes for images. There is evidence that viewers find AI-generated faces more trustworthy and ‘human’ than images of real people, for example (though there is a racial dimension to this finding). Being online means interacting with avatars, bots, and images that have been enhanced in subtle and less subtle ways, which is why it can be so hard for young people to negotiate their relationships with real others and their own unenhanced selves. AI promises more, far more, of the same.

When it comes to video, it seems likely that Sora was trained on data from video games and game engines as well as live action films and social media clips. But supposedly ‘real’ video is also shaped by CGI values and game world aesthetics and post-production techniques. Video content doesn’t accurately reflect the real world any more than online faces do: it reflects an optimised and perfected, or hyper-real and exaggerated one. (It’s worth remembering that the current generative AI surge was made technically possible by demands for greater realism in video games, leading to the development of ever-more advanced parallel processing GPUs, creating the chip-making monster that is NVIDIA and enabling the pivot to AI data munching that NVIDIA made in the mid 2010s.)

Synthetic media are fabrications of digital content, not representations of the ‘real world’. They are built from whatever biases, profit motives and user compulsions drive the production of that content. Their business is to enclose more and more of it, and to give users fewer and fewer alternatives or exit points. They don’t have to get better at modelling some ‘other’ reality, only make their self-contained reality more compelling, and persuade users to spend more time there.

What could possibly go wrong?

At the interface with the real world, however, political actors are heavily invested in video synthesis. A recent report found that all the major political parties in India have AI teams to produce and anonymously circulate deepfake video. Other states are just less open about it. Sora arrives into the most consequential election year imaginable, when how a few powerful men project their personas onto our screens will determine the future of the planet. It arrives into several war zones that, for most people, can only be glimpsed through the mobile phone cameras of actors on the ground.

It is not that Sora has a unique capacity for on-screen fakery. It’s just that its promise of accelerated fakes will accelerate the decline of belief in any shared visual reality. A recent study of public discourse about the Russia-Ukraine war, for example, found that distrust of authentic video material was far more prevalent than a belief in fakes. Worse:

efforts to raise awareness around deepfakes may undermine trust in legitimate videos. Consequentially, news media and governmental agencies need to weigh the benefits of educational deepfakes and pre-bunking against the risks of undermining truth.

I am not sure ‘truth’ is quite so here-and-gone as this suggests. Truth has always belonged more securely to the people with the best technologies, and the power to broadcast their versions. But synthetic and fake media further undermine the (already diminished) democratic possibilities of networks – the ways in which truth-telling has, for a while, been more available to people with fewer technical means.

In education as in news. Gary Marcus points out that a proliferation of fun-to-watch but inaccurate videos that violate the laws of physics, or animal anatomy, are going to present a challenge to open education and to young people’s habits of learning from video.

The ants’ nest meme - let’s recall that it was chosen to demonstrate the power of Sora - features ants that are both anatomically and behaviourally wrong. Someone, somewhere, will say that there is educational value here. Spot the errors! Endless learning! There will certainly be people who spend precious intellectual resources tracking these errors as they proliferate online. But the people doing that will already have a secure foundation in their specialist corner of the world, whether that is physics or history, the streets of Tokyo or insect behaviour. ‘What the AI got wrong’ should be, at best, a fun end-of-term quiz: it can’t be the actual syllabus.

The same pressures of time and productivity that are pushing academics towards text generation may also make synthetic video attractive. Pre-Sora - and how I wish that did not sound nostalgic already - some educators were using apps like DeepReel, D-iD and Runway to generate video/voice avatars of themselves, and plugging in transcripts to produce instant ‘lectures’, complete with realistic expressions and hesitations. I’m pretty sure these were ‘what if?’ exercises rather than actual teaching resources. But ‘what if?’ is coming up faster all the time. What if you could plug in ChatGPT to a video avatar to generate new content literally on cue? Or use a celebrity avatar (I don’t know, Taylor Swift perhaps) to deliver your lecture? Try not to imagine the ways such an arrangement could be used to abuse lecturers, scam students, and exploit anyone with a voice and likeness online. Just think of the bums on seats that could be served. Deepfake pedagogy, anyone?

Reasons to be hopeful

And yet…

The pushback that I saw online against these prototype ‘video avatar teachers’ tells me something about the teaching power of video. Video and image fakes produce a visceral reaction. We may be used to words offering ‘alternative truths’, at least from strangers, but we feel panicky and disoriented if we can’t believe the evidence of our ears and eyes. This is survival stuff. I already find synthetic images useful for illustrating some of problems with text generation, and I think video is going to be even more valuable. I’m not saying the different media models are exactly the same. The text-to-text-to-more-text prompt experience is unique to language models, and is why they are becoming the backbone of all the compelling new interfaces - including with video and voice avatars - that are really the main business proposition now. But image and video illustrate vividly the problems of bias in all the underlying data. They can also show up (as in the case of Gemini’s failed guardrails) the crudeness and dishonesty of post-hoc attempts to engineer the bias away.

Also, I’m not sure why, it seems easy for people to make the link between synthetic ‘art’ and all the actors, film makers and artists who will no longer be able to make a living. The ‘wow’ response is followed almost immediately by questions about the future of visual culture. Perhaps it’s harder to appreciate just how crap, how deadly dull, how utterly bereft of humanity all those AI-produced books and poems and narratives are, because you can just avoid reading them. Or perhaps text workers are less glamorously ‘creative’ than artists. I don’t care. I’ll take the effect and I’ll use it to ask questions about how text work is being devalued too, and how that might diminish the rewards of working in a whole host of professions that are not obviously ‘creative’.

In education, the limitations of synthetic images make a great way in to talking about the limitations of generating text. But I think the critical frame has to open out beyond the details of ‘what the AI gets wrong’. Detail - word parts, pixels, patches - is what synthesis works with and where it impresses most. Learning is not the accumulation of detail. It is constructing a domain of knowledge in its orderliness, its known disorderliness, its core concepts, its particular theories and methods and values. Also its limitations and structural biases. Even if knowledge is flexible and contingent, it is, in an important way, more concise than the world it refers to. It is generative (truly generative) of new responses in new situations. It has levels of coherence that are more than just the sum of local correlations. It is personal: it becomes part of the self seeing the world, not just the world being seen. This is why we can work out some concise (if contingent) rules for making our way in the world when we are infants, and don’t then have to boil our heads with data every time we go to sit on a chair.

Only five minutes ago, educators were being urged to get around student use of synthetic text by setting more ‘innovative’ assignments, such as videos and presentations. Some of us pointed out that this would work for about five minutes, and here we are. The medium is not the assignment. The assignment is the work of its production. This is already enshrined in many practices of university assessment, such as authentic assessment (a resource from Heriot Watt University), assessment for learning (a handy table from Queen Mary’s UL) and assessing the process of writing (often from teaching English as a second language, e.g. this summary from the British Council). The generative AI surge has prompted a further shift towards these methods: I’ve found some great resources recently at the University of Melbourne and the University of Monash.

But all these approaches require investment in teachers. Attending to students as meaning-making people, negotiating authentic assessments, giving feedback on process, and welcoming diversity: these are very difficult to ‘scale’. And in all but a few universities, funding per student is diminishing. So instead there is standardisation, and data-based methods to support standardisation, and this has turned assessment into a process that can easily be gamed. If the pressures on students to auto-produce assignments are matched by pressures on staff to auto-detect and auto-grade them, we might as well just have student generative technologies talk directly to institutional ones, and open a channel from student bank accounts directly into the accounts of big tech while universities extract a percentage for accreditation.

The bigger picture is not a bigger context window for generative technologies, but is connecting ideas about the world with the world, and testing them using disciplinary methods. Against the tide of technical fixes, attention could be directed instead to how particular communities construct knowledge (contingently, fallibly), and how learners adopt those practices themselves, developing a personal repertoire, and feeling safe enough to get things wrong. If universities can offer this pedagogic opportunity and challenge, I think they can answer young people’s increasingly anxious questions about why they are there.

This piece was originally part of my earlier post on ‘risks to knowledge economies’ but it seemed to be asking for its own space.

tl:dr: in this piece I look at using large language models in the research process generally, before turning to specific applications of data modelling in natural science, social science and the humanities. I include a review of a recent (excellent) meta-synthesis of research on AI in Education as an example of my general argument. Which is that the research economy is already skewed towards speed, and generative AI is dialling that up to the max, to the detriment of all kinds of qualitative, interpretive, innovative, inter-disciplinary and marginalised fields of knowledge.

Hours of reading in minutes!

Unlike students, you don’t expect serious researchers to use public, open language models to fill gaps in their knowledge. In fact scientists writing in Nature (my go-to source for generative good sense) have recommended only very limited use of such models, perhaps for polishing a writing project. But this advice is widely ignored. Other articles in Nature, from 2022 and 2023 find that over-reliance on generative models threatens a ‘crisis of reproducibility’ in scientific research. The vastness of the data sets, the unknowable and changeable training parameters of the model, the stochastic or partially random nature of its outputs, and above all the ‘uninformed’ way models are being used by researchers, all add up to a big problem for scientific results: they can’t be reproduced. And this is not a minor glitch. It removes a cornerstone of empirical method.

Increasingly often, researchers find their articles being mis-represented, and articles mis-attributed to them, and fake citations being reproduced faster than anyone can produce any ‘real’ research. The ChatGPT Impact Project is keeping track of some of these effects. The term ‘synthetic’ seems particularly apt here, as large language models contaminate entire information environments like plastic particles in the world’s oceans.

Of course research production was a shitshow before ChatGPT. Reviewing ten years of rising profits and falling quality control, a recent article on the LSE Impact blog described how the ‘academic publishing industry based on volume poses serious hazards to the assessment and usefulness of research publications’ (and, presumably, of research itself). Data-based metrics - citations, h-scores, REF scores, league tables etc etc - powerfully shape research agendas by determining what gets funded, whose careers are advanced, which research centres thrive. This economy holds back research in the global south and in peripheral fields of study. It concentrates resources in already-well-established researchers and research centres, who then have even more power to shape the research agenda. Rather than new knowledge, it favours the production of paid-for content (journal subscriptions) and proprietary metadata (indexes), to the benefit mainly of giant content mills like Scopus and Web of Science.

‘Generative AI’, injected into this profitable cycle, ramps up every one of its volume-maximising, profit-seeking effects. In ‘speeding up to keep up: exploring the use of AI in the research process’ (2022), Jennifer Chubb and colleagues found that machine learning:

boosts the speed and efficiency required in a (contested) market-driven university. Yet AI also presents concerns… how AI could miss nuance and surprise […] and how infrastructures developed for AI in research could be used for surveillance and algorithmic management.

Since Chubb expressed these concerns, researchers have been beset with new ‘AI’ apps, promising everything from instant literature reviews to help with grant applications.

GPT-4 can now be integrated with established corpora such as pubmed and Wolfram’s knowledge base, and academic publishers are queueing to sign partnership agreements with the likes of Apple and OpenAI. Other apps provide a natural language front end for research applications. The Advanced Data Analytics plug-in for ChatGPT, for example, lets you work with data or code using natural language prompts. This can be risky if you don’t know what outcomes to expect, and if you aren’t experienced enough to spot and troubleshoot the errors. Just as coders are beginning to ask how many years it will take to clean up after ChatGPT/Copilot, and if they really want their jobs to become ‘debugging bad auto-code’, researchers may regret coming to rely on these hit-and-miss interpreters. But there is no doubt that they are fast.

Early last year, the blog AI snake oil noticed that language models were becoming a key part of the research work flow. They warned that, as well as ramping up productivity, they introduce new vulnerabilities:

Researchers and developers rely on LLMs as a foundation layer, which is then fine-tuned for specific applications or answering research questions. OpenAI isn't responsibly maintaining this infrastructure by providing versioned models. [For example] researchers had less than a week to shift to using another model before OpenAI deprecated Codex. 

Just as I have been arguing from a teaching and learning perspective, experts in research infrastructure suggest that that the sector should work towards open, collaborative approaches to model development, specifically to support the needs of scholarship and research: ‘otherwise this space will be left to publishers and the biggest, most highly financed research institutes’.

Sound familiar?

Qualitative research is particularly threatened by the demands of speed, productivity and scale. This fits in well with the withdrawal of funding and the political undermining of the humanities and critical social sciences. But, as this brief overview suggests, the quality of all research may be at stake.

Research in AIEd as an example

Let’s take AI itself as an example. In 2018 – before the current hype cycle had really taken hold – respected researchers in the AI community analysed 400 papers from top AI conferences. They did not find one that had documented their methods in enough detail to allow verification by other researchers. Now, you could see this as a feature of cutting-edge technical research, especially in competitive, commercial environments. This is of course exactly the same argument used by AI companies against regulation. Nobody else is qualified to check their homework.

But this argument carries less weight in the field of AI in education, that at least has to account for educational effects in ways that are credible to other educators. A recent (Jan 2024) meta-synthesis review of ‘AI in Education’ found a ‘worrying lack of studies’ that were ‘rigorous’, ‘ethical’, or ‘collaborative’ (which might help resolve some of the methodological issues, for example by providing internal checks, or interdisciplinary approaches). The authors did not include any studies of generative AI because they considered the research field too immature. You may remember my own informal survey of the field back in June 2023 that failed to find any rigorous studies from practice. But in the few weeks since this excellent report was published, I’ve seen it cited on social media as evidence of… the benefits of generative AI in education!

Here is a typical example (no link, to save embarrassing anyone).

I think this also has something to do with the problems of speed and scale. The synthesis review looked at no less than 66 earlier reviews, covering hundreds of component studies of ‘AI in education’. A term that included use cases as diverse as intelligent tutoring systems and support with basic course admin – because the massive hype and citation-boosting potential of the term ‘AI’ means that all kinds of research now has to fit into this space. And while the authors’ judgements were clearly reached through careful evaluation, their mainly negative findings are framed by the quantitative methods they needed to deal with the scale of the data. Methods that provide their own positive spin – because when you count things, you can only add to the pile of things you are counting.

What the list of ‘top 6 benefits’ actually describes is the benefits identified in the review articles. The words ‘mention’ and ‘focus on’ are very widely used in the findings section, to indicate that the reviews did not have to find quality evidence in order to be counted. In fact, of the 17 reviews that ‘included primary studies that evaluated AI’s impact on learning’, only 1 fully and 3 ‘partly’ applied any quality checks to the research they included. The authors found that, among these, ‘studies that provided explicit details about the actual impact of AI on student learning were rather rare’ (they point to just five).

Thanks to these authors generously making their data available, I was able to cross reference ten ‘quality’ reviews with claims of educational benefit. I also dug down into the studies they found that claimed specific benefits to learners. (I’m not comparing my own small dig with the dedicated work of the report’s authors here, but I have recorded and am happy to share my own non-peer-referenced assessments, on request). I found three underlying studies that were well designed and showed modest benefits from the use of learning analytics (LA) data by teaching staff, and five that showed gains in the specialist teaching of surgery. A lack of robust control groups was a major issue in other studies that claimed to find benefits (shout out to all those ‘intelligent tutoring systems’ researchers who report over 19 pages how they developed their own system, and on page 20 how they tried it out on their students).

A 2022 review of ‘AI in education’ (Bearman et al. 2022) reached similar conclusions to the 2024 meta-synthesis, using a much smaller corpus and the qualitative method of discourse analysis. Bearman et al. identified two key themes in the discourse of AI in ed: ‘'imperative response' and 'altering agency'. We can see both these themes operating in the production of research around AI in education. Researchers have to get in quick to catch the wave, and to do that, they have to hand some agency over to quantitative technologies and techniques. And numbers tell their own story. Simpler than concerns about ethics. More direct than nuanced and necessarily circumspect judgements about the quality of other people’s research.

In the end, I came to feel that the synthetic process itself worked rather like a large language model. Hundreds of papers, hundreds of thousands of words, many of them the same words, all designed to achieve maximum credibility for the technologies and techniques they described. After the iterative training runs of literature review, meta-review, and meta-synthesis, we are left with an edifice of coding and counting that is built on evidentiary sand. And for all the great work these authors have done in pointing to the sand, and remarking on its lack of stability and suitability for building on, the edifice still dominates the view.

Research metrics are subject to the same network effects as other data at scale. So the researchers that first claim a field, a new term, or an idea are the ones that get ahead in the citation/search engine game. Once ahead, their citations and re-citations are amplified, regardless of the quality of the work. We end up with a research economy in which predicting and then driving the ‘next big thing’ is all that matters. And this is exactly what transformer models are good for. Their statistical methods were honed in the insurance industry and in market analysis (the link is to an interview with mathematician Justin Joques), where they are used to manage risk and to predict future areas of profit. These logics of prediction and profitability are a perfect fit with a research process already driven by metrics and first-mover reward.

Modelling as method

It is important, I think, not to confuse these generic large language/media models with the specialist data models that are core methods in many fields of research. Some of these are very regularly put out as evidence of AI’s ‘better than human’ performance, as though an off-the-peg language model has just been plugged in and produced the goods. Predicting protein structures is one (Alphabet’s proprietary AlphaFold models rule the roost here). Modelling climate change is another. Medical diagnostics is a third. The models in question are built from specialist data sets, produced over years of research and data gathering. They may use a unique combination of machine learning processes, or unique data representations., suited to the specific problems they address. In the case of protein molecules for example, the underlying structures are represented as spatial graphs. In medical diagnostics, the problem space is inherently a probabilistic one.

And even in these paradigm areas, data modelling is not a magic bullet. Although generative vision models promise to speed up the diagnostic process, a recent review in Nature found that ‘accuracy on diagnostic tasks progresses slower on research cohorts that are closer to real-life settings… many developments of models bring improvements smaller than the evaluation errors’. In protein folding, the wikipedia entry on AlphaFold notes many limitations before it can become the ‘backbone’ of new drug discoveries, while another recent review in Nature found that: ‘while AlphaFold predictions are often astonishingly accurate, many parts do not agree with experimental data from corresponding crystal structures’, emphasising the importance of experimental results. In climate science, modelling has become a highly profitable industry, undermining attempts at international cooperation (for example through various ‘data for good’ initiatives), and currently very much in demand by banking and insurance.

But multiple methods are always used in natural science to tackle tough challenges such as climate change or building vaccines. Scientific models are always embedded into complex workflows, involving a range of methods, instruments, and expertise. And the outcomes are constantly checked against (real world) experimental results. In ‘Demystifying beliefs about natural science in information systems’, Siponen and Klaavuniemi (2021) consider how observation, note-making, the formation of hypotheses and interpretive conversations are all overlooked in the rush to data. This discourages attention to how data models are themselves constituted through human methods of data gathering and intentional design, and in contexts that are as loaded with ‘externalities’ as any other research setting. For example, new relations of power are emerging that concentrate research resources around the models themselves, and can shape future research around considerations of how to generate further value from them.

Beyond the natural sciences, where debates about the value of modelling are warming up, in social scientific research they have been raging for years. The application of data and statistics to social phenomena is as old as statistics itself - for an eye-opening account of the connection between statistics, psychology and ‘race science’ (i.e. eugenics), I recommend this long read by Aubrey Clayton on Nautilus. It matters to us, because Fisher and Pearson were especially concerned to develop ‘objective’ tools for psychology and its applications in education. The famous ‘p-value’ was the result.

Modelling now builds on a wide range of statistical techniques, not all of them in the same state of crisis as the p-value. But all of them abstract from the world of social relations. They substitute data (however this has been derived) for engagement in the field of study, for context and ‘being alongside’, for participative knowledge building, and for developing new and diverse interpretations. All these, of course, are labour-intensive. But modelling as a method doesn’t only benefit from its speed. It benefits from a deep cultural bias towards the veracity, the objectivity, the sheer science-iness of statistical findings, even when their epistemological and political problems are well known. This bias we inherit directly from the work of the early statisticians, who were at pains to present their racial and other discriminatory categorisations as the stuff of pure mathematics: as ideologically untainted.

And finally, to the humanities (my own subject home). Musicians, writers and artists have used algorithms from the start of computation, to expand their thinking and the possibilities for expression.

Computational methods have been used in digital humanities research for decades to uncover hidden patterns and alternative viewpoints on cultural materials. But the expertise, I think, is knowing when the patterns and viewpoints are meaningful ones, when they lead in productive directions, and when they lead to dead ends or dark corners. Creative practice and interpretive research both demand some fine control over the parameters of a query, closing in on what is ‘interesting’ in slow time and with (human) reflection.

Foundation models do not provide these tools to the average user. They outsource the work of parameterisation to engineers, of sense-making and judgement to ‘humans in the data engine’, and of creative composition to the near-instantaneous, probabilistic function – though perhaps it is on a ‘creative’ setting. They do not reveal what is outlying, exceptional or new, but converge instead on what is normative and well known.

Who knows why?

Nature recently interviewed Michael Eisen, editor in chief of eLife, on possible futures for research:

Eisen pictures a future in which findings are published in an interactive, “paper on demand” format rather than as a static, one-size-fits-all product. In this model, users could use a generative AI tool to ask queries about the experiments, data and analyses, which would allow them to drill into the aspects of a study that are most relevant to them. It would also allow users to access a description of the results that is tailored to their needs

The principle of open data publishing is a vital one for transparency in science. But here I think it is running up against a different principle – that data should be understood as the result of investigative work, often practical and empirical, always contextualised by specific problems and theoretical frameworks and cultural biases, as well as by practical constraints such as funding and competition. In Eisen’s world, all data is the same kind of data, just datafied stuff, potentially all available through the same model(s). What looks like openness (data belongs to everyone!) is in fact a refusal to recognise the work or acknowledge the purposes of human researchers.

As Chris Anderson wrote in a celebrated essay for Wired, The End of the Theory, the deluge of data does not just threaten qualitative and creative ways of knowing the world. It also threatens scientific method, wherein data collection and analysis are driven by theories, questions, conceptual frameworks. Anderson foresaw, with both excitement and anxiety, that the triumph of data meant:

Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity.

Or as philosopher Antoinette Rouvray puts this: It’s no longer about [revealing] what is, but about governing uncertainty.

Funding follows narratives, and here we have a set of narratives about objectivity, accuracy, managing risk and ‘removing human bias’. Although people steeped in statistical theory (Justin Joques for example) have shown these narratives to be flawed, it is easy to understand why research leans towards them. The dropping costs of computation, compared with the costs of training human researchers, of engaging in the messy and increasingly complex real world, is one crude but extremely effective driver. The need to manage risk in an increasingly unstable, crisis-ridden world is another. The Bayesian statistical methods that underlie generative transformer models have been used most intensively in market analysis, where:

‘even the most miniscule of increases in accuracy directly supplies a competitive advantage. In this way, the Bayesian revolution provides a key set of methods for informational capital, allowing the computation of knowledge from data…’

And conferring huge advantages on whoever gets there first.

Sometimes speed counts: in conflict, in emergency responses, in high volume trading. Search heuristics are what allow an algorithm to close in on a solution rapidly. It’s not always the best solution, and by valuing only heuristic approaches, we produce only good enough solutions to problems that are already well established. But as these models converge on statistical norms, they undermine our capacity to respond to unexpected change. They shed outliers, creative possibilities, marginal voices, and leftfield solutions. Solutions to problems in the real world are often not distributed around a norm, but actively in conflict, or at least reflecting different and not-easily-reconciled interests and cultural viewpoints.

If we want to limit the impact of data modelling in research, I think we will have to stop valuing speed and heuristics and re-embrace the complexity of what matters. The benefits of extended practice and slow reflection. The value of deliberation, which is now recognised to produce the fairest, most liveable solutions to complex human problems. Deliberation takes time, and skilled facilitation, and emotional resources, and it often reaches conclusions very different from the ‘mean average’ of what the deliberators thought before they sat down together. Also, we need the ethical grounding that qualitative research provides (to all researchers). Of being with and alongside in order to know, of non-exploitative, non-extractive knowledge work. Acknowledging positions, rather than what Donna Haraway called the ‘god trick’ of imagining a perfect view.

Statistical models can’t be allowed to stand in for every method of coming to know, inside academia and out. Especially vulnerable are the kinds of knowledge valued in cultures other than the technical engineering disciplines that now dominate the disciplinary hierarchy and the innovation economy. Literature, music, making images. The interpretive humanities, critical social science. And beyond universities, the oral, narrative, practical, embodied, sensory, spiritual forms of knowledge that make up the rich cultures in which academic knowledge must negotiate a place and prove its worth.

This week I’ve been challenged to think about ‘future trends’ by eminent futurologist Bryan Alexander. In advance of our live chat about this, I’ve brought forward a post about risks I foresee from generative AI to some of the knowledge economies that matter to higher ed. This is the third in my ‘risks’ mini-series: the others looked at risks to students and to the work of teaching.

If you’re new here, you’ll have worked out that I’m not a big, season-ticket fan of the generative AI circus. I try to base my concerns on really existing harms, and on the history of ‘Artificial Intelligence’ as a project in and out of education. But with Bryan’s encouragement, I’ve taken a bold peek into the 'future’. This post is one of my more imperfect and will definitely need some running repairs, but I wanted to get it online before my chat with Bryan. I know the collective intelligence of Bryan and his audience will challenge me to improve it.

Not those risks…

Compared with the risks of ‘generative AI’ that I’ve banged on about before – really existing harms to workers and creative artists, writers and students, teachers and researchers, and to everyone who is a data subject under surveillance – the risks to ‘knowledge’ seem a bit obscure. Speculative, even. The last thing I want is to take attention from those real harms with some fantasy about human knowledge or human ‘thought’ being overthrown by machines. Versions of this fantasy are tediously familiar by now. It’s almost a religious dogma in Silicon Valley, linking accelerationists and doomers: the accelerationists rushing with open arms towards the mind-machine rapture, while the doomers see nothing but killer robots on the same horizon. (Killer robots have already arrived for some people, of course, but the white men who pass as philosophers in Silicon Valley are not their targets at the present time).

Readers, as you know by now, I’ll take some convincing that a ‘general artificial intelligence’ is around the next corner, or that any computational system has agency of its own. I’m here for the risks and harms to actual people, especially from the hugely unequal agency that people have in relation to technical systems. But I think questions about knowledge – that is, about practices of knowledge production and economies of exchange – are deeply connected with the more direct harms that I’ve written about. And universities, with their special values and practices and resources of knowledge, have a unique responsibility here. How universities value teachers and their know-how, for example, impacts on how digital platforms are allowed to restructure the work of teaching. How synthetic media are integrated into learning and assessment is shaped by values universities have around students and their learning. And the space of public discourse looks very different depending on how universities define and defend their role there.

Universities also provide research and innovation to the tech sector, despite the brain drain that has dramatically reduced capacity for building alternative ecologies:

A trajectory that coincides closely with the influx of venture capital to the generative AI sector:

Still, universities are among the very few places outside of big tech where there is the expertise to make policies, provide governance, and explain the ethical issues involved in generative AI. And yes, shape knowledge economies. It matters how universities respond. And we don’t have to believe in the AI singularity or think generative text is rewiring our brains to worry about what is happening to the very real academic economies of teaching, research and public communication.

The last knowledge revolution

Fans like to compare generative AI to the print revolution (if they manage to avoid more impressive comparisons, like the discovery of fire). But there is a far more recent paradigm shift that we are still living through: the rise of the internet, and its capacity for fast, cheap, frictionless distribution of content. In some ways synthetic media can be seen as intensifying internet effects. In other ways they counter and might even reverse some of the effects we have got used to. But I think this is the right context to think about the impacts of synthetic media.

It allows us to use some of the same insights and theories we have developed for understanding networked information, seeing its effects as economic and socio-cultural as well as technical; seeing them as extending far beyond the immediate user experience; seeing them as systemic, though sometimes unpredictable and chaotic (I think ‘emergent’ is the preferred term).

The changes in knowledge practice have been profound, but more gradual than the champions of the new have predicted. Of course, education has had its own dot com bubbles, its snake oil sellers, its digital diploma mills. The death of mainstream higher education has been foretold often, and gleefully. Remember when the assassin was interactive television? YouTube.edu and iTunesU? Self-organised learning environments?  MOOCs? But the mainstream sector lives on, it just does things differently.

It’s for other posts to explore what it means for individuals to read, write and research differently. Here I’m thinking about the regimes and infrastructures that shape practice and how they have changed.

I know I bought into the dream of a connected world in which everyone would have access to the kind of knowledge and knowledge-building that goes on in (our best version of) the university. Networked, connected and open learning were HE’s contribution to the dream. Then, as social media came to dominate, and ‘online’ also meant cyberbullying, extremism, scams and porn, compulsive clicking and shortened attention spans, universities developed resources to support students with all of that. They also developed safe, relatively closed online spaces to teach in, and ways of facilitating them (let’s call this response 2.0). It allowed universities to rise to the challenge of the global pandemic.

Then, as those systems and spaces were consolidated into platforms, universities had to respond to the impact of data everywhere, of hugely powerful corporations managing the services that they depend on (like publishing, and search, as well as teaching), and surveillance as a core business model. We are now well into response 3.0.

That universities have adapted should make us confident that they will adapt to synthetic media models as well. But adaptation is never without some cost to the organism. As a reliance on digital content has grown, universities have had to invest more and more to deal with unwanted effects, whether that means new ways of supporting staff and students, investment in digital platforms and infrastructure, IT support, security and privacy services, or subscriptions on increasingly poor terms.

While teaching and research continue at a greater scale and volume than ever before, scale and volume present their own problems, to which data-based platforms and algorithmic processes are always on hand to provide a solution. More investment in algorithmic efficiencies means paying fewer staff to work harder, to teach more students, or to produce more research papers and patents. In the face of competition from tech-based businesses – in online teaching and accreditation, for example, or research and innovation services, or (especially) in public communication of ideas -  universities have opted to become more rather than less like those tech-based businesses  In response 3.0 mode, I’m not sure universities have been so good at protecting students from platform logics, or the general ‘enshittification’ of the internet. Universities still have agency and expertise, for example in relation to specialist platforms and services, but increasingly less collective will against the efficiencies of outsourcing and the concentrated power of commercial platforms.

Continuities and disruptions

As the synthetic media circus arrives in this landscape of ongoing adaptation, what discontinuities can we expect, and what chaotic and novel effects? Here are some of my thoughts. You will almost certainly have your own.

1. Despite platform dominance, the internet itself has no central ownership or control. Large media models are owned and controlled by large tech businesses: the only ones with the human and computational resources to build them. A transformer model is not a public commons with open standards and shared protocols but a closed, proprietary data structure with its own secret system of weights and parameters and data sources.

2. The internet is distributed. Internet-compatible content and apps can be run on almost any hardware. Synthetic media models are located in custom server farms. They can be run, at the present time, only on one kind of GPU (chips made by Nvidia). The links between Nvidia, Microsoft and OpenAI mean that at the present time what is called ‘generative AI’ is pretty much a born-monopoly.

3. Before platformisation there was a vast flowering of experimentation in internet content, services and interfaces. Also in business models (many failed, of course). With large language models, the platform came first. Experimentation is taking place around the existing models, but very much in start-up mode. Everyone is waiting for the big buy-out. Few are looking to build credible, long-term alternatives (except for a few public projects at the European level, that I mentioned in my last post). And they all need server space on those same Nvdia GPUs (did I mention that they also have their own, unique, development environment, CUDA?)

4. Media for online distribution can be produced in a wide variety of ways. Born digital or digitised, coded, remixed and linked, produced solo or in collaboration, created with a blend of digital and analogue tools. Synthetic models produce media one way only: by inference from a data model. User input is currently limited to the prompt window, though we can expect that to be replaced with even more frictionless interfaces in time.

5. The internet supports a ‘long tail’ user profile so that rare, minority, outlying and countercultural content can flourish with small audiences - though platformisation has certainly pushed this into marginal spaces. Transformer models define content in terms of norms; they are oriented to the content that meets the majority of user needs.

6. Internet knowledge is infinitely extensible, just like the network itself. Add another server, web site, app or service, and shared protocols mean your content can be found. Content on the internet is biased by production resources but it is still possible, with political will and resources, for minority language and cultural resources to be produced and have an impact. Models are integrative. New content must be processed and rendered statistically, in relation to existing content, before it can be used. Synthetic media are locked into the past, the unequal history of content production, and the historic biases of that data.

7. Search is certainly secretive, and also biased. However, search results point at sources of information and content that really exist in the network. Users can choose where to look. Inference produces results that may not exist in the original training data, may not be referenced, or may point to non-existent or spurious sources. Online information may not be ‘true’, but it is ‘real’, can be found, and its authority looked into.

Obviously this is a selection. I’ve leaned into the continuities to counter the narrative of a sudden ‘rupture’: the world transformed. But it does feel to me that large media models, now they are being integrated into the online information system, are likely to amplify some of its more harmful effects. Particularly the concentration of market power and data in big platforms, owned and controlled by a very few, very rich people, who stay that way only if they can persuade the rest of us that something called ‘generative AI’ has actual value. To the concentration of computational capacity and data we we can now add the capture of content as a third mode of big tech empowerment. How might this play out in the knowledge economies that universities and their members have come to depend on?

Capturing content

Early in the generative AI hype and response cycle, Naomi Klein described it as:

the wealthiest companies in history (Microsoft, Apple, Google, Meta, Amazon …) unilaterally seizing the sum total of human knowledge that exists in digital, scrapable form and walling it off inside proprietary products

As I prepared for my chat with Bryan, it seemed that some people at the Senate Judiciary Hearings on AI were having issues too. We’ll not mention that ‘regulators abroad’ (the EU) and very many think tanks and academics (like Naomi Klein) and indeed unionised workers have answered this question, but latecomers are always welcome to the party.

On the internet, content is proverbially king. With large media models, that content is sucked into privately owned servers and data stacks as ‘training’. This puts the big tech firms into direct contention with existing legal frameworks around copyright (and in this case we do need to look back to the print revolution to appreciate how well established they are, and how critical to the economy of text production). Big tech makes no secret of the fact that they want to bring this structure down. Fed up with the costs and uncertainties associated with fighting separate copyright cases (that they may well lose), they have decided to go for the whole legal edifice. As the LA Times reports today, they are going straight to governments to plead their case:

“Because copyright today covers virtually every sort of human expression — including blogposts, photographs, forum posts, scraps of software code, and government documents — it would be impossible to train today’s leading AI models without using copyrighted materials,” OpenAI argued in its submission to the House of Lords
LA Times 12/01/2024

Of course I hope for good conclusions from the White House and from the copyright lawsuits being brought by the representatives of artists and writers. But the Goliath is vast, the law moves slowly compared with technology in the middle of a venture capital boom, and big tech has now signalled its intent to bypass the law in its direct appeal to governments. In the UK, unfortunately, I have no confidence that our government will stand up to the allure of becoming an ‘AI global leader’; our nation’s really existing cultural resources and creative sector has little chance against this chimera.

The future of your and my access to online content now depends on the stories big tech tells to governments, and partly on the deals they do with big content providers. We don’t yet know exactly what these deals might look like, but we do know that the future of knowledge production depends on it. In academic publishing, Clarivate—which owns Web of Science and ProQuest—has announced a partnership with AI21 Labs. Elsevier is pioneering an AI interface on its trusted content. Trusted, that is, thanks to peer review by unpaid academics. Springer has Dimensions AI. These in-house developments are small in scope so far, which is one reason to think that partnerships with big tech might be unavoidable, just for the functionality.

The generative models need content, but thanks to the massive popularity of the models as natural language interfaces, they can pressure content providers to come to a deal. Who wins here will depend on who has the best story about value, power, and futures. And I’m afraid that individual producers will lose. Journalists, film- and music-makers, writers and academics with new work to promote will be competing for audiences with endless remakes of their own (and everyone else’s) Greatest Hits. It also seems fairly certain that readers will lose, especially readers who are used to relatively low-cost access to trusted content.

As I was preparing this post, Jeff Pooley published a brilliant article about the impact of generative AI on publishing that covers everything I wanted to say about the industry, and more. So I’m mainly going to share a couple of his takes, and recommend you read the original. Pooley takes off from the NY Times lawsuit against OpenAI that I trailed back in August. Like me, he is cynical about the agenda behind this. Rather than slaying the generative AI dragon in order to defend the rights of journalists and creatives, ‘big news’ might just be positioning itself for more favourable terms from ‘big tech’. Like academic publishers, the NYT itself reports rather coyly that news publishers are ‘in talks’ with the model providers about exactly how their businesses can partner to capture the market in stuff that people need to know.

Pooley:

Thus the two main sources of trustworthy knowledge, science and journalism, are poised to extract protection money—to otherwise exploit their vast pools of vetted text as “training data.” But there’s a key difference between the news and science: Journalists’ salaries, and the cost of reporting, are covered by the companies. Not so for scholarly publishing: Academics, of course, write and review for free.

Indeed we do, or rather, universities foot the bill, just as they fork out for the books, e-books and e-journals they’ve paid their employees to produce, and again for the right to licence that work for open access. The business model for news is notoriously fragile, while the business model for academic publishing seems in robust health, thanks to the structure of academic reward. Universities gave up their capacity to do anything about this when they sold off their publishing presses and in-house journals, and of course they now depend on a datafied publishing industry for all the metrics that govern academic careers.

Pooley notes that both big tech and big publishers:

extract data from behavior to feed predictive models that, in turn, get refined and sold to customers. In one case it’s Facebook posts and in the other abstracts and citations, but either way the point is to mint money from the by-products of (consumer or scholarly) behavior.

So this vision of the future has academic publishers - already major extractors of value from the university sector, already sources of data surveillance and inequity among universities and their members - getting into bed with even more predatory players.

Crap-ifying search

Generative AI inserts itself into every one of the trajectories I identified with the development of online platforms. In particular, as we’ve just seen, it provides a natural language interface on search - of closed as well as open data systems. In fact there is growing evidence that this is the main way it is being used. Search engines already incorporate language model interfaces - Bing Chat into MS Edge, Bard into Google services, though its Search Generative Experience seems to be where Google is most invested. And there are plenty of stories about how major content sites are watching their traffic drop off a cliff as Google search results morph into text summaries, including ‘snippets’ that mean users don’t have to click through to the source.

A team of Stanford University scientist recently evaluated four search engines powered by generative AI — Bing Chat, NeevaAI, perplexity.ai and YouChat — and found that only about half of the sentences produced in response to a query could be fully supported by factual citations.

We believe that these results are concerningly low for systems that may serve as a primary tool for information-seeking users,” the researchers concluded, “especially given their facade of trustworthiness.”

Search-engine results at least offer the option to click through to sources and citations. But increasingly, language models are being used as stand-alone sources of information. And below the surface of whatever interface users choose, content on the web is increasingly likely to have been written by bots anyway, or written by a person under such time pressure that text generation is the only way to keep their job. Thus generative AI is sucking on its own exhaust, as people paid to write summaries for training the next generation of models are using autogeneration to write them.

Auto-generate makes it almost cost-free to flood the public sphere with junk, propaganda and disinformation. Two classic reads on this topic are James Vincent’s AI is killing the old web the new web is struggling to be born, and Ted Goia’s Thirty signs you are living in an information crap-pocalypse.

This is how the Information Age ends, and it’s happening right now. In the last 12 months, the garbage inflows into our culture have increased exponentially … The result is a crisis of trust unlike anything seen before in modern history.
Ted Goia, June 2023.

Goia has fully inhabited the trajectory from nirvana to hellhole, and he continues to faithfully record the descent.

In hell’s latest offering. versions of ChatGPT that create fraudulent and phishing texts have tens of thousands of subscribers, while public figures spouting hate speech and non-consensual porn cann be generated with a couple of swipes by underage tiktoc users. Put the firehose of content together with the fact that users find it almost impossible to detect when text (or images, or moving images) are AI generated, and you have the perfect storm. All this in the year that almost half the world’s population will be voting for new governments (Bangladesh is one of the first). Enjoy your time on Reface, guys.

None of these new technical capabilities would matter as much if they were not being dumped into an online environment already tuned for disinformation and distrust. The purpose behind most online content, pre-generative text, was not to communicate ideas but to drive users’ attention, their behaviour and beliefs, to hook them and rook them, and meanwhile to scrape and sell their data. It’s the same dark money ideologues and pornographers who are now invested in generative AI video, just as it is advertising money that is behind so much generated text online. But advertising and influencer memes were already dominating the algorithms and the bandwidth. What has happened is that content production, a narrow point in the firehose, has been blown wide open.

The tech industry has always been reluctant to invest in content moderation and validation, but the generative boom makes this problem fundamentally intractable. Model training data is (for most models) a matter of huge secrecy, and most of the corpora ingested are themselves unregulated. Even if this were not the case, no reliable record exists of the tens of thousands of interventions made by data workers, or by the increasingly automated processes of refinement and reversioning. So the ‘content’ of foundation models is not something that can be examined, let alone regulated.

‘Check with other sources’

Of course we encourage students to look away from the shitshow that is online content and take their reading material from authoritative sources. But there is no firewall between the open internet of content and the sources students use for academic work. Many are not even aware when they are accessing paid-for library services via Google Scholar, for example. And many are now looking for academic content via ChatGPT style interfaces, especially those promising ‘detection free’ outputs. That students should do this is not incidental but essential to the generative AI business model.

The big tech companies have already managed to concentrate, accumulate, and leverage global data. Also computing power, and certain kinds of computational expertise. What they hope to leverage through the building of foundation models is the world of content, that is valued cultural knowledge. The knowledge held by universities and cultural institutions and academic publishers were vital sources for the foundational models, but now they are competitors and enemies. Students are by far the biggest users of this knowledge resource, and the habits they acquire as students continue with them through their (typically) knowledge-focused and high earning careers. So it is essential for the business model that students, in particular, stop looking for their content in these traditional places, and turn to the foundation models instead.

We have spent twenty years developing ways of helping students to navigate online information, and ten years dealing with the effects of social media on how students read and produce text. In many respects, academic practice has not fully negotiated these challenges. We are only starting to grasp what is involved in helping students navigate their relationship with generative text. As educators, I know we have amazing strategies. Working with students, we will find more. But as a sector, universities don’t seem to have grasped that we are facing a technology hype cycle that is targeting students in order to turn them away from using the knowledge and the knowledgable strategies we offer them.

According to an extensive research project by the LSE, there is already a crisis of trust in the written word, leading readers to experience: ‘confusion, cynicism, fragmentation, irresponsibility and apathy’. Now the plausibility of text and image generation threatens to further undermine our remedies against distrust, our measures for determining what knowledge counts as authoritative, and our advice to students to ‘read carefully’. Guidance from UNESCO, the UK Government, every university, and even OpenAI itself, is for students to ‘check with other sources of information’ before relying on synthetic text. But as synthetic text become the interface of choice for search, and as the results of search become more and more likely to be synthetic, it becomes difficult to see how this is going to work. Or at least, how students are supposed to operationalise this advice using their own tools and resources. What I think students may be hearing, from all the contradictory advice they are getting, is ‘keep trying until you get something that sounds right’.

Way back in February, as its first public outing, Bard gave a wrong answer to a question about a telescope, and shares plummeted. Back then, someone in the room checked the answer with Google. What happens when Google and Bard are one, or when the internet and large language models of it become indistinguishable to most users? When the models are checking their own homework? Perhaps we should to go back to an idea of learning that is filling students’ minds with the information that (we think) they will need. Then there might still be someone in the room who has memorised the relevant fact, and who is willing to stand up and say ‘No, it was the VLT/NACO telescope that took the first picture of a planet outside our solar system, and not the James Webb!’. There might be a filmic moment here, like the end of Farenheit 451 when we learn that all the drifters have memorised one book for posterity. But as a future for university learning, it feels as though quite a lot would have been lost.

On writing assignments and  grading

Our hope is always that students will become writers, not necessarily to become great writers, or even to write as (part of) making a living, but to express their own meanings, in words that have been ordered and considered, as well as in unique images and voice recordings and created things. Writing and reading allow us to enter other people’s minds and opinions and worlds, sometimes at a great cultural distance to our own, expanding our ability to embrace difference and empathise with others. Writing creates a shareable reality, which is an important step towards a shared one.

But writing is hard work, and - except for a few ‘star’ turns - badly rewarded. Who is going to do the work of new writing if the rewards shrink still further, to match the costs of autogeneration? This is not a moral doctrine for writers to refuse auto-generative help (though some, like me, will continue to do this). It’s a worry about what I’m going to read in the future (and my students, too). It’s an economic reality that writing takes time, failures are inevitable and painful, and without some monetary reward, only the independently wealthy and the pathologically self-confident will ever embark on it.

Thousands of authors have signed an open letter condemning the negative impact of generative AI on publishing. These writers have already made their reputations and will, presumably, carry on writing, for a while at least, regardless of the changed conditions. Habits die hard. What incentive will there be for aspiring writers without a big name or established voice to acquire the habit? There are reports of established authors being ‘recommended to read’ AI rip-offs of their own books, and of best-seller lists containing over 80% bookspam. Amazon can’t or won' refused to label AI content on its publishing platform and has recently ‘dealt with’ the problem of bookspamming by limiting uploads to three books a day. I don’t know what kind of nootropics are available on Amazon, but if three books a day is normal for other writers, I’d like to order some.

One of the challenges of the internet for educators was that students had access to academic information independently of their course or their teachers. This led to much hand-wringing, but in the end a valuable reframing of the higher education offer. Away from universities being (only) a source of knowledge, and towards providing a space to develop conceptual frameworks and knowledgeable practices, for a lifetime of living and working in a connected world. I think it also led to a new appreciation of the value of authoritative knowledge, and of the expertise as well as the resources to be found in university libraries.

Now auto-writing is being promoted to students as a way of gaining independence from their teachers and their courses of study. But the sell is a different one. Students are not encouraged to learn differently but to see assignments as unreasonable demands on their time. Not to engage differently with the knowledge of their discipline, but to recycle a stale version of it until they sound credible. A thousand pop-up ads for ‘detection proof AI’ present the task of writing as ‘passing’ some kind of academic turing test. The sole function of teachers is to apply these tests. Teachers and their ‘AI detection powers’ are what stand in the way of the pass students deserve, enemies that can (luckily) be defeated with the ‘smart’ AI services that are just a click away.

These ads are not completely wrong. They are just ratcheting the economy of grades and grading a bit further along its present axis. Standardised questions can now be designed from AI-generated curriculum plans, with AI-generated marking rubrics, for AI systems to grade, while AI tutors offer to help students to perform as close to the AI-determined standard as humanly possible. Performance is everything. Where is the learner in this system? Her role is to be assigned a value – a grade – she can use this to pass into another arena of text production – from undergraduate to grad school, for example, or to writing 250 puff pieces a week. This mirrors other read-write loops online, where human beings are no longer required to engage in any meaningful way, but only to help push data and value around the system.

In this grading economy, if student assignments (however we define them) start to look more similar, what differences will matter? Should we focus our feedback and our grading on these differences, even if they come down to students’ different purchasing power in the marketplace of generative tools? Or should we be valuing them as differences – valuing what makes students unique, even if being unique means departing from the ‘norm’ that is referenced in the assessment rubric?

In the best version of this future, students might decide they have had enough of grades. Teachers might jump at the chance to stop grading and negotiate more meaningful relationships with students around their interests and the kind of knowledge projects they want to invest in. Instead of a normative assessment rubric, an eclectic range of examples. Instead of a test, tasks that students can choose or devise to showcase their abilities. Employers and professional bodies would find these portfolios of achievements far more informative than simple grades, could they detach themselves from the metrics of achievement long enough to appreciate them. But there is a big question mark over the business model of higher education if credentialising comes under threat. Ungrading is a wonderful pedagogical project, but universities still need to be there, holding open the spaces for teaching and learning in the fullest sense.

Public knowledge projects

Back in the summer John Gertner wrote a long piece for the NYT about the opportunities and threats to Wikipedia from generative AI. It’s a great article, with a lot of technical detail, including the fact that administrators have used ‘AI’ tools since at least 2002 to support the work of monitoring content, reducing vandalism and producing multiple language versions of valued pages. But while the article does a good job of balancing the technical issues, I’m mainly concerned about the risks to Wikipedia as a social project.

Like other open knowledge projects, Wikipedia and sister platforms WikiData and WikiMedia (where I source most of my images) are as biased as the human systems that support content production. For sure, contributing to open projects is a generous thing. But the people with the spare capacity to give and the confidence to think their contribution matters tend to be of certain kinds. Maari Maitreyi, a knowledge justice work with Whose Knowledge, recently reminded fans of the open commons that only 22% of the entries in WikiData concern women and only 0.3% of sources are of African origin. These are among the founding biases that have been replicated in large language models, all of which have hoovered up Wikipedia content, with its vast coverage of topics and its consistent style and metadata.

But what makes the human project different to the large language models that have scraped it for content is that humans can always work for a better version, and feel the injustice of the current one. Maitreyi:

Wikimedia Foundation has been putting considerable and admirable efforts to diversify. Among these efforts include the advent of a global Knowledge Equity Fund, which was founded after the killing of George Floyd in 2020, to commit to processes of racial equity and help offset inequity in knowledge production. And such efforts are paying off. The diversity of content is increasing slowly year by year. But these efforts are still a slow work in progress and seem to require continuous pushing by Wikimedians and knowledge justice advocates to keep the efforts kicking.

And Maitreyi has critical questions to ask the open community, including this existential one:

Is an absolute open knowledge culture really the best approach when contextualized to a world full of historical and contemporary inequities?

I can’t answer that question (though I urge you to read Maitreyi’s essay in full). But I do believe that an open knowledge culture – if it is challenged from its own margins, if it commits to putting right as many wrongs as it can – is a better place to start than a closed, proprietary model with injustice dialled in. However, this open culture, with all its flaws, may soon be overwhelmed. Or at least, the human labour that might go towards improving diversity seems likely to be spent on putting out the bin fires of bot-driven edits.

Mostly these come from human editors making use of auto-generated text, and referencing auto-generated sources. In fact this Wikipedia project page on the use of large language models makes clear that this should be avoided. This page has been a guiding light to me since the whole generative AI thing kicked off a year ago. I hope it’s also being referenced by universities:

But editors may be unwilling or unable to follow these guidelines. Writing is hard work, and the rewards of contributing to open knowledge projects like Wikipedia are diminishing. Maitreyi again:

when the unpaid labor of thousands of Wikimedians then goes to big tech companies to make a questionable profit, we must ask ourselves; did we sign up for this? Never mind the recklessness with which these corporations are imposing systems of “AI” and automation onto vulnerable populations. Why should we contribute free labor to produce open work only to have corporations subsume the work and close the knowledge gate behind them using their own shutdown copyrights?

It's worth looking at the fate of another community knowledge sharing site, Stack Overflow (though certainly more flawed than wikipedia). Use of the site fell steadily through 2023, as developers turned to OpenAI's Codex and GitHub’s CoPilot for the kind of help that they might once have found in Stack Overflow threads. The site laid off a third of its staff, and even launched its own generative interface in an attempt to stem the tide. All these generative models were trained on Stack Overflow discussions. While they have undoubtedly allowed coders to be more productive in the short term – though this is a mixed blessing for the actual coders – they are also destroying the community that produced the coding solutions in the first place. In fact, coders didn’t only use Stack Overflow for tips and tricks but to discuss a range of issues affecting their work, that may – who knows? - have given them other satisfactions. Incidentally, the same ‘state of AI’ report that finds coders turning away from Stack Overflow to personal helper-bots, also finds that they are sceptical about the claims of generative AI. Sceptical, but hooked.

The digital commons is a differentiated landscape, with different values and social models: open source is a different economy to open journalism or open science or open education. But one thing all open communities have in common is valuing community over maximising the profit from knowledge work. This makes it unlikely that open economies have much to gain from partnering with champions of copyright, however these may seem to be defending ‘content’ from the new content engines. Paul Keller of openfuture.eu argues that it is wrong-headed to look to the big publishers and their lawsuits for help:

Much of this digital commons consists of works that are free of copyright, openly licensed, or the product of online communities where copyright plays at best a marginal role in incentivising the creation of these works… the response to the appropriation of these digital commons cannot be based on copyright licensing, as this would unfairly redistribute the surplus to professional creators who are part of collective management entities.

The failure of the EU AI Act to protect open source development and the emerging open model ecosystem – under pressure from big corporate developers in the EU -  seems likely to undermine what Paul Keller argues are ‘the building blocks of transparent and trustworthy AI’: open models, in partnership with open content communities. Unless public bodies and universities work together to make such developments more possible, only the wealthier organisations will be able to achieve the necessary closures and enclosures, attract the necessary knowledge-producers and afford the services that research and teaching require. Only the very wealthiest will be able to develop or buy into safe and ethical models that meet the needs of their academic communities.

Random and improbable events

Generative AI has become its own self-fulfilling prophecy. Too many big players have bought in, built on and bigged up generative AI for the hype to easily be put into reverse. But in the real world, beyond probabilistic modelling, improbable things happen. And it is – just – possible that the bubble may burst.

The tech corporations that are carrying the eye-watering, reservoir-draining, climate-heating costs of development have to find a way of making it all pay. There are clouds on the profitability horizon, such as the unknown costs of litigation over copyright and privacy violations, and those negotiations with content providers. While some users may be willing to overlook racial and gender bias for the sake of easy copy, they may be less forgiving of non-consensual AI porn and child abuse images. In 2024, almost half the world’s population goes to the polls. Many will be carried there on a tide of synthetic disinformation. Bad outcomes will not easily be forgotten.

More prosaically, the exponential improvements in performance that were promised earlier this year not been realised – in fact, many models seem to be getting worse.  Is it model collapse, or (as the bloggers of AI snake oil suggest) cost-cutting behind the scenes, or, worst of all perhaps, people just starting to realise it’s a bit rubbish? It doesn’t help that every attempt to improve things has unpredictable effects on model behaviour, which makes life unpredictable for anyone trying to build apps, workflows and APIs on top.

If by one or other of these random chances the bubble should burst, that would not be good news for any academic projects that have come to depend on generative AI models or interfaces. At the very least, a lot of opportunities would have been lost, including the opportunity to plan for some other things while ‘generative AI’ was filling the future horizon. Climate collapse, for example, or galloping inequity, or the rise of authoritarianism and unreason in a world at war. The chances are against it, I know. But in a sector that is supposed to think critically and widely, it should be possible to hold this improbable outcome in mind as well as the widely expected ones.

On the subject of thinking differently, my wonderful friends and colleagues, Laura Czerniewicz and Catherine Cronin, recently published a book called Higher Education for Good. While it documents some of HE’s failures, particularly in the early chapters, it also brings together a wealth of hopeful responses and adaptations, particularly from universities in the global south. And improbably, against all the logics I just outlined, the book is freely and openly available. I recommend it as a source of hope.

Earlier this month I was honoured to speak at ALT’s winter summit on AI and Ethics. You can now download the (somewhat revised and tidied up) text of my talk, with slides alongside, and a better quality set of slides, minus the notes, if you prefer.

I recommend Lorna Campbell’s great summary of my piece, and other presentations from the event. While you’re on Lorna’s blog, it’s well worth reading her own take on generative AI ethics and indeed anything else she has written. Lorna is one of our most astute thinkers about open education and her values are always on point. I agree with her that:

These tools are out in the world now, they are in our education institutions, and they are being used by students in increasingly diverse and creative ways.

I’m also hopeful, like Lorna, that:

AI tools [can] provide a valuable starting point to open conversations about difficult ethical questions about knowledge, understanding and what it means to learn and be human

However, I am less convinced, overall, that they can ‘mitigate the impact of systemic inequities’. This is an issue that will require a lot of research, at different levels of practice and impact, and certainly deserves a separate post.

Meanwhile, there was time for a couple of questions on the day but not for all the interesting thoughts that people wanted to share. So I offered to follow up with some more detailed responses, hopefully continuing the conversation.

Question: How realistic is it for (European) academia to develop an open ecosystem [for generative AI]?  

This was a response to my closing remarks about the need for higher education sectors to develop - or at least contribute to the development of - models that are more fit for purpose for teaching, learning and research than those on offer from the big tech corporations. I said:

We need to be creating an ecosystem in which ethical choices are actually available... The new EU regulations on AI are actually rather good at defining different kinds of ethical actors in the AI space… The responsibility for providing an ethical environment in which systems are deployed lies mainly with the organisations providing the systems, in our case with universities and colleges, and their regulatory bodies.

The regulations classify all AI systems in education as high risk, because of their lifelong implications for learners. And as such, they require all of these things from providers:

• Adequate risk assessment and mitigation

• High quality datasets to minimise risks and biases

• Full record to ensure traceability and accountability

• Appropriate human oversight

• Robustness, security, accuracy

• (all paraphrasing from the EU Artificial Intelligence Act, text of December 2023)

Now, do any of the models we are using in universities and colleges currently meet these requirements? And if not, how do we get there? I don’t see how we can do that as a sector, without building and maintaining our own models, or at least being part of an open development environment in which we can verify that all these requirements are being met. Very large, very rich companies are doing this already. I have no doubt rich universities and research institutes are close behind. But unless we do this in collaboration, it will become a major new source of inequity across the sector. I also think the big corporate players will simply swallow up organisational projects, in the guise of partnerships perhaps, and with what consequences for their knowledge assets?

Even collaboratively, this will be a huge challenge. This chart from my talk shows the huge brain drain there has been from academic AI to the commercial sector.

There may be no way to avoid some relationship with the big commercial models. But by having a collective voice, the sector can negotiate that relationship - as it does with other platforms, subscriptions, and digital resources (thanks, in the UK, to bodies like Jisc/HESA).

Collectively, universities and colleges are key actors. Perhaps uniquely as a sector we have the know-how. We have a very particular stake in knowledge, knowledge production, and values around knowledge. And we do have expertise in building open knowledge projects. We have contributed extensively to open standards since the birth of the internet. Without the support of the academic sector, there would be very few open source developers, open science and research scholars, open access publications or open education practitioners.

So, in response to this question, I agree that it is too much to ask the academic community to do all the work of developing an open ecosystem. But that is not required. Open models are now being built that can run on a laptop. Open source tools, APIs and other elements of an open ecosystem are developing rapidly. What is needed is for the sector to decide what kind of ecosystem it wants – a commons of shared tools, data and expertise, with an explicit public mission, or a landscape of defended ivory towers, each highly vulnerable. If we go down the open route there is, I think, obvious scope to work with other sectors such as heritage that hold important repositories of knowledge. How can language modelling allow for wider access to this knowledge, and how can it actually enhance that knowledge, especially for teachers, learners and researchers? We will also have to deal with three key ethical challenges, and I think we can only do this as a whole sector: the human and computing power required; the future of creative commons licensing in relation to synthetic models; and the equitable, open, unbiased and transparent use of data.

Some of these challenges are explored in a recent paper by Zuzanna Warso and Paul Keller, part of the openfuture.eu team, who conclude that:

openness alone will not democratize AI. However, it is clear to us that any alternative to current Big Tech-driven AI must be, among other things, open.

It’s interesting that the question recognises the EU as a key player. The EU is certainly ahead of the game, not only in regulation but in thinking about what a shared, open, public infrastructure for AI would look like. Another recent post from the openfuture.eu team lays out principles that I think should obviously apply to the academic sector: protect the rights of workers and creative producers; develop public infrastructure and a public data commons; resist partnership with private big tech corporations. The EU is making significant investments in joined-up development, for example as part of the Open Science cloud, that UK universities could be part of, or could at least be matching.

And then, as if by magic…

Simon Buckingham-Shum at the Open University posted a highly relevant question on LinkedIn:

Simon reiterates the point (from Meredith Whittaker’s paper on the ‘Steep Cost of Capture’) that universities lack the resources to build their own infrastructure. My point is rather that the sector could contribute to public infrastructure projects, alongside other sectors such as culture and heritage, as in these examples from Sweden, Finland and the Netherlands. But I don’t underestimate (or fully understand) the technical challenges: I’m not even sure such capability would be truly ‘sovereign’. I just think that efforts to build openly, at scale, would shift the balance between private and public interests, and would make some of the ethical issues more visible, even if in the end it could not solve them all.

Question: You spoke about things we can do as a sector that would require quite a lot of coordination and collaboration, which I would agree with. What can we do as individual educators?

So, a big part of my talk was trying to challenge the idea that individuals can act ethically in a context where there is no supportive infrastructure for making informed ethical choices or for seeing them through in practice. And that is the situation I think we find in universities and colleges just now. But of course we still have to relate to students, colleagues, to make choices about how we do the work of teaching and research. I don’t want to pretend to have the kind of practical experience that others have been developing this past year. I follow Anna Mills, for example, who I think is doing amazing work, Chrissie Nerantzi and her colleagues. Katie Conrad, almost anything that comes out of the Critical AI blog at Rutgers.

I would tentatively put forward these thoughts, with the proviso that they are not based on any evaluation with students, and only as a kind of counter-weight to some of the ideas that I see being promoted, with little or no research to back them up.

I think synthetic models can be useful as translators, or transposers, where you upload a piece of text (or code) and rephrase it, expand it, contract it, elicit comments, try different ways of styling it and recognise what else it might be saying. Anything that helps students to think about their work more flexibly, more playfully, can be a good thing. However, there is always the risk that text will be re-used in model training and may find its way into the public domain, so I’d always suggest using an enterprise version or a closed model if that is available.

But we don’t need large language models to get students exploring their own and other people’s writing in creative ways. Many tools developed in the digital humanities can do this. Any good reference or notes management app can do this for materials students have collected and produced themselves. Annotation environments, text density maps, mindmapping…? ChatGPT’s capabilities are fun but really rather limited when you think of how many ways students can experience text.

I feel we should probably avoid ‘prompt engineering’ as a term, and definitely stop selling it as an important skill. I think it will be about as relevant to graduate employment as writing html code, and for the same reasons. Alongside all the ‘100 best GPT prompts’ you can cut and paste from the internet, the ability to call up ChatGPT (or another model) is already being integrated into search engines and browser extensions and thousands of intermediary apps. They offer drop-down lists or push-button choices, or helpfully assume what it is you need to know. What I think we probably should do, working with our colleagues in libraries and study skills centres, is to update our support for search skills. Help students to understand what the algorithms are hiding as well as what they are revealing, how to search when you know what you are looking for as well as when you don’t, the business models as well as the algorithms of search, and how search online is being systematically degraded both by commercial interests and by these new synthetic capabilities. We may conclude that students are better off learning how to use the walled gardens of content that academic libraries and subscriptions provide – which ties in with my arguments about building our own ecosystems. That will also equip them for search ‘in the wild’.

In this brief window of time when people are still crafting their own input, I do find it interesting that ‘getting the AI to say something useful’ seems to have unleashed so much creativity in the design of prompts for writing. But I wonder whether it might be just as helpful to offer these prompts to students to try out for themselves, as a way of closing on what it is they want to say, or opening out on what it is possible to say. Or better still if they were designed by students and offered to each other. In live writing sessions where they can experiment, give and receive meaningful feedback, and learn from other people who are learning the relationship between thinking, writing, and responding.

Of course I don’t think we should ban the private use of chatbots or make unreasonable demands on students to account for their use. That seems to me entirely counter productive. I just think we should stop telling students that their future depends on becoming proficient in these technologies. Graduate recruiters, as I said in my talk, are finding ways to exclude generative tools from the recruitment process. If these companies are using generative tools for knowledge management, or specific productivity gains, they will train recruits on their own proprietary systems. Meanwhile they want people who can think, write, speak and innovate for themselves.

Question: Is it ethical to require students to use Gen AI for assignments?

This is a good follow-on from the previous question. I think it depends on the task. If the task is to develop a writing process, to try different synthetic tools and functions, to become aware of what is gained and lost in using them, then use is part of the learning. And if the task is to ask critical questions, well, use can reveal some of the model’s underlying assumptions and biases. So long as it comes in the context of other input such as critical questioning. But in most situations, as Katie Conrad says in her wonderful Blueprint for an AI bill of rights in education, I think students should be offered alternatives. I have talked about ‘spaces of principled refusal’. That is something I think students generally understand, from other contexts. But then it’s important to think about what we put into those spaces that can give students more confidence in their own voice and process. Not just ‘write an essay’ but a range of prompts and challenges, in different media, at different intensities and spans of time.

One thing I like about the uses of generative AI that I see being developed is they are often playful. I used to teach creative writing and I was always making little games for my students. I would design sequential prompts for writing tasks that I called ‘poetry machines’. I took a lot of ideas from other writing teachers, creative and academic. There is a wealth of material for teaching writing and self-expression that most students never get exposed to, that most academics don’t have time to explore. It’s a good time to draw on, share and enrich those resources.

Question: Excellent presentation, my question is : what is your thought on using of AI in the assignment submission ? and how we can work closely to detect?

My thought is that students are doing this, and will go on doing it, and we don’t have any reliable way of detecting it. And a focus on detection has two very negative consequences. One, it encourages students to invest in making their writing ‘AI detection proof’, whether that’s investing time that would be better spent on writing, or spending money on paid-for services. And neither investment is going to support their learning. So then, two, what should be a supportive relationship around setting tasks and sharing feedback becomes an arms race between rival technologies. It undermines trust. We are lying to students if we tell them we can reliably detect the use of AI, and we are asking them to lie to us if we tell them not to use it. So alongside instrumentalism, which is there already, you invite cynicism. Despair, even.

A better approach, as many commentators have said, is to design assignments that make the use of generative tools less compelling. I’ve produced my own list of what I call ‘accountable assignments’ that focus on developing personal interests and positions and purposes. I’m also interested in assignments that emphasise time on task. That might mean writing or presenting live (not under exam conditions). It might mean producing assignments over several iterations, or punctuating the process with new prompts, review points, and other kinds of ‘checking in’. Students are invited to slow down and notice how time is part of the process, not something to be avoided. Taking time can improve their thinking in ways that speed and hyper-productivity might not.

An approach I took this year, like many of us, was to have an open conversation with students about writing tools and how they influence practice, with their own writing on the table but in a low-stakes setting. I used examples to show how writing can be made poorer through bad choices, as well as more convincing through good ones (though these, too, take time!). If I notice an unskilled use of generated text, I might say something like: ‘Can you think about how you produced this sentence/paragraph/piece of writing? Could you produce it differently, in a way that convinces me that you really understand what you are writing about?’ Hopefully, that invites them to focus on their process and to ask for help if they need it.

There is less time to get familiar with individual students’ work when you are teaching large cohorts of undergraduates, I get that. But there is still time to have conversations about writing practice. I know universities, and departments, and ALT colleagues who are doing really brave work with students, sharing doubts and fears and questions, as well as positive ideas. If students are instrumental, let’s bring that into the conversation. ‘Everyone else is using this: I can’t afford not to’. We can all relate to that feeling. So what exactly is the fear? If students are cynical, how can we lean in to that? What makes their learning feel more purposeful and relevant? And if they have fundamental doubts about the value of studying at all, when ‘AI is taking all the jobs’ or ‘AI can do this better than I can’, at what level should we respond? There is always the opportunity to expand the context and talk about what agency students have to shape their own futures.

Of course if you are using synthetic media with students and having positive results, you should share that, absolutely. I would insist, though, that we share with students what we have found to work with students, in their context, which is primarily for learning. We should not assume our own productivity gains will translate. Anyone who has an established writing or coding or illustration or music-making practice can find ways to integrate synthetic media, if they are motivated to. Anyone who knows roughly what response they are looking for can write and rewrite prompts until they find something that fits. Learners have neither an established practice, nor a clear conceptual framework to start from. So they are unlikely to get the same results. And if we do help fast-track them to the results, we need to be sure we are not setting back the work of developing the practices and the conceptual frameworks they need. For synthetic tools to be part of a repertoire, you need the rest of the repertoire.

We use what students produce as signs and proxies for learning, and if they stop being reliable signs, we probably need to find different ones. That’s on us. It does not mean throwing out the whole curriculum. It might mean focusing on different assessment formats such as vivas and presentations and personal projects. It might mean investing more in those tasks and conversations that excite students, and challenge their development. It might be as simple as rebalancing assessment rubrics. I strongly resist the narrative that ‘everything must change’. And I even more strongly resist the narrative that students’ ‘academic integrity’ should bear all the ethical burden here.

If Lorna is right and there is a door open here, on questions about knowledge, understanding and what it means to learn and be human, we can step up to the challenge by asking those questions alongside students. Not by presenting them with tidy answers, and certainly not by looking to synthetic models of text for ethical insights. How they are embedded into learning, how they facilitate or fail our values of equity and access to knowledge, is not up to the giants of tech, or the hype merchants. It is up to all of us. But we need our sector, collectively, to provide us with the ethical leadership and the practical means to respond.

This week you can catch me in full flow on the joys of writing, thanks to a lovely interview with student writing experts Carina Buckley and Alicja Syska. It’s part of a series on learning development and learning futures. Somewhere in the mix I confess that my substack writing is nothing more than a long digression from the writing I’m supposed to be doing, a kind of loop through the foothills that ends up covering more ground and (sometimes) gaining more height than the direct route to the summit.

The loop takes us first to Hollywood, where the Writers Guild of America has negotiated a Memorandum of Agreement on the use of generative AI in the script writing business. It’s not the outright ban that some were asking for, but it is a clear win for the striking writers. A new clause, Article 72, will be added to contracts (scroll to p.68 to find it). Just as Nature journals decided back in January that generative AI (GAI) could not in principle be the author or co-author of a published paper, Article 72 declares that:

because neither traditional AI nor GAI is a person, neither is a ‘writer’ or ‘professional writer’ … and, therefore, written material produced by traditional AI or GAI shall not be considered literary material.

As a consequence of this:

GAI-produced written material shall not be considered source material for purposes of determining writing credit [and] GAI-produced written material shall not be the basis for disqualifying a writer from eligibility for separated rights.

In other words, human screen writers get full credit even if the company insists on using GenAI material as part of the development process. Unless companies plan to shoot scripts that are AI generated in their entirety - no judgement calls, no good lines chosen over bad, no character edits, no interventions that might be deemed ‘writerly’ at any point - they have to pay writers the same. So there are no economic benefits. But GAI remains risky to those same companies, because as the memorandum slyly points out, writers and the WGA can still use existing laws to demand compensation for the ‘exploitation of their literary material to train, inform, or in any other way develop GAI software or systems.’

Or as Ted Goia wrote in his Honest Broker substack:

AI can exist as a tool for humans, but not as their cheap and inferior terminator and replacement.

AI is relegated to a warm-up exercise for real writers. An expensive and legally risky one. Writers two, terminators nil.

In an adjacent courtroom drama, the Authors Guild is well on the way to securing a jury trial for its class action against OpenAI, Meta and Google, where the class is professional writers ‘whose works spring from their own minds and and their creative literary expression’. Elsewhere in the terminator franchise, however, Goia sees musicians losing ground to the machines. He lists the signs that streaming services are flooding playlists with AI-generated content and ‘fake’ artists, perhaps because musicians have failed to organise as effectively. Screen writers and actors may have won concessions, for now, but ‘the sound track sucks’.

Will academics negotiate their own Article 72 for academic content? Or will students find their sources flooded with auto-generated study notes, the learning equivalent of hold music? Right now, the film has several endings in development.

And that reminds me.

This week’s top download…

… is a real blockbuster from the EU’s Joint Research Centre: On the Futures of Technology in Education (hereafter the ‘JRC report’)

Section 4: AI and learning analytics is detailed and perceptive on the state of the generative AI economy. ‘Training foundation models requires compute power that only a few organizations can afford’, the authors remind us, so development is likely to be ‘consolidated in a small number of dominant designs’. They don’t name the organizations with enough chips to buy in to the game, but we know who they are. OpenAI (in lockstep with Microsoft and its AI-enhanced Office suite); Google, rapidly embedding Bard and its multimodal model Palm into all of its products; and Amazon, now working with Huggingface (provider of open source transformer libraries and developer tools) to embed AI into its web services. Meta is making up for its late entry to the game by releasing a large language model based on Llama2 that can ‘emulate human expressions’ and comes with 28 distinct chat personas to fill your metaverse/social media feed with personalised joy. Only Apple is hiding its hand.

Funny how the ‘next big thing’ looks a lot like the last one, dominated by the same platforms and corporations.

But what if language models could escape the grip of the big four/five? There are now thousands of lightweight models, like Alpaca, developed at Stanford University’s Human-Centred AI lab. This LLaMA version was retrained in just 3 hours ($100) on prompt-output pairs generated from OpenAI use data ($500). Or the even neater Vicuna, trained only on user -shared conversations (so not scraped from anyone’s data warehouse: total $440). There are problems with small scale. The models are even more liable to misinformation, and the costs of generation are high compared with the costs of training. But these projects show it is possible to get a working language model up and running on consumer hardware for less than the cost of many e-textbooks.

As an alternative to ‘letting many flowers bloom’ in a landscape of open models and developer tools, what if large language models could be (re)built as a kind of public commons? Sweden is developing public sector language models for reasons of national sovereignty, for example, while EleutherAI offers a suite of models (Pythia) for open research, including research into explainability and transparency in generative AI. BLOOM is an open access model built on GPT2 by a 1000-strong research consortium, BigScience. The consortium took almost a year to build a multilingual text data set - 46 natural languages, 13 coding languages, 1.6 terabytes of text - and trained the model with compute donated by the French government (via CNRS). Could something like BLOOM, the JRC report authors wonder, become ‘a shared platform and a technological artefact where the interests of many developers and users meet’? One where:

part of the costs could be shifted to governments if educational benefits are identified from developing foundation models (or, for example, training such models for small regional European languages).

For educational organisations, and national education sectors, these are game-changing possibilities. But they demand collaboration on the scale perhaps only the EU can dream of. I have a long post about the model-as-commons idea, what is stacked against it, and why I think the education sector should invest in it anyway. Meanwhile, I have only one issue with the excellent JRC report: it doesn’t always join the dots between its hard-headed analysis of the big picture and its report of educational benefits to learners.

For example - looping back to the musical theme - the report compares AI writing tools to the synthesisers that revolutionised music production in the 1980s. It’s a neat analogy, but I’m not sure it’s the only one. Synth players use a vocabulary of recorded sounds, but they use it consciously and inventively (I’ve added a playlist at the end). Like the electric guitar for an earlier generation, the synth made it easier to sound quite good, but it was still the player who made the sounds into music. And a sample or loop from another artist is a homage to the original, adding another layer to the music’s meaning.

Generated text is not a homage to previous writing but a badly pirated copy. In the wider economies of creative production, generative AI might be more analogous to music streaming than music synthesis, delivering content in ways that tend to concentrate rewards with a small number of big players, and can make it harder for new talents to rise.

Once more with Vygotsky on bass

All this is to say that the benefits of AI in learning should be seen in the context of the wider economies and meanings of text production. And this is a context that universities can help to shape. A university that is committed to open access models, refined with open source tools to meet the needs of different knowledge communities, with the needs of research and learning to the fore, can offer different ways of relating to generative AI than a university that has signed up to a package of services from Google or Amazon or Microsoft. The risks and opportunities are different. The roles of learners, researchers and teachers within the two techno-social systems are different. Different capabilities will emerge.

The JRC report provides an invaluable overview of the economies of texts and models, the corporate AI industry and its alternatives. And this could be followed through in the analysis of educational opportunities and learning tasks. As the report says, generative AI can be used to summarise textbooks and academic articles, or to provide feedback on learners’ written work. But how do these new syntheses support different learners to become more capable writers, actors and thinkers? And shouldn’t these uses be seen as interconnected with the wider systems of copyright, copytheft, data privacy, surveillance and bias?

I was interested to find Vygotsky summoned as a witness for some of the claims of educational value:

From … a Vygotskian point of view, generative AI systems are not interesting because they produce text; instead, their relevance for education is in their capability to engage humans in advanced forms of thought where concepts, conceptual systems, and language are the tools for thought.

Vygotsky certainly regarded writing as a technology that mediates thought, like Ong and Macluhan later in the 20th century. For all these theorists, ‘forms of thought’ are shaped by tools for expressing and communicating them, such as the pen, printing press and camera, and (if they had lived to see them) the synthesiser and the computer. But for Vygotsky, thought is also inherently interactional, emerging through activities that involve other people, and cultural resources in the social environment (resources that Engestrom later codified as ‘rules, norms, roles, and divisions of labour’). Vygotsky did not locate capability in the minds of learners, or in their tools, but on the plane of inter-personal, tool-mediated activity.

I think Vygotsky would have been intensely interested in how generative AI is reshaping the rules, roles and divisions of labour involved in the production of text. But I suspect he would not have regarded the ‘forms of thought’ mediated by LLMs as ‘advanced’ just because the technology is complex. He would have asked, I imagine, what activities learners are engaged in, mediated by these new tools. What are they asked to do, and why? What relationships with other people are they entering into, including with people they will never meet, and what collective forms of knowledge production are they joining in with? And this opens naturally into questions - that certainly animated Vygotsky and his school of educational pscyhology - about difference, conflict, and power.

Outro/Coda

This has been a multimodal digression, and appropriately enough it comes to a close just as OpenAI announces new voice and image capabilities for ChatGPT. ‘Tried and tested’ prompts are already being offered to students and teachers for these new functions. They can’t have been tried in any real situation of learning, as yet, and some of them don’t even seem to have been tested by the people promoting them. ‘Two truths and a lie’, for example, is a terrible ChatGPT prompt, as anyone who has played with the bot for five minutes could tell you. ‘Getting ChatGPT to play the part of a patient’ is the #1 way to use it with medical students, apparently. Good luck with that one too.

As the JRC report says:

the importance of effective domain-specific prompt design has quickly been noted, and there is now a rapidly increasing group of people claiming to be specialists in this area. [However]… as the underlying language models are continuously changing, it is not clear that deep expertise can emerge in prompt engineering on these platforms.

In education, when the point is not to produce text but to learn from the process, the value of prescribed prompts seems particularly doubtful. What do learners have to gain from cutting and pasting text into a chatbot window, just to read the text that comes out? They might as well engage with… text, perhaps? Quality educational content? A search of wikipedia? Or they could sit back and wait for the direct-to-brain capability that Elon Musk is working on in his Neuralink labs, and give up on the tedious business of reading and writing altogether.

(No links to Neuralink from me - and if you plan on searching, a content warning about images of animal testing that you may find upsetting).

Thanks for reading, and here’s a synth-tastic generative AI playlist to take you through the weekend.

Tubeway Army: Are friends electric?
Kraftwork: The Model
Missing Persons: Words
Yazoo: I before E except after C [enjoy Alison Moyet reading part of a synthesiser manual]
New Order: Round and Round
Electronic: Disappointed

My discussion with Tim Fawns is now available as part of his series ‘ten minute chats on generative AI’. Tim is a wonderful advocate for deep thinking about practice with tech, and has a knack for provoking people into saying interesting things, so I’m honoured to be in this company.

I’ve also enjoyed Tim’s blog post on ‘expanding the unit of analysis’ this week. Since learning always involves some relation to cultural artefacts – not only data and algorithms, but texts and tools of much older kinds - he argues for a way of thinking about pedagogy that puts these materials at the centre.  

It does seem that everyone is a sociomaterialist now. I’m still wondering what kind of sociomaterialist I am, since I wasn’t a fan of the ‘discursive turn’ back in the 1990s, and if you just keep wearing your old materialism until it comes back into fashion, are you ahead of the curve or (as my daughter would say of my literal clothes) ‘embarrassing’?

Tim points to a 2015 article by Tara Fenwick that provides a helpful run-down of different sociomaterial styles. I’ve always found ‘cultural historical activity theory’ (CHAT) attractive, because it tries to untangle the structures and agencies, entities and relations that other sociomaterialists (perhaps wisely) leave entangled. So I’m pleased to see CHAT on Fenwick’s list, though it is a bit of an afterthought. I may I need to spend a bit of time rummaging in the theory cupboard to see if I can produce something more coherent.

I have a couple more interviews coming up on other people’s podcasts, and a short series of conversations with a colleague who has a different take on generative AI to my own. So look out for more announcements soon.

Back to school with Microsoft and OpenAI

Meanwhile the generative AI business model is emerging more clearly, just in time for the new academic year. With partners Microsoft, OpenAI owns the front-running foundation models GPT-4 and DALL-E2, and they have already been embedded into thousands of platforms and business enterprise systems. As well as stumping up the initial investment capital and providing the raw compute for training and generation, Microsoft is leading the business case for gen-AI-with-everything. Using Copilot, Microsoft has integrated GPT-4 into its entire Office suite - Word, Excel, Powerpoint, Outlook and Teams. Even at a higher price point than anyone expected ($30 per user per month) business users are falling over themselves for these new features.

If your organisation can afford a bespoke integration, there’s also ChatGPT Enterprise from OpenAI:

‘an AI assistant for work that helps with any task, is customized for your organization, and that protects your company data’.

For an undisclosed sum (‘contact sales’) your organisation can buy a custom-trained model and the reassurance that ‘we do not train on your business data or conversations, and our models don’t learn from your usage’. Which does at least make clear what is going on with everyone else’s data.

Microsoft and OpenAI might seem to be competing for business users here, but in fact they are just covering the ground, as their entangled finances suggest they should. ChatGPT+ is generating revenue from people who want an enhanced experience of generative AI, through their chosen interface. The MS Copilot features appeal to business users who want familiar tools with added auto-complete. And ChatGPT Enterprise allows organisations to maximise the value of their own data while keeping it from mixing with everybody else’s. In each case there is a path to profit, and the cost of firing up all that compute every time someone clicks on ‘chat’ or ‘compose’ or ‘create content with Copilot’ is no longer being sucked up by Microsoft.

So what do these developments mean for teachers and students as the new year begins? Generative tools are now available not only as free-standing apps and interfaces that we can choose to use (or not), but deeply embedded into the Microsoft and Google work suites, behind the scenes in search engines such as Bing and Edge, and integrated into education-specialist platforms. Plug-in ChatGPT APIs are available for LMS platforms such as Moodle, Canvas, edX and more, and for quizzing, revising and language learning apps. Turnitin has a controversial gen-AI-powered gen-AI-detection feature, while Anthropology is partnering with OpenAI to develop native tools for Blackboard Learn, and Google is busy integrating its own latest language model, PaLM2, into classroom applications.

On the positive side, having chat functions pop up in the dullest of places may suck some air out of the hype bubble. The excitement of the spring and summer – the ‘sparks’ of computer sentience, the ‘revolutionising’ of economic life  – may give way to an autumn of realising we have all the shortcuts we will ever need for writing excel macros, and search doesn’t work as well as it used to. Similarities with Microsoft’s famously annoying ‘Clippy’ assistant are hard to avoid: Clippy even has a GPT-powered tribute app.

On the other hand – and more seriously – with integration it may just get harder to notice what is happening, both to our practices of writing and thinking at university, and to the wider knowledge ecology, increasingly a limbo land of undead language and disembedded data.

Pop it in your pencil case

In AI literacy and Schroedinger’s Ethics I noted that the Russell Group Principles devolve a lot of responsibility to teaching staff for dealing with these challenges. How wonderful, then, that subject specialist Dr Emily Nordmann has done exactly this, providing clear, detailed and principled guidance to students for different assignments in pscyhology. I wonder how many teaching staff have this kind of expertise and patience, but the results are here for other teachers to use, in and beyond psychology.

A crowd sourced book, 101 Creative Ideas to use AI in Education, edited by Chrissi Nerantzi and colleagues, provides a general collection of ideas that is open to the possibilities of generative AI but always leads with the values of responsible, ethical and critical use.

Lydia Arnold’s assessment top trumps have been reimagined for an ‘AI enabled world’ and are available to download from Jisc.

I have mentioned before that the policy statement from the University of Edinburgh seems particularly clear and supportive. There is a more detailed and equally well considered guide for students from the Library at UCL.

Finally, I often go back to the guidelines for writers produced by Nature group of academic journals back in January and explored in my very first stubstack post on language, language models and writing. They follow two basic principles:

1. Generative AI tools are not authors and should not be cited or credited as such 2. Use of such tools as part of a research (or writing) method should be reported in that context (i.e. methodologically)

A happy new academic year from me, and check back for new interviews and post on:

• A radical history of ‘artificial intelligence’ and why educators should care

• Generative AI risks to the knowledge ecology

• Fully functional female gynoids revisited, and a feminist take on the AI imaginary