It’s marking season and my heart sinks as I turn to the all-boys group project that is the Government’s AI Action Plan. Despite the feedback I gave them earlier in the term about poor planning and a lack of research, they have ploughed on with the bravado that only ignorance and testosterone can supply. And it does seem to be in full supply.

Back then I suggested that tax breaks and deregulation for building data centres was not enough to procure ‘sovereignty’ in AI governance, and that ripping up laws protecting the UK’s creative industries - a real source of value and prestige - would not boost UK productivity. Similar feedback has now come from higher places. The UKAI trade body agrees that the Action Plan - written and led by major AI investor Matt Clifford, informed by the Larry-Ellison-funded Tony Blair Institute, and stewarded by Peter Kyle (with ChatGPT for policy advice) - does not strengthen the position of UK companies globally. Nor does it, according to the wonderful Beeban Kidron, protect UK citizens from the predatory intentions of global AI corporations.

The big bet – that AI can produce ‘efficiencies’ in public services while cutting costs – is looking worse and worse. The Netherlands is four or five years ahead of the UK in developing public sector AI, and (unlike the UK) has put ‘responsibility’, ‘fairness’ and public accountability at the heart of its project. But after a series of AI related scandals, including tens of thousands of families accused of benefits fraud, flagship projects have been dropped. Research from the Ada Lovelace Institute involving 16000 members of the UK public found low levels of trust in AI for services impacting their lives, and that poor governance has ‘the potential to damage people’s trust and comfort in the use of technology in the public sector, and even to undermine the perceived legitimacy of services and of the behaviour of government itself’.

But Starmer’s Government is not interested in what the public thinks of its behaviour or in learning from actual public sector projects. It is running with the big boys’ gang. Last year the CEO of Microsoft UK was appointed by Starmer to lead his Industrial Strategy and within a week the Government had signed a 5 year deal with MS, locking it into Microsoft cloud services. A couple of months later the Chair of the Competitions and Marketing Authority (CMA) was forced out by ministers and replaced by the former CEO of Amazon UK, and 100 staff were sacked. Odd timing, just the new ‘Digital Markets, Competition and Consumers’ act was about to come into effect and in the middle of CMA investigations into the conduct of Microsoft, Apple, AWS and Google. You’d think they’d need all the staff they could get. Luckily, the new Industrial Strategy (January) included a major ‘reset’ for the CMA requiring it to be resolutely ‘pro-growth’ and ‘pro-investment’ and to interpret its brief of defending UK consumers ‘flexibly and proportionately’. So the former CEO of MS in the UK is advising the former CEO of Google in the UK not to regulate big tech in the UK, and this is what we call the public interest.

According to its two latest Memoranda of Understanding - a cute name for a declining world power signing its assets over to the new ones - Google will upskill public sector workers with ‘a new Google Cloud training programme’, while moving public sector systems over to ‘Google Cloud technology’. And OpenAI will ‘develop sovereign solutions to the UK’s hardest problems, including in areas such as justice, defence and security, and education technology’.

Remember these priorities: justice, defence, security, and education technology. I feel sure we will be seeing them again.

TL:DR

• The dubious sovereignty of ‘sovereign compute’

  There is no ‘sovereignty’ to be had from getting as close as possible to big US tech. There are other ways of developing public infrastructure in the public interest, if they are allocated and governed accountably.

• Supercomputers and startups
  Hyperscale data centres are not ‘supercomputers’ and there is no real evidence that they support local prosperity. In fact, around the world, communities are organising to resist them.

• Jobs, jobs, jobs, and Amodei’s dream
  I look at the evidence for - and against - a jobs bonanza from ‘AI’ in the UK, and interpret Amodei’s dream

• Skills on a shoestring
  ‘Mainling AI into the veins’ of education for £8 a shot, and the first of several appearances for the UK Spy Agency

• Homework marking - again
  The magic marker-bot and the highly educational careers of Michael Gove and Dominic Cummings

• Who should be safe?
  The militarisation of the nation state, spooks again, and why AI security and AI safety are not the same

The dubious sovereignty of ‘sovereign compute’

Data centres are at the heart of the Action Plan for AI. Removing planning constraints and clamping down on protests will make it easier for US companies to park their polluting, power-grabbing, water-guzzling data centres on UK soil. When I compared this with a former policy of parking US cruise missiles here I thought it might have been an analogy too far, but it has since emerged that as well as data centres the UK will host US ‘tactical’ nuclear missiles at RAF bases for the first time in 20 years, at a time when hosting US missiles is working out really quite poorly for other countries that happen to be closer than the US to its various theatres of foreign intervention. Last week’s fanciful metaphor is this week’s headline, and the only thing more depressing is the failure of proper paid journalists to look into it any further.

Even if you doubt Ed Zitron’s figures, there is no question that a huge swathe of the US economy is currently floating on Nvidia chips and nothing else. Nvidia is where the froth and promise of AI turns into material infrastructure: chips and cloud (i.e. data centres) are the only parts of the AI stack that are seeing any profits. All the big five are losing on AI and would be losing bigger if they did not have access to compute at privileged prices (these companies, with Nvidia, make up over 30% of the US stock market - and growth is down for all of them). Now, helping the US economy to stay afloat on its AI bubble may be considered good economic sense, since when the sky falls in on the US economy everyone suffers. But that’s what buying Nvidia chips is doing.

The UK’s first ‘Sovereign AI Forum’ hosted by resolute nationalists (that is, US nationalists) Nvidia, claimed ‘founding members’ Babcock, BAE Systems, BT, National Grid and Standard Chartered.

The first two of these are weapons manufacturers, reinforcing the impression that when the Starmer government says ‘public’ and ‘AI’ it is really thinking militarisation and national security. A trawl of all five UK business web sites finds no mention of the ‘sovereign compute’ initiative, and the promise of £1billion in public spending, comes with no details of when and where. The new partnership with OpenAI (July 2024) – presumably part of the same initiative - is similarly short on specifics.

However, we do know a bit about Nvidia’s plans for the UK

I’m not sure Jensen Huang was fully briefed on the Goldilocks story, but the time is always ‘just right’ to buy Nvidia. The company that by some estimates has 98% of the global market in AI GPUs, and provides the only AI-specialist hyperscale solutions, is certainly desperate for more places to build data centres. Local communities are resisting them across the US and in South America, and attempts by the Trump administration to prevent states enacting their own guardrails have temporarily stalled. Building data centres is hugely complex and expensive: even the US government/OpenAI $500billion Stargate project has been scaled back to building just one in its first year.

And so Huang has been traveling the world talking up the need for governments to invest in ‘national’ or ‘sovereign’ compute, since few other buyers can take on the kind of debt involved. Hyperscalers or ‘bit barns’ make extrarordinary demands on power and water supplies, as well as network infrastructure, and these must be uninterrupted, even if that means depriving local populations of resources. So bit barns can only be built where local, regional and national governments are on board, tax regimes and planning laws have been made favourable, infrastructure is in place or can rapidly be improved, and suppliers of power, water and bandwidth are all signed up to the new priorities. (Just wondering: has anyone looked into the state of the UK water industry recently?). Hyperscale certainly demands sovereignty - control over planning, services and infrastructure, and the policing of local communities. But it’s not so clear what sovereignty it is giving back.

The macho and frankly imperialist demand for the UK to be an ‘AI maker, not a taker’ ignores that supply chain for compute is complex and globalised, with most of it firmly anchored in the US (or China). While Nscale seems to be the front face of UK ‘sovereignty’, the bit barn boom is actually being driven by four massive US hyperscalers, CyrusOne, CloudHQ, ServiceNow and CoreWeave. NScale, the company that is supposed to supply the first tranche of ‘sovereignty’, only launched in May last year. It announced in January that it had secured a site in Loughton, Essex, ‘that can house up to 45,000 of the latest Nvidia GB200 GPUs’, and will go live with 50MW of capacity by the end of 2026. You’ll notice that NVidia is only allocating Nscale 10,000 GPUs and not the 25k that would be needed to meet this capacity. (If further proof is needed that Nvidia is the real sovereign here, how many industries can ‘allocate’ their product based on their control of supply? Oil and gas maybe?).

In fact Huang seems to regard the UK as a very minor courtier, skipping off smartly from the UK launch to Paris where France is allocated more Nvidia GPUs in the next year than the UK’s infrastructure project overall. But then, France is investing 100 times as much public money in compute (109 billion euros). Because the only serious alternative to the US and/or China AI stack is the one Europe is (sort of) trying to build, at least on common market, common regulatory terms. And if the UK were willing to throw some weight behind that effort it might be a little more possible. But for ‘sovereign’ investment, nothing Europe can invest can match what is being put up by the gulf monarchies, the ones that pack the real ‘sovereignty’ when it comes to wealth and hardware. UAE has just brought 100k Nvidia chips online and plans to import 500k every year, while Saudi has struck deals worth $600bn with Nvidia, AMD and AWS. These are the market ‘makers’ that the US takes seriously as partners.

Perhaps to discourage close reading of the AI geopolitical power map, the definition of ‘sovereignty’ in relation to UK data centres is vague. And so we must look to Nvidia for help:

Here’s my reading.

1. This is Nvidia’s version of OpenAI/Stargate’s controversial ‘AI for Countries’ initiative, ‘supporting’ nations to build AI infrastructure that will remain under US control. In return for chips, countries must ‘invest in expanding the global Stargate Project—and thus in continued US-led AI leadership’. OpenAI unblushingly uses the cultural biases and misalignment of its models as a reason for countries to plug in their own cultural data. In return for being part of the ‘AI for democracy’ empire, countries will hand over control of critical national infrastructure, vaults of cultural and biometric data, their most promising researchers and a chunk of sovereign wealth. Our friends the Tony Blair Institute were out in front at the AI for Countries launch, ready with their own blueprint for sovereign AI: ‘countries that have the strongest relationships with large technology providers will be best placed to provide effective public and private computing infrastructure’. And that is democracy. Talking of strong relationships, did I mention that the new head of public policy for Nvidia in the UK is fresh from directing the TBI?

2. An interesting list of key players, but whether the ‘developers, researchers, scientists’ who are actually building AI are doing so for public projects or private profit depends entirely how ‘AI investors’ are constrained by the ‘government’ to allocate resources. At the moment all the power, compute and data are on the first side of that relationship. How could ‘government ministries’ take a more engaged role than waving little union flags from the ‘supportive’ sidelines? There’s plenty they might try. The UN has a well established list of principles for the development and risk-management of Digital Public Infrastructure, and there are serious attempts in Europe to articulate - beyond regulation - what public interest compute could look like (though, whoops, again the UK seems to have opted out.)

3. Given the brain drain, and the need AI has for ‘basically literate’ workers in its data engines, training is a public cost and an AI industry benefit. More on skills in the next section.

4. Once built, the infrastructure will be owned by the hyperscale company and their creditors. How will access for UK users be allocated, and on what terms? Unless this question can be answered with real savings for publicly beneficial projects, most users - who do not have fibre optic connections to a specialised supercomputer in their back garden - would be better served by having better broadband infrastructure (the UK ranks 43^rd in the world) so they could access cloud services wherever they happen to be located. ‘Local laws’ could tip the balance of infrastructure development towards the public interest, but the AI industry is leaning heavily on governments to deregulate as the price of compute, and as we have seen, the UK Government has shown itself exquisitely sensitive to such pressure.

5. AI factories is how Nvidia chooses to characterise its vertically integrated stack, extending its monopoly power all the way from GPUs down into minerals and rare earths, and up into local energy plants and water supplies. Comparing the scale of ambition with the development of the factory system is honest at least. As I have always argued, AI is not a ‘tool’ it is a system for restructuring labour, and it’s increasingly obvious that means restructuring the whole of social and economic and planetary life as well. Perhaps Jensen thinks the factory metaphor will go down well in the birthplace of the dark satanic mill, it’s just that historically the UK ruling class has been building the factories and exporting their ‘externalities’ to the rest of the world, and not the other way around.

So how could the UK Government secure capacity that is owned, used and governed in the public interest? First, recognise that very idea of ‘sovereignty’ in compute is a chimera. At least European nations understand they can only do this as a bloc. The brilliant Cecila Rikap argues that South America has a chance if its governments can work together, but that it is slipping away. For nations without the UK’s post-imperial resources, or power bloc solidarity, the choices are even more stark.

Second, recognise that the interests of democracy may not be served by ramping up the data and surveillance capacities of the nation state. Not only are nakedly autocratic parties increasingly able to seize state capacities - something the UK is far from immune to - but the tech giants that provide these capacities are actively fuelling antidemocratic forces, as they increasingly find democracy an annoying drag on their ambitions.

Instead of a spurious ‘sovereignty’, suppose the aim was to procure and secure a digital public infrastructure, composed of material elements (cloud compute, networks, data) and governance arrangements that would prioritise the needs of diverse interest groups. These might be local, regional, sectoral (voluntary, third sector, health, education, research), but the priority would be the participation of those least served and most at risk from the current commercial infrastructures of AI. If I could set some homework for Peter Kyle, it would be these two recent papers, Digital Public Infrastructure and public value (Muzzacato et al. 2024) and the White Paper on Digital Sovereignty, (Rikap et al. 2024) both from international research teams working in the UK. Muzzacato’s Common Good Framework offers five principles for securing public agency in relation to infrastructure:

• purpose and directionality;

• co-creation and participation;

• collective learning and knowledge-sharing;

• access for all and reward-sharing;

• transparency and accountability.

And the White paper suggests how these might be applied in specific policy proposals for public compute. (This is my paraphrase of the original – but please go and read it!)

1. A democratically accountable, publicly-governed digital stack, comprising digital infrastructure as a service, platforms, a public marketplace without lock-ins, and state procurement preferentially from this marketplace.

2. A research agenda focused on digital developments that could solve collective problems and enhance human capacities, with public research agencies funded to counterbalance the expertise in commercial labs.

3. ‘Ecological internationalism as the basis for national digital sovereignty’, minimising the resources needed and the environmental impact (we could add ‘frugal compute’ or system-wide sustainability as a sixth principle to Muzzacato’s).

4. Dismantle state surveillance and regulate against the ‘misappropriation of collective solutions by specific governments’.

Whatever the future of AI, there is a need for public control of public infrastructure. But this isn’t it.

Supercomputers and start-ups

This whole deal with the devil is supposed to be in the interests of economic growth. Remember, Huang has said that ‘supercomputers in the UK will naturally attract more start-ups’.

So there are two things to point out here. First, supercomputers sound, well, super, but they are not the same as data centres. Supercomputing infrastructure is highly integrated on a single site for specialised data operations, such as advanced research. Data centres are racks of GPUs connected to the cloud for whoever can afford to burn them. While some of the networking technologies are converging, the deployment models and network requirements are quite different.

There is a strand of the Action Plan that does concern supercomputing. The UK AI Research Resource promises to multiply available research computing for AI ‘by twenty times’. So far the £750million announced for a new supercomputer at the University of Edinburgh does not quite replace what the previous government had already allocated to this project and that the Starmer government put on hold when it took office. So this is not a promising start.

Whatever the finances available, the AI Research Resource could set a precedent for public oversight over the allocation of supercomputing resources and the projects that run on them. The Ada Lovelace team (again) suggests that this could mean ‘board seats and mandatory public engagement exercises’ for those most impacted by AI. Priority could go to those researchers and research topics that do not have access to corporate funding, which is likely to mean topics that do not centre the interests of the AI industry (sustainability, safety, risks and harms for example). We must wait to see whether ‘breaking the mold of traditional academic-led software projects, and ‘a stronger role for industry’ will indeed involve public representation and public interest projects, but it isn’t a great sign that DIST is effectively asking the data centre industry to tell them how to build it.

All that being said, the proportion of sovereign compute allocated to supercomputing is small. So can data centres also ‘attract’ innovation and prosperity, as Huang claims? Sunny predictions on this issue spilled from a Public First report that was shared by Nvidia at the launch of the AI infrastructure strategy, luckily being published on exactly the day the public might be asking questions:

The report does not link to any actual research, only back to the Nvidia press release about it, but it does include some tantalising details of PF’s methodology. Skip ahead if you can’t be bothered with the working out.

THE WORKING-OUT So the key figure in PF’s calculation of added value is something they call the ‘Computer Access Metric’, calculated using an impressive sounding ‘Gravity model approach considering data centre capacity and inverse square distance’. I contacted an expert in planning who told me that the gravity model is indeed used to calculate access to services such as healthcare, where capacity and distance obviously matter. But why does distance matter when it comes to accessing cloud compute? The speed and quality of networks are what really count here. And if PF did include some measure of network quality, either in ‘capacity’ or in its measure of ‘distance’, it’s not clear how network quality translates into local economic benefits - not unless all your users have their own fibre optic link-up. The expert told me that the methodology was, at best, unclear.

There is one application of computing, however, where distance matters at a local level, and that is high frequency/low latency stock trading. In the finance sector, miliseconds make the difference between gaining and losing a trade, which is why trading platforms are typically integrated with data servers on site, using fibre optic cables and microwaves to achieve ‘ultra-low latency’ in transmission. It’s also why so many UK data centres are in the City of London.

Surely it can’t be that the figure PF has used for the local benefits of data centre construction includes all those data centres in the heart of the UK’s financial services sector, one of the country’s very few growth sectors, and one of the very few places where co-location of data centres with operations really does translate into economic benefits?

THE ANSWER (did you skip here?) The benefits of a data centre nextdoor to a financial trading platform in the City – or indeed a supercomputer cluster in a research park in Oxford or Edinburgh - are unlikely to be replicated near a bit barn in Loughton, or scaled up from there to the UK as a giant bit barn farm in the North Atlantic. So let’s hope this is not how the calculation was done.

Because trashing the UK’s climate targets and planning protections really does require some benefits on the other side of the scale. Given the huge impact of data centres on the National Grid, scarce water resources and natural environments, their minimal job creation, and the rising tide of protest against them, surely they should only be built after extensive auditing of the environmental impacts, where local communities agree that the benefits outweigh the costs, and within a democratic framework that has public service and public science contractually locked in.

Instead, public representatives who try to block data centres are kicked out of the way, and members of the public who want a say over what gets built with public resources are more likely to be criminalised than invited to join the board.

Jobs, jobs, jobs, and Amodei’s dream

So you’ve heard me say elsewhere that there are no signs as yet of the promised productivity bonanza from generative AI. Workers are being asked to work faster, do more with less, accept deskilling and casualisation, to use AI rather than enlist the help of a junior colleague, but it’s not clear that any of this is improving the quality or the productivity of work.

But how about in the AI industry itself? Isn’t there a jobs boom there to be ‘unleashed’?

CloudHQ’s £2 billion hyperscale campus in Didcot, Oxfordshire promises 100 permanent jobs. The data centre at Loughton promises 250, though it’s worth noting that such projections have proved wildly optimistic in past projects. Even if these numbers are in the right ballpark, that’s not a lot of jobs. In general, tech companies tend to have few employees per unit of capital, and data centres are among the most capital intensive of all tech projects. They are called ‘bit barns’ for a reason. If you own a data centre you aren’t exploiting workers you’re extracting value from a pile of computer chips running night and day. Anthropic’s CEO, Dario Amodei, shared a dream of ‘$1bn companies with just one employee’.

The most recent (2023) report on the UK AI sector found there were 60,000 UK jobs in AI, generating £14 billion in revenues, of which £1.2billion was ‘gross value added’ (GVA = overall value to the UK economy). To put this in perspective, in 2023 there were 60 times more people working in education, and the creative industries generated over 100 times the GVA. Looking closer, less than a third of those revenues came from ‘dedicated AI companies’. How AI is defined in both company designations and in job roles within ‘diversified’ companies depends on information from commercial databases, that often draw from company web sites. So the ‘AI industry’ is essentially marking its own homework on even the most basic issue of what it actually is.

I fully anticipate news that the AI industry has grown ‘exponentially’ since this report was produced, as hundreds of companies re-designate what they do in order to cash in on the hype and investment opportunities.

Talking of investment, ‘negative GVA’ means that companies are taking more out of the economy – especially as inward investment – than they are producing as goods and services. We know that since 2023 there has been a massive investment boom in companies that can claim to be developing services in some part of the AI stack. We know that most of this investment is not actually producing revenues from goods and services, but is being churned around the speculative economy, where mergers and acquisitions (big companies buying up small companies) can provide financial returns to investors. What this gloomy FT assessment in June really means…

… is that the proportion of UK AI companies that are in ‘negative GVA’ is likely to be much higher now than in 2023. Indeed, from these figures it seems plausible that the whole UK AI industry is currently a value sink rather than a value producer for the UK economy.

Now, much of the revenue and GVA in 2023 came from a few big, international AI corporations. The ten big AI players, accounting for nearly two-thirds of the GVA, provided only 10% of AI employment. Because basically they are big international corporations with small UK offices. Google (plus Google Deepmind), Microsoft and Meta feature large among them. Databricks is a Silicon Valley mega-corporation that is trying to secure a monopoly of AI data services by buying up everything that moves.

The offices of large international tech companies are typically managed to reduce that corporation’s tax burden and to benefit from different regulatory regimes. So although AI companies generated over £14 billion in revenues in the UK 2023, 80% of that was generated by these large firms, and in the same year (2023) the UK government missed out on £2bn in lost tax as these same firms shifted their profits overseas. In other words, economic value and tax revenues that do arise from AI in the UK may not stay here.

There are – or were - at least some UK companies among the big players. Not long after this graph was generated, Oxford biotech company Exscientia was acquired by US corporate Recursion (HQ Salt Lake City) who immediately sacked its chief executive and a quarter of its (small) workforce. Chip manufacturers ARM and GraphCore used to be British companies until both were acquired by SoftBank, banker to the Silicon Valley billionaires and founder/funder of the whole ‘national AI’ adventure. DeepMind, as I mentioned, was taken over by Google and has recently been merged entirely into GoogleBrain causing disquiet among its UK staff. Dyson is the UK company that famously moved its HQ to Singapore and axed 1000 UK jobs. That just leaves the mysterious Quadrature, that seems to be some kind of financial trading operation turning over nearly £600million with just 143 employees. Amodei’s dream is getting closer.

In fact, US takeovers of British AI start-ups have reached fever pitch since the launch of the AI Action plan, though on the day after this embarrassing headline appeared, the Government managed to announce a ‘raft of investments into the UK by leading tech companies’ to counteract the impression that the UK is just their offshore R&D lab. Examining the list, it is almost entirely made up of international consultancy firms, investors and fintech companies shifting their office locations, sometimes even within the UK (e.g. Capgemini and Netcompany), and almost certainly for tax and regulatory reasons. Only 180 new jobs are actually promised here.

This whole sorry picture can be summed up in the story of ScaleAI. With much fanfare from the last Government, Scale AI opened its European HQ in London a year ago, creating less than 50 jobs of which an undisclosed number were US employees relocating. ScaleAI is a datawork company, notorious for its exploitative labour practices and for providing the data work for ThunderForge, the US military’s ‘AI-powered, data-driven warfare’ engine. The UK was graced with its presence because London gave access to the European market for data labelling without the downsides of what seemed likely to be a tough EU regulatory environment. CEO Alexandr Wang was endearingly keen to ‘support’ the UK Government to avoid the EU’s approach:

Plus la change, plus la meme chose, Wang visited Starmer on 12 February this year to discuss regulation with the new Government. By coincidence, on 13 February Starmer announced that the UK would be ‘going its own way’ on AI regulation (that is, not aligning with the EU’s pending AI Act), and on 14^th he announced a further shift of policy:

ScaleAI was in fact involved in safety security testing for the US AISI until it was abolished as ‘too woke’, leaving the UK AISI high and dry. But all of this quickly ceased to be Wang’s problem as his company was acquired by Meta in June and Wang moved to head up Meta’s ‘Superintelligence’ lab. ScaleAI immediately shed 200 of its already meagre workforce because ‘we ramped up our GenAi capacity too quickly’, leaving the company worth $14bn (that’s how value gets added, you see) but with just 700 employees worldwide. Amodei’s dream gets closer still.

I apologise for spending so long on a sector of the UK economy so small that a couple of acquisitions or office relocations can produce effects of ‘expansion’. But it matters if these effects are being used to suggest that the whole UK economy would be booming if only more of it was ‘AI’. Since the government seems to have commissioned no reports on the likely impact of AI on the 99.9% of jobs in the rest of the economy, we are going to have to extrapolate from a few indicators.

One, reported in the FT in January, is that as economic prospects worsen, employers are investing in AI rather than hiring staff. Even senior executives who have gone all-in on AI report only ‘modest’ returns (all these headlines are from the past three months) but reducing staff costs can make those modest gains look better.

A recent survey (modest sample and reliability) suggested UK businesses were spending on average £321k per year on AI last year, and if it’s anything like that figure, CEOs have to find ‘returns’ from somewhere. Whatever they are currently spending is being subsidised by the AI industry itself as they try to restructure workplaces to create an enduring market, a situation that won’t last forever as some day even AI must make an honest profit. Subverting a well-known bon mot, this is the cheapest AI you will ever use.

So AI does not have to enhance anything for investments in AI to create downward pressure on skilled employment. We can see signs of this happening already in the graduate job market.

For an idea of how this might play out, I have had to turn to research in other countries. The Social Policy Group in Australia has produced a comprehensive and credible report with both micro and macro economic modelling. Two caveats. The first caveat is that the Australian economy is different though comparable to ours in the UK. The other caveat is that I do not buy the high productivity and efficiency gains that SPG take as their baseline assumption. But, as I have argued, these gains don’t have to materialise for jobs to be deskilled and displaced. And anyway, the UK Government believes in them, so it should be modelling how its own assumptions play out.

The main effect SPG predict on a complex economy with a large service sector (like ours in the UK) is that AI ‘reduces labour costs without creating equivalent demand for increased productivity… leading primarily to layoffs rather than economic growth’ and ‘a cascade of job displacement and casualisation’ (page 9). With a larger tech sector, the UK has more reason to invest in AI-driven productivity than Australia. But the UK is carrying other disadvantages such as a huge over-reliance on financial services and a worse balance of trade. The 32% job displacement by 2030 that occurs in the SPG’s ‘high impact’ scenario may seem far-fetched, but since the UK Government believes some far-fetched productivity gains from AI, it’s surely worth an Industrial Strategy at least considers the risk that, with a very large professional services sector and a much smaller tech sector, the transfer of investment from skilled people to (mainly) US technologies might produce a lot of middle class unemployment.

In a retail and service economy like the UK, dependent on consumer spending and consumer credit, middle class unemployment produces a spiral of negative effects.

The one bright spot in the SPG report is that with 20% of Australians working in the public sector, the Government can stabilise the situation by ‘re-skill[ing] workers to fill critical vacancies in aged care, childcare and disability services’ (page 3) and by investing in quality jobs in health and education. The public sector plays a similar role in the UK, and with a push to become self-sufficient in sustainable food production and green energy, there is a real opportunity to boost the foundational economy too. Not to mention the creative sectors and universities, where the UK delivers real international value. With the impacts of AI disruption likely to be felt most in ‘unregulated, high-capital sectors’, the regulated, high-labour sectors that actually contribute to quality of life could be expanded. Some UK Trade Unions are coming up with proposals for safeguarding jobs in these sectors from the disruptions of AI.

But in the very sectors that could stabilise the UK economy, the Government seems determined to make every worker an AI worker, every job an AI job.

Skills on a shoestring

To secure all those AI jobs, the Government has announced £187m to be spent on upskilling young people.

‘The flagship strand of this programme “TechYouth” – backed by £24 million of government funding - will give 1 million students over three years across every secondary school in the UK the chance to learn about technology and gain access to new skills training and career opportunities.

That is £8 per secondary student per year, or £36 per secondary teacher (FTE), to ‘embed AI throughout our entire education system’. But giving money to schools is not the plan. Rather, TechYouth/TechFirst will be a scaled-up version of CyberFirst, a modestly successful programme to encourage young people into cybersecurity, run by the UK spy agency (GCHQ). Within this same pot of funding there will also, apparently, be scope for an online platform like CyberExplorers, currently provided by QA, an AI and cyber-security company that specialises in Microsoft Co-Pilot training.

Now, it isn’t immediately obvious that spooks, security wonks and CoPilot trainers can provide the critical approaches to AI that the UN considers necessary for young people. Are the folks at GCHQ well known for their ‘embodied ethics’ and ‘human accountability’, for their ‘safe and responsible use’ of cutting edge technologies? I must look it up. An open letter from an embarrassment of experts (yes, that is the collective noun when their expertise is not being listened to) recently begged the Government to invest in ‘an informed and engaged citizenry’, working with ‘civil society and academic partners’ in their approach to AI skills. But at £8 a head, the model is surely more likely to be QA’s franchise with MicroSoft, or indeed the Government’s own partnerships with Google and AWS, that offer corporate training materials in return for lock-ins to a particular brand of ‘AI’ use.

All of that still only gets us to £24million. How will the rest of the £187million public funding be spent? The vast majority (£145m) is going to student scholarships in AI and related fields at a small number of ‘leading’ UK universities. Whether this comes on top of HE funding already committed or whether it comes directly or indirectly from other allocations, the beneficiaries are not rushing to ask. But it certainly comes on top of vast investments by AI companies themselves in PhDs, student scholarships, Chairs, partnerships, prizes and other inducements, most of them to the same elite slice of UK HE. For a decade now, commentators have demanded more public funding for AI research so that ‘the future of this powerful technology is not dominated by private interests’, and the AI brain drain can be slowed. Funding on this kind of scale could make a real difference. Except, once again, private interests seem be driving the bus. PolyAI, Quantexa, CausaLens, Flok, Beamery, Darktrace and (of course) Faculty have all ‘signed up’ to the studentship scheme, and it’s not clear whether they are signing to put money in or just to take brains out.

None of this seems likely to help teachers to adapt as AI-for-everything rips down the year groups towards pre-school. Nor does it seem likely to support education leaders as they navigate AI policy and procurement, with established tech platforms, a shoal of new AI consultants and now the Department for Education all shoving them towards unfettered adoption.

As it turns out, the distribution of funding for AI skills maps very neatly onto the division of labour in the AI stack, with a jot of valued talent at the top and millions of AI-ready data workers at the bottom, and everyone in between scrabbling to do more with less. And in that sense, perhaps AI skills for the start-up nation will do exactly what it says on the tin.

Homework marking - again

At least the Government has one homegrown AI company that it is keen to support. I have already drawn attention to two shockingly bad reports from FacultyAI regarding their attempts to generate feedback for a few synthesised examples of student work. In marking Faculty’s own homework, I assumed it was so bad because the reports were simply a footnote to their shoo-in bid to manage the country’s educational data. But that has not stopped the Department for Education’s Policy on Generative AI for schools being pegged on them.

Peter Kyle - who let’s recall had a Faculty employee working on AI policy, in his office, for the whole of last year - is continuing to claim that Faculty has a homework-marking AI tool when in truth it does not have even a minimum viable model. Faculty has marked nobody’s homework: it hasn’t even tried to. Rather, Faculty is allowing other developers to access to the educational data it is now stewarding so that they can take on the costs and risks of maybe one day producing the markerbot that teachers need so badly. Because as AI-adjacent UK thinktanks keep reminding us (I give you this report from Google/Public First as well as the ones we know and love from Faculty) teachers do too much marking when they could be teaching. Silly teachers.

Now, I’d say it’s up to teachers to know what kind of feedback is meaningful for their students’ learning and what kind is just ‘routine admin’, but we can still wonder why so much teaching work in the UK can be categorised in this way, creating the perfect opportunity for ‘AI’ to appear as a solution. It was Michael Gove, alongside special advisor Dominic Cummings, who drove the education reforms in 2010-2014 that resulted in all the standardised tests, the subject-by-subject documentation, lesson-by-lesson planning and weekly report writing that has found its way into Faculty AI’s data pool. For ordinary teachers these reforms meant a doubling of marking load in three years: a treadmill of test and mock-text marking alongside regular assessments of student ‘progress’ to support school comparison tables (an initiative that was also used to drive competition and privatisation). At the time nine out of ten teachers felt they had less time to teach and support students, and many left the profession rather than becoming marker-bots themselves.

The same pair were behind the National Pupil Database being opened up to commercial users in 2012 to ‘maximise the value of this rich dataset’, with consequences that were reported in a study titled ‘What happens when public data is shared with the wrong people’? (The authors of this report, DigitalDefendMe, have recently asked whether the inclusion of ‘pupil data’ in the latest Faculty contract means that Faculty will be have access to this sensitive dataset). Could it really be the same Michael Gove, in a different role entirely, who launched the new, low-scrutiny, ‘preferred supplier’ system for government AI contracts that proved so controversial during the Covid years? The system under which Faculty AI was the only contender for the contract to run safety testing for the UK AI Safety Security Agency? I do believe it was.

We may never know how this illustrious public career, so deeply entangled with education and educational data, has inspired Cummings’ many musings on AI. But here are some things we do know. We know that Public First (remember them from the data centre report?) was set up by two Gove and Cummings associates, who worked with them on the Vote Leave campaign, and were then inspired to set up an astroturf outfit for taking schools out of public governance. We know that Faculty CEO Mark Warner was deeply involved in both the Vote Leave campaign and Cummings’ handling of the Covid19 response when he was (briefly) inside Number 10. We know that it was Cummings who set up the Advanced Research and Invention Agency, and led it as a British version of DARPA (great writing here from David Kernohan) before handing over to Matt Clifford, the author of the AI Action Plan.

We know that, despite their supposed political differences, Gove has always been a huge fan of Tony Blair, whose Tony Blair Institute (funded by Oracle’s Larry Ellison, Amazon Web Services and SpaceX) joins Public First and Faculty in an unholy trinity of thinktanks providing ‘research’ and policy advice on AI in the public sector. Future blog posts will, I feel sure, have more to say about Tony Blair’s role in the UK’s trajectory from centrist technocracy to swivel-eyed techno-authoritarianism under the thumb of the US military state.

For now, let’s get back to Faculty and its data store, currently being touted by Peter Kyle ‘as an early prototype for ministers as they draw up broader plans to sell anonymised public data to researchers and businesses’.

I am imagining. ‘Access for free’, remember, is for developers to build prototypes and use cases that will allow the data to be commercially leveraged. And whether there is any educational value to these use cases or not, it is no longer Faculty’s business, since the mere prospect of a magic marker has put the data into their hands.

Who should be safe?

Faculty’s business is data, not education. Their main commercial product, Frontier, is a data analytics layer that promises ‘better, faster decisions’ from organisational data, and the company’s success in securing government contracts is based on a similar model: sweeping up tracts of public data to offer it back as an intermediary layer between privatised apps and services.

As you might imagine from a company that is focused on ‘the frontlines of the world’ and colours itself khaki, much of Faculty’s expertise in data has been honed in the military and security sectors. It has partnerships with major defence companies to develop AI for autonomous weapons, and its role in security and border control was one reason for handing Faculty the safety testing contract for the UK AI Safety Security Agency. See, it is very easy to confuse safety with security, and ‘keeping data secure’ with ‘using data in security systems’. So here is a short primer:

‘Security systems’ use personal data such as facial recognition, location data, known contacts and home addresses to make people as unsafe and insecure as possible - to detain them, deny them border crossings, and target them for violent death.

Keeping data secure means not allowing data about people to fall into the hands of anyone who might use it to make those people unsafe, something the UK military has struggled with in the very recent past.

Now, it almost seems as if the first meaning of security is the opposite of the second, but there is an important difference. The people in the first scenario are bad people and the people in the second are good people, which is why ‘security’ can mean such opposite things. So deciding which kinds of people are which, who to keep safe and who to deprive of safety, and in the interests of what national ‘security’ agendas, are among the gravest responsibilities of the state.

These are the kind of decisions that should not, you’d think, be farmed out to algorithmic systems or the unaccountable private enterprises that own them. Yet even now, three years into the story that ‘generative AI’ is the future of economic growth, most UK government spending still goes on predictive AI systems: that is, systems for making decisions about people’s lives. Systems like the ‘real time data-sharing network’ Palantir is building across UK police forces, or Palantir’s software, offered to the Ministry of Justice, for calculating prisoners’ reoffending risks, or the massive investment in precision targeting and autonomous weapons unlocked by the Strategic Defence Review - substantially penned by Palantir. And while it might seem odd that a foreign company has so much power in UK Government, dw, Palantir is a long-time partner of Faculty. Why, as long ago as 2010 our own GCHQ was employing Palantir’s software to gather data from around the world, reassured by the fact that the company had every appearance of being set up by the CIA.

How reassuring, also, that there are such close ongoing ties between the UK state and the AI industry so they can always be on the same page about these critical issues. The former Director General of GCHQ and the very newly retired Chief of Air Staff of the UK Armed Forces have both joined Faculty AI in recent months - the links are to ‘advisory notices’ from the Committee on Business Appointments. Faculty is cited or mentioned 41 times in the Defence Committee’s report Developing AI Capacity and Expertise in Defence, the one that led to the massive spending boost for AI warfare.

All this might seem low rent compared with the US, where the world’s biggest AI companies are signing multiple contracts with the world’s biggest war machine, AI chief executives are being sworn in as US army officers and Palantir is in line for a contract to track the data of every US citizen. But that same Palantir, whose CEO boasts of ‘powering the West to its obvious innate superiority’, has been promoting the idea of a common operating system across UK government. Word has it that civil servants facing DOGE-style cuts of 15% are being forced to report their ‘time saved with AI’, though it’s not clear if they also have to log the time wasted, the tasks that take longer, or all the times AI makes them miserable, or whether using more AI will make their jobs safer or less secure.

So here’s an idea. Perhaps civil servants could instead be developing the expertise to offer ministers independent guidance on public sector AI. Independent from the US military and security companies that are cited in every report and shooed in for every public contract. Independent of the seconded tech workers proposed for every government department, and independent of the ‘independent think tanks’ that are stuffed with Silicon Valley assets.

In early 2023 I wrote about the dangers to the public sector when corporate data and data services become core to service delivery (‘AI and the privatisation of everything’). While this danger has only intensified, I now see an even more potent threat. The neoliberal state is being transformed into an authoritarian, securitised state, with AI acting both as a powerful narrative for the manufacture of consent, and as a means of co-ordinating commercial and state capacities to govern through the use of personal and social data. The government’s role in collectivising risk to provide shared social security has been replaced by an obsession with ‘national security’ and military spending, as illustrated by the Orwellian rebranding of AI Safety as AI Security. From monitoring AI risks to using AI for surveillance and war.

And here’s a last thought for my friends in education technology. When edtech is bracketed with ‘defence, justice and security’ as opportunities for a cabal of cranks and spooks to do what they like with UK citizens’s data, perhaps it’s time to get off the AI train.

This post continues my thoughts on AI as the opposite of education: you can check back there for a list of them all as they come together.

(Not so much) innovation

While the benefits of AI in learning and teaching continue to be debated, its transformational impact on research and innovation is widely accepted.

Here, I am going to deal mainly with scientific research. Innovation also belongs to the arts, humanities and critical social sciences, and I have plenty to say in other posts about the impact of AI in these, my ‘home’ subjects. But in these areas the AI industry tends to rely on brute force of capture and less on claims to innovation: nobody is claiming these subjects can go home and leave research to the machines. From the famous stochastic parrots essay to the Resisting GenAI in Writing principles, there are coherent arguments as well as cultural biases against the idea that statistical iteration over old forms can produce new creative or critical work. Creative workers have been among the few groups in the global North to organise effectively against the threat to their livelihoods, and theft from creatives is one objection to generative AI that is widely understood. In fact the AI industry no longer even pretends to respect creative work, attacking the laws that protect its value, and gaslighting creatives who stand up to them. Prominent among these, Hayao Miyazaki of studio Ghibli, once said of synthetic art:

‘Whoever creates this stuff has no idea what pain is whatsoever. I am utterly disgusted… I strongly feel that this is an insult to life itself.’

Still, scepticism about AI’s creativity coexists in the public mind with an absolute conviction that it is ‘revolutionising science’, and it is in scientific research that AI makes its strongest claims to transformative impact.

In research, we are mainly talking about specialised machine learning models that have been trained on a refined diet of meticulously prepared and filtered data, sometimes collected over decades, and not on the junk that goes into generic models. These models are also trained by scientists to ‘see’ the things that scientists are looking for: there are an infinite number of patterns in data, of which only a fraction are meaningful. Darren Acemoglu, an economist whose downbeat assessments of generic AI I have cited before, writes in Prospect magazine that the only hope for a positive outcome from AI is to focus on the potential of these specialised applications .Dan McQuillan pushes back on even these modest hopes. My position is somewhere between the two. I believe that expertly collated data, viewed with expertise, is often of value, but the value has to be carefully weighed against the costs, both the local costs of collecting and managing data, and training experts to work in relation to that data rather than in others, and the systemic costs of over-investing in data as method at the expense of many others - also of course the concentration of power and expertise that comes with data-intensive methods.

I previously wrote about the use of machine learning in materials science and drug discovery, where AI is predicting molecular structures in ways that may guide but do not replace the hard benchwork of discovery and testing. These ‘new’ molecules only exist in theory. And even at the molecular level, it seems now, the real world is messier than data models allow for. The much-vaunted protein-folding models get basic physics wrong and (from the same article) the ‘accuracy of these models may degrade substantially when predicting non-standard or previously unseen protein-ligand systems’. In other words, they cannot be used reliably for new discoveries. In climate change research, a recent review finds that machine learning techniques are not ‘viable alternatives to traditional numerical models’ and after decades of investment, the two approaches have still not been integrated effectively. In another high-profile case of ‘AI beating humans’, partial differential equations turn out not to have been solved after all. Even where machine learning has been used for decades to deliver efficiencies in research tasks, the benefits are in narrow parts of the workflow, results typically need to be verified by other methods, and the whole enterprise is costly. An overview recently published in Nature concluded simply that ‘an over-reliance on AI-driven modelling is bad for science’.

But the hype continues. Consider a recent headline, also from the Nature stable:

The article behind the headline provides a table of promising applications of ML in epidemiological modelling, none fully mature, and the best offering only ‘incremental’ gains. But besides overstating the potential impact, ‘help prepare the world’ recognises - without actually saying so - that it is up to other people to realise the actual benefits from any new insights. Despite the fact that ‘AI’ and indeed ‘machine learning’ have been around for 75 years, the benefits exist almost entirely in the conditional future tense, and in moving from the present diagnosis to that future solution, someone, somewhere has to do some actual work. Prediction is not amelioration, diagnosis is not cure. The gap must bridged by the mobilisation of people and resources, by investment, by embodied activities of construction and care.

And there, more precise knowledge of the problem is only sometimes useful. Public health measures, vaccine programmes and less exploitative farming practices are all preparations for the next pandemic that can be applied today, with technologies we have, and with benefits that have already been proved - even if the pandemic should not arrive. The greatest gains would come from applying these solutions in parts of the world that don’t have them already – and models are a poor fit in these left-behind regions (no resources: no data). So the problem isn’t a lack of precision in targeting, the problem is unequal resources. In fact, the big problems that AI promises to solve all have this same catch: the political and material resources to solve them still have to be mobilised.

Recall that the Bayesian statistics behind all this modelling were nurtured in the insurance industry and speculative finance, and consider the move that makes precise knowledge of the future more valuable than taking action for everyone’s greater security. Bayesian calculations certainly allow people with better predictions to make money. NVIDIA’s climate modelling behemoth Earth-2, a ‘digital twin’ built largely on public data, advertises its core users as Spire, JBA Risk Management and Climasens, all insurance and risk management corporations. With its own private satellite network, Earth-2 can be seen as a privatised double of the earth that increasingly extends into the real world in the form of data monitors and orbiting hardware, all to benefit the people who benefit from monetising risk.

The ‘good’ uses of ML in scientific research might again be thought of as new ‘lenses’ on existing data, guiding scientists to pay better attention to patterns that they in turn have guided the model to look for. Sometimes the value of data makes it worth investing in some massive shared project to collect it - think of the large hadron collider or James Webb telescope, an instrument whose history has become entangled with generative AI. But even where the new data is of great value, the concentration of that value in a single project carries risks. It tends to reduce the diversity of the research workforce and to harm the diversity of methods and perspectives, as well as (in fact, as symptoms of) concentrating expertise and funding.

Unlike the telescope, microscope or CT scanner, ML data is not being collected in real time. The lens is a historical one, and large-scale historical data sets carry additional risks of their own, that this paper identifies as: ‘representational harms’, ‘faulty science’, ‘over-generalisation/de-contextualisation,’ ‘poor labour conditions’ and ‘a lack of care for data subjects’. Philosopher Thi Nguyen reminds us that data has intrinsic limitations. To decontextualise means to choose what is worth recording as data and discard the rest. To categorise means to choose the meaningful features. To measure means to decide on the norms. ‘Data-based methods are intrinsically biased’, not accidentally so.

Entire research funding systems are now being skewed towards past data and their biases, especially in already-data-rich areas of research. A recent study on research careers titled ‘Innovations in Technology instead of Thinking?’ found that:

considerations about potential future academic revenues derived from innovative research technologies sometimes seem to override particular epistemic valuations.

Among the ‘potential futures’ and ‘epistemic values’ that may be over-ridden in the rush to machine learning are real-world experimentation and observation; the collection of data that does not fit existing parameters or categories; new contexts; methodological innovations beyond data crunching; and new theoretical paradigms - all drivers of innovation for most of the history of science. Armin Grunwald, a philosopher of technology, notes in a recent essay the ‘risk of closing down the openness and potentiality of futures to a model-driven prolongation of the past’.

And in every field where ML is prioritised as a method, the values of ML (‘Performance, Quantity, Efficiency’) produce opportunities for commercial capture. Earth data, like that collected by Nvidia’s Earth-2, is sold for commercial advantage. Google DeepMind retains the right to assetise patents arising from the use of its AlphaFold database although the data involved was all derived from investigative science. Microsoft and Google are competing to assetize environmental data, and there are intense efforts to control genetic and epigenetic data, with big AI recently investingly heavily in genomics and pharmaceuticals so that discoveries based on the DNA of people and other species can be assetised in exactly the same way as the art of Studio Ghibli.

Just as Miyazumi found AI art an ‘insult to life itself’, so the capture of ‘life itself’ as data portends new harms to all of us and to our common planet. Meanwhile the AI hype machine siphons precious resources from the foundational research and the putting-into-practice of well-researched solutions that we all need to survive and flourish.

(Not) ‘enhancing’ research

If machine learning has achieved less for research than is advertised, and at greater cost, still the generative AI industry has been desperate to associate their off-the-peg consumer products with those data-curated, expertly-trained specialist models. Lorena Barba, a professor of machine learning, collects examples of such hype and has rather wonderfully classified them as: ‘performance claims out of context, renaming old things, incomplete reporting, poor transparency, glossing over limitations, closet failures, overgeneralization, data negligence, gatekeeping, and puffery’ (Barba 2023 n.p).

With such a fine-grained bullshit detector available, there is really no excuse for universities to buy it. Leon Furze has done a great job of evaluating the so-called ‘deep’ research capabilities of generic LLMs (and if you don’t already subscribe to his work, you really should), concluding that they may be OK ‘for businesses and individuals whose job it is to produce lengthy, seemingly accurate reports that no one will actually read’. Anyone who engages with student writing at any level will be familiar with the miasma of credibility effects produced by these so-called ‘research’ models: fluent summaries that include real sources, though the summaries are repetitive and over-generalised and the real sources rarely say exactly what they are supposed to say. More than two years after Ted Chiang found that ChatGPT is a blurry jpeg of the internet, the billions spent on tuning it for ‘research’ effect have made it a blurry jpeg of Google Scholar, while simultanously turning both sources to absolute crap.

Research is not a general purpose activity and there is no general accelerant you can apply. A recent preprint explains that ‘The promise of foundation models relies on a central presumption: that learning to predict sequences can uncover deeper truths, or optimistically, even a world model.’ Cleverly, the authors devise a series of ‘inductive bias probes’ to test this presumption with Newtonian physics (and if you are thinking: ‘why didn’t they just ask the AI to explain Newtonian physics?’, please read my piece on why AI is not reasoning, and how very difficult it is to find out what it is doing at all). The ‘related work’ section of this article is particularly thorough and convincing. What the authors found, repeating findings from a host of other studies, is no. There is no deep understanding. There is no world model. There is just a ‘bag of heuristics’.

Nonetheless, the last year has seen a flurry of surveys from academic publishers – mainly the ones that have just sold our life works to big AI – purporting to show that researchers are excited about the opportunities of AI in research. Most of these surveys are so biased in design and so dishonest in reporting that I can’t in all conscience bring them to your attention, but despite a misleading headline and an odd discrepancy in the figures, Oxford University Press has passed on its results with good candour. Among its researcher participants (n= 2345), 27% claimed to have ‘benefitted from AI’ (not the 36% reported) and just 5% of these - a little over 1% of the total sample - mentioned using it for their ‘own work/analysis’. The other ‘beneficiaries’ had used it for research-adjacent tasks such as ‘to write more polite emails and to think of cool titles for my papers’.

Of the eight clusters of opinion identified in the OUP report, 933 researchers were negative about AI in research - it’s not obvious why this group has been split four ways. Another 811 were undecided, though we don’t know how this group weighted ‘saving time’ and ‘excitement’ against the rather more research-relevant factors of ‘distrust’ and ‘concern’. Just 345 were positive overall about the impacts of AI on research, and in true ‘frontier’ spirit these were labelled ‘pioneers’ rather than, say, ‘careless optimists’ or ‘reckless corner-cutters’.

Surveys of this kind deal in belief, and researchers are not immune from having their beliefs shaped by the direction of research funding, or by academic social media with its relentless FOMO, or indeed by surveys of their peers. And if your research goal is to publish an early AI use case in your field, you can probably get support from your institution and perhaps even industry partners to do this. Your local case study can be cautiously positive while your introduction and discussion project all kinds of opportunities into the future conditional tense, in which there is always room. There is little evidence that these cases do scale up beyond the favourable scenarios to which early adopters have applied themselves, but there are certainly rewards for producing them.

Titled ‘AI is revolutionizing science’, a recent report notes that ‘mentions of AI’ in published papers rose ‘from ten to thirty percent’ between 2015 and 2019 and that: ‘papers mentioning AI n-grams were roughly twice as likely to be a “hit” [highly cited] within their respective fields’. We can only assume these effects have been amplified since this work was done. Higher ‘mentions’ are glossed throughout this report as ‘AI benefits’, as though citation statistics are the benefits of research – a little like grades being the benefits of study. And just like in AI-generated literature reviews, ‘mentions’ must be pumped for all they are worth. Because generative models don’t attach qualities to instances, or explanations to correlations, so mentioning is their only move and counting mentions their only power.

For those whose research is not being boosted by AI ‘mentions’, the harms to research quality are now impossible to ignore. Those sharp AI commentators, Arvind and Sayash, recently asked the apparently paradoxical question ‘could AI slow down science?’ and concluded that yes, for all the frenzied activity, there is an overall drag on science as a shared system.

AI Snake Oil

Could AI slow science?

AI leaders have predicted that it will enable dramatic scientific progress: curing cancer, doubling the human lifespan, colonizing space, and achieving a century of progress in the next decade. Given the cuts to federal funding for science in the U.S., the timing seems perfect, as AI could replace the need for a large scientific workforce…

Read more

a year ago · 147 likes · 20 comments · Sayash Kapoor and Arvind Narayanan

Because as well as cutting out all the parts of research that actually advance human understanding, AI is crapifying the shared systems that research depends on. There is an epidemic of fake research, fake citations, fake peer reviews, and whole fake research topics (‘vegetative electron microscopy’ anyone?). Google Scholar is flooded with AI-generated slop; academic publishers and editors are overwhelmed by AI-generated papers and even AI researchers; there are threats to the viability of archives and a creeping influence on library catalogues where the use of AI in metadata creation is now widespread. Open knowledge projects including Wikipedia, source of most actually useful AI content, are being drained of funds and even taken offline by AI crawlers, and the supply side of intellectual labour is failing. The peer review process, is corrupted by AI and now by cynical manipulation of the corrupt AI review system (this last link is to a study of papers in computer science that found hidden text prompting AI reviewers to provide only positive reviews).

There are calls of course to move to a 100% AI review system, because for every AI fail there must be an AI solution. And if AI could reliably provide that it would just be an extortion racket, but since it can’t, it’s more like paying the extortionists to burn your house down.

In this episode, I talk to Doug Belshaw, co-founder of We Are Open cooperative, an old friend and sometime sparring partner – I’m sure he won’t mind me saying that. And if you’re here for the AI, we dig into large language models, the need for playful as well as critical spaces, and current AI literacy projects, all in the second half. In the first half we revisit some older ground, discussing the various digital literacy frameworks we’ve been involved with and whether they are still relevant. By way of Walter Ong on literacy and William Empson on ambiguity we come around to whether AI literacy is a useful term. No, we don’t have any simple solutions and yes, we do disagree about several things, hopefully in an interesting way. As always there are lots of references in the show notes below.

Please do like, comment, subscribe, and consider becoming a paid subscriber so I can keep working with my good friend and audio wizard Ans to improve the sound of the pod. Their contact details are also in the shownotes, along with Bryan’s who drew the ‘contexts’ image and others you’ll find on the blog.

Digital literacy stuff we mention:

Doug’s ‘open thinkering’ blog: https://dougbelshaw.com/blog/

Doug’s book about his digital literacy framework: https://dougbelshaw.com/essential-elements-book.pdf

Jisc Digital Capabilities framework: https://digitalcapability.jisc.ac.uk/what-is-digital-capability/individual-digital-capabilities/our-digital-capabilities-framework/

Knobel and Lankshear: the ‘new’ literacies https://newlearningonline.com/literacies/chapter-2/knobel-and-lankshear-on-the-new-literacies

Kahn and Kellner: Reconstructing Techno-literacies: https://pages.gseis.ucla.edu/faculty/kellner/essays/technoliteracy.pdf

DigCompEdu (Digital Competence Framework for Educators): https://joint-research-centre.ec.europa.eu/digcompedu_en

PISA 2029 Media and AI Literacy assessment: https://www.oecd.org/en/about/projects/pisa-2029-media-and-artificial-intelligence-literacy.html

Angela Gunder and team: AI literacies: https://aiopeneducation.pubpub.org/pub/fmktz5d3/release/4

Other stuff we mention:

Danah Boyd context collapse: https://www.zephoria.org/thoughts/archives/2013/12/08/coining-context-collapse.html

William Empson 7 types of ambiguity: https://en.wikipedia.org/wiki/Seven_Types_of_Ambiguity

Richard Rorty on dead metaphors: https://www.lrb.co.uk/the-paper/v08/n07/richard-rorty/the-contingency-of-language

Walter Ong Orality and Literacy: https://monoskop.org/images/d/db/Ong_Walter_J_Orality_and_Literacy_2nd_ed.pdf

Walter Ong on secondary orality: https://en.wikipedia.org/wiki/Secondary_orality

Richard Seymour Twittering machine: https://www.versobooks.com/en-gb/products/2505-the-twittering-machine

The people who help me could help you too:

Ans for Audio: https://anshassel.net/

Bryan at Visual Thinkery for visual thinkery: visualthinkery.com

It’s been a while, and I’m sorry, but like everyone else I’m tired of AI. Dealing with the small local fires is bad enough without the horizon being constantly shrouded in smoke. It’s not as though there aren’t other crises. Austerity, redundancies, ideological attacks on forms of scholarship that don’t contribute directly to military or industrial advantage, ‘crackdowns’ on student visas and student protests (and if universities can ignore the warmongering and human atrocities, they still have to deal with the protests). Then there’s the collapse of democratic norms and public discourse, the very things that education is supposed to secure. Can’t we just agree some accommodation with AI and focus on these other, more existential issues?

Unfortunately these other issues are - to quote a recent keynote from Jennifer Sano-Franchini – ‘interconnected and co-constituted’ with AI. Where the AI industry is not actively contributing to them it is drawing energy from these other crises. Sometimes, as with Trump’s latest Bill, it is doing both at once.

So while I am tired of AI, I am even more tired of the cope. From ‘this is the worst AI you will ever use’ (readers, I did not agree) we are now deep into ‘this is the best it gets, deal with it’. Feel the mid and do it anyway. Meta’s Llama has hit a roadblock, GPT4.5 is underwhelming and Gemini 2.5 faces a backlash from angry developers. Inaccuracies and fabrications (classified in this article as ‘botshit’) are actually getting worse, perhaps due to a training diet of AI-generated slop and synthetic data. Even the true believers admit that AI is running out of data and scale won’t improve things anyway. This last link, from AI fanaticist Toby Ord, really does say the quiet part out loud.

Let me translate:

Since 2020 it has been known by the industry that scaling up AI would make catastrophically increasing demands on power and compute, while returns on model accuracy would quickly tail away. But the industry is betting that the financial returns will continue anyway, because everyone will be locked in.

Yet every time the mask slips on the ugliness, greed and rapacity of the AI industry, whenever the lies and misdirections and fabrications become unsustainable, come the grown-ups to tell us to calm down and carry on. Because polarisation is unhelpful. No matter that on one side are the four or five largest corporations that have ever existed, the biggest bubble of financial over-investment, the most powerful military and surveillance states and all the combined forces of tech hype and mainstream media, while on the other side are thoughtful people with arguments. The good academic must always plot a middle course between naked power and poor thought.

But what if there isn’t a middle of this road? What if the project of ‘artificial intelligence’ is not a road to new kinds of education - not even a slow and bumpy one – but the reversal of everything education stands for? What if, at at least in its current, (de)generative, hyper-capitalistic guise, the project of AI is actively inimical to the values of learning, teaching and scholarship, as well as to human flourishing on a finite planet? In my next few posts I consider the evidence.

The links will go live as I add them. They are, as always, imperfect thoughts. I wait for counter arguments. Please also let me know if there are other propositions you’d like me to include.

Thoughts so far:

(Not) telling the truth

(Not) reasoning or explaining

(Not) developing expertise

(Not) learning

(Not) teaching (either)

(Not so much) innovation

(Not) enhancing research

(Not) education as a right

(Not) education as the practice of freedom

(Not) thinking

(Not) telling the truth

Generative AI doesn’t tell the truth. According to the Association for the Advancement of Artificial Intelligence, in a factual accuracy test from December 2024 ‘the best models from OpenAI and Anthropic correctly answered less than half of the questions’ posed. Other concurrent studies also found AI outputs to be wrong at least half the time and with so-called ‘reasoning’ models things are even worse. That’s when they aren’t being used deliberately to spread conspiracy theories and encourage delusions.

I fully agree with John Noster when he says that generative AI is a ‘counterfeit’, but I want to push that observation beyond the everyday meaning of the word. Yes, AI is a fake, but more importantly, it is making (feit) in a way that is counter to - opposed to, inimical to, actively working against - the makings that people engage with together in order to flourish together. In being ‘fluent, convincing [but] fundamentally ungrounded and untethered to our humanity’, as Noster rightly exposes, AI productions extract value, attention, investment and care from the conditions in which people can decide what is true and what is worth believing. Generative AI is not making knowledge in an interesting new way, producing some unfortunate biases and incidental untruths that can be overcome by people with ‘AI literacies’. It is a mode of production that captures, devalues and finally exhausts the ways of making knowledge that belong to people and, in return, offers up only past representations, rehashed through forms of hyper-exploited and alienated data work.

‘Artificial intelligence’ itself is founded on a lie. It does not exist, or not as its key players define it. Sam Altman, for example, has said for years that the goal is a system ‘more capable than most humans across all areas of intelligence’. He announced a ‘breakthrough’ towards this goal in November 2023, confidently predicted human-surpassing AI ‘within a year’ in 2024, and in 2025 has postponed the rapture until 2029, though still confident that ‘Superintelligent tools could massively accelerate scientific discovery and innovation well beyond what we are capable of doing on our own, and in turn massively increase abundance and prosperity’, although AI was supposed to be doing that back in 2022. What has now been renamed ‘artificial general intelligence’ or ‘strong’ AI is impossible on principle, according to many philosophers and computer scientists (Luciano Floridi, Herbert Dreyfus, John Lanier, Ragnar Fjelland, Francois Chollet and Iris van Rooij among them). It is economically unrealisable, according to an entirely different group of experts (Richie Etwaru, Darren Acemoglu, Ed Zitron, and The Economist). It is unachievable with current techniques, according to most AI researchers, and recent polling has found that even if it was possible, available and affordable, most people wouldn’t want it. And yet it is in pursuit of it we are asked to put up with everything AI can offer in the present.

Behind that one big lie come a hundred lesser lies like beads in a row. AI will make work more creative and rewarding. AI will make workers more productive - but somehow that won’t mean a choice between accelerating your work rate or losing your job. AI will help students learn (‘reduces cognitive load’). AI will stop students learning (‘reduces cognitive effort’) but don’t worry, we know the difference. Don’t use AI for this, because it’s cheating. Do use it for that exactly same thing because it’s future proofing your skills. We can definitely tell when students are using AI. We aren’t using AI. We may be using AI but only when it is safe, responsible and ethical to do so. All our platforms and AI enterprise systems and agents and co-pilots are safe and ethical too and we absolutely 100% have audited that.

Big tech will be fine. It managed to push on through the MOOC lie, the metaverse lie, DNA testing scams and non-fungible tokens and crypto. Ok, the crypto lie is thriving under the Trump dynasty, but only as a manifest corruption scheme. Lying is business as usual for big tech. But education is supposed to have a rather different relationship with the truth.

Probability is part of the problem. The Bayesian probability that machine learning systems are based on does not, like the frequentist probability we were taught in school, express the probability of something being true. Rather they express the value of a particular prediction about the future, assuming that prediction were to pay off against alternative bets. Model training is ‘adversarial’ in exactly this sense, betting different states of the architecture against each other to find which produces the best, least ‘lossy’ outcome. In the world of financial speculation, where Bayesian methods have their most enthusiastic and indeed value-generating users, the bet you make does not matter so much as your power to make the market in which your bet turns out to have been the right one. Speculative financiers have bet big on generative AI, knowing that it does not and cannot produce any value in the usual sense of profit until knowledge production is substantially turned over to AI. But those investors are relying on the rest of us to make that market for them.

Education leaders who insist on the adoption of AI are actively producing a future in which the bet on AI turns out to have been the value call.

Of course education leaders may have reasons to prefer the AI future. Far more likely, though, they have become convinced that no other future is worth believing in or striving for. Like other situations where the truth is obvious but uncomfortable - and we don’t have to look very far in the contemporary milieu - there are personal costs to speaking up. But in some possible futures, the costs of not having spoken up may be high - the costs of complicity, for example. So what starts out as a wholly understandable reluctance to speak unpalatable facts becomes a commitment to producing the future in which that bet was the right one: the ‘facts on the ground’ have been transformed, the nay-sayers were mad or bad or have shut up and gone away, and we are living in the only future that was possible.

Education leaders who want to hedge their bets even a little at this point could start by telling the truth about the crises unfolding from the release of ChatGPT and other large language models. The targeting of students as users and the capture of educational researchers and policy makers has been relentless. No other group of people (with the possible exception of PR professionals) has adopted ChatGPT as thoroughly as students have or is using it so indiscriminately. We may not have answers but we could share truthfully with students that our practices of teaching, learning and assessment are being broken and that we need to find ways of continuing to learn together. We could stop pretending this is all OK.

The truth has a hard time in parts of academia, some of the parts where I feel most at home in fact. Universities - literal palaces of ‘one truth’ - have been complicit and remain complicit in projects of extraction and exploitation in the name of reason, not least in recent times through their partnerships with the AI industry and its military correlatives. But if this imperial certainty is to be overthrown, please can we hope for some more thoughtful kinds of uncertainty in its place? Some better accountability to history, some more equitable relations among knowledge cultures, some honesty in self-examination as well as rigour in methods. Can we not just put up a new tyrant to insist that truth is relative (to power) and nobody can believe the evidence of their own senses?

Because in a world of live-streamed genocide and hourly lies from the highest authorities, who does ‘post-truth’ really serve? We are not witnessing a scholarly debate about the plurality of interpretations here but a fascistic turn to revelation, authority and affect. While predictive AI is deeply implicated in the oppressive violence of Israel, China, Saudi Arabia and the US military, in state-sanctioned sectarian violence in India and in the violent deportations of Trump’s ICE, its generative branch is helpfully putting all the evidence of these crimes beyond belief. Experience is de-realised: everything smells fake. And when we append our advice to students with that small disclaimer ‘AI may be wrong’ we are teaching them that this doesn’t matter. Give up striving for truthfulness. There is no important connection between words and accountable actions in the world.

‘AI’ is a scheme of misdirection. The original Turing test and all of today’s AI benchmarks lay it down in black and white. How many people can you fool? That is literally the value proposition. And it is essentially the same question as: how many people can you enthral with your power?

Data power, military power, political and media power, computational heft… as Alex Karp of Palantir has said (not quite quoting Humpty Dumpty), ‘AI is a weapon that allows you to win’. All power, no truth.

(Not) reasoning or explaining

Most scholarship does not aim at anything as ambitious as the truth but does aim to apply reliable methods and to explain the results. Methods are fallible but they are accountable, in the sense that other scholars can review their use and replicate or challenge their findings.

Generative models do not allow for such accountability. Their inferences are non-replicable. Their ‘black box’ architectures are constructed through trillions of discrete operations on billions of data tokens and tracing even a fraction of their hidden structure requires probing methods that are specialised, laborious, poorly understood and imprecise. On top of the models’ intrinsic obscurity there is deliberate secrecy over the data sources and the human data work involved pre- and post-training. To produce ‘chain of reasoning’ style interactions, for example, data workers are used to rate model ‘explanations’, to provide sample explanations for model refinement, and to build whole ‘explanation’ datasets for model training. User preferences are also gathered and re-used to make ‘reasoning’ sound reasonable. You thought it was just the maths talking?

Unfortunately the more models are tuned to ‘explain’ their ‘reasoning’, the more errors they introduce. So-called chain of reasoning (CoR) does not express how a model is actually ‘reasoning’ because ‘models are not reasoning’ (Apple 2025): instead, as the same Apple team demonstrated in a 2024 paper ‘they replicate reasoning steps from their training data’. CoR reports are untrustworthy on principle: they are plausible explanations for plausible responses, and since the inferences involved are more complex, they burn more compute and carbon per query as well as introducing more mistakes. With complex problems, the supposed ‘reasoning’ can collapse suddenly and unpredictably, while with simple problems the ‘reasoning’ model can continue ‘its computational wheel-spinning’ long after a solution has been reached. (I wrote about this ‘halting problem’ in an earlier post on the Turing test).

People need explanations. By explaining we develop rubrics, models, theories and schemata so we aren’t starting afresh every time we meet a new situation. In research, methodological clarity matters so we can invite others to query our reasoning. In teaching we want students to arrive at explanations for themselves, as part of building a mental world, and to test those explanations in different contexts. But we face an advancing doctrine that what these data models are doing, obscure as it is to human understanding, is a better route to knowledge – faster, more ‘objective’ and more scalable – than the old evidence-and-reason routine. Data models are vast enough to contain approximations to every new case. Data does not need to generalise. As Chris Anderson notoriously argued in Wired (2008):

Petabytes allow us to say: "Correlation is enough." We can stop looking for [explanatory] models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

The ‘we’ in this sentence are not learners reaching for new understanding, or research communities trying to build useful shared knowledge, but people with ‘the biggest computer clusters’ and ‘petabytes’ of data, whose answers must now be accorded authority despite (or perhaps exactly because of) their resistance to human-readable justification. Along with post-truth, data models usher in an era of post-science, post-explanation or more broadly of post-theory in which whole cultures of human reasoning are less valued than data architectures, whose inferences people can subscribe to but never share in, still less question or debate.

Among the rebutters of Anderson’s thesis, philosopher Byung-Chul Han points out that when we give up on some correlations being more meaningful than others, we give up on reasons and reasoning altogether. Absolute information (if it were possible) would be the same as absolute ignorance:

Navigating one's way through Big Data is impossible… Dataism arises from the renunciation of meaning and connections; data is supposed to fill the void of meaning. Han (2013)

I respect Han’s philosophy but I prefer a different argument. Data models are not unreasonable, to my mind, but they are new regimes for coordinating the work of reason. Data and datasets are the products of reasoned collation, curation and labelling. The data workers involved in model refinement are making sense – interpreting, evaluating, comparing, deciding, often completely rewriting outputs. AI engineers have human sense-making in mind as they guide models towards ‘desired’ states - those that seem (to their users) reasonable. And finally, those unpaid users must endlessly check, correct and compensate for the ways that models still fall short of good sense in their contexts of use. It is in the interests of the AI industry to obscure this reasoning work behind the mask of data and algorithm, but in truth LLMs are only useable as a regime of reasoning because they involve human reasoning at multiple points.

But these disparate reasoners don’t constitute a community in the way that a university department does, or a local history group or a fan fiction site. They are separated from knowledge of their own knowledge, which becomes knowledge only within the regime of the data architecture. They are separated from each other, working on different terms, unable to reach a consensus or agree on a shared project or even to know what anyone else is doing. The low entry bar for using AI allows the industry to claim it is more ‘accessible’ than a university course or a difficult book, but beyond this low bar is a rigid division of labour and a structurally-enforced lack of visibility and mobility. At least the research community has to justify its openings and closures, its codes, procedures and rules.

Just as with ‘the truth’ we don’t have to believe that regimes of knowledge production in education are open and fair to believe they can be made more so, and that they work better (reasons are better justified, knowledge resources are better used) if they are as open, fair and diverse as possible. The open education community has promoted these values for many years, one reason why in a recent keynote I insisted that ‘AI is the enemy of open’ in education (another thought - full recording coming soon).

The AI stack is not a community. It is not amenable to dialogue, critique, consensus-building or action to challenge injustice. Its data architectures are impenetrable by nature, obscure by design, unaccountable by the ruthless prerogative of commercial power.

Technology, as Peter Thiel has opined, is ‘an incredible alternative to politics’ as a mode of power. It has made some people so powerful they can put themselves beyond reason and explanation entirely.

(Not) developing expertise

Even if they take the most transactional, neoliberal view – that the point of education is to develop economic capacity via people with expertise – education leaders should examine what the AI project means for that transaction. The goal is for expertise of the kind currently accredited by the education system to be represented, organised and operationalised through data architectures. While the jury is still out on how far this is possible, given the poor showing of these models in real-world contexts of expertise, workplaces are already restructuring workflows as data flows so that AI agents and APIs can be made to ‘work’. This redesign is not incidental. Following Stafford Beer’s ineluctably materialist logic, the purpose of a system is what it does - and as I have argued from the start, the purpose of the AI system is to redesign work, especially what is called expert or ‘knowledge’ work (though, in my view, all work is also that).

Anyone who followed ‘artificial intelligence’ through the era of ‘expert systems’ will be reasonably familiar with its failures, and may even know some of the reasons advanced for them. Most famously Herbert Dreyfus, philosopher and AI critic observed that representable knowledge is only one element of expert practice (that is, what experts actually think and do when their expertise is being put to work to produce social value). Other elements include embodied routines, situational know-how, ‘perceptual expertise’ or the way that senses become re-calibrated (experts notice different things), remembered experiences, the values and norms of the expert culture, and the ‘feel’ or flow that comes from integrating all of these fluently. Dreyfus was also interested in purpose - why certain things are worth doing and thinking about, and what animates sustained projects such as learning to become an expert. Representable or conceptual knowledge is still a crucial element of expertise - but even managing representations and concepts involves know-how, that can itself become more expert and specialised.

Now there is evidence, summarised here by Punaj Mishra, that the use of generative AI by experts can actually make them less productive; astonishingly, this is even true of that most happy user, the professional coder. Interruption to the flow of practice is a drag. Also, did I mention, generative AI is crap and monitoring the crapness is a further drag on time, attention and productivity.

Generative AI does not serve professionals well. It can be used to offload some of the standardised, repetitive tasks that would once have gone to junior colleagues. But who exactly does this serve? Not the established professionals who are expected to be more productive while ‘minding the AI’ for errors and failures; not the younger workers whose roles are squeezed out by ‘AI’ (or, more accurately, by data workers in the AI labour stack) and who miss out on valuable learning as a consequence; and not the professions whose knowledge is being siphoned into data structures that they must lease it back from. Not even the workplaces that have paid for the AI and must now pay to make their own data ‘AI ready’ and AI-secure, so having less to invest in people with talent and expertise.

The whole point of cognitive automation is to turn expensive, specialist expertise into data work, some of it to be done by experts themselves (fewer of them, working harder) and the rest by more generic workers who can be casualised. In fact as generative models fail to improve by the magic of mathematical emergence, the volume of data work being done behind the scenes is actually increasing. It is also getting more specialised, meaning that expert work is becoming data work. AI labs are ‘hiring thousands of programmers and doctors and lawyers to actually handwrite answers to questions for the purpose of being able to train their AI’, according to Mark Andreessen, who should know. Its continued reliance on human experts to pull the strings of the ‘smarter than human’ system is why Meta recently acquired ScaleAI, a datawork outsourcing company notorious for its exploitative labour practices.

Whether captured by outsourcing companies or held in the databases of business users, expert knowledge is being captured, to be put to work again by other experts - because it is only when that knowledge is put to work in human contexts and for human purposes that it acquires actual value. But more of that value is now falling towards the companies that own the knowledge-extraction-and extrusion machine.

For the whole bait-and-switch to work, expertise must be made more representable at the one end (that is data work) and more exploitable at the other end (that is the demands for greater productivity through datafied processes).

The neoliberal university ought to understand all this, since the economy is its raison d’etre. It should be able to explain why the wider economy shows so little AI impact on earnings or efficiency, so little AI-enhanced growth and productivity beyond that most capital intensive sector the AI industry itself. It should have predicted that AI adoption would be hype-driven and shallow rather than deep and structural, and that the majority of workplace AI projects would end in failure while every thinktank repeats the mantra that ‘work must become AI, so AI can work’. The neoliberal university may not have many philsophers left to explain the epistemic fallacies but it is swimming in business analysts who could advise about the economic risks. The general risks to the world economy of AI’s over-financialisation and under-delivery of value, its diversion of capital away from other economic priorities such as energy transition in favour of computational infrastructure, and its ponzi-scheme investment structure that relies on the rest of the economy conforming to AI and its demands before anyone makes any money. And the particular risk that – if it succeeds – the value of people with expertise and therefore of universities will be pushed to the floor.

Given the colonisation of expert knowledge by computation, it’s hard to see why societies and/or young people in them would continue pay for qualifications that might or might not add value to work in the expertise-data system. Major companies already prefer to gather their own evidence that applicants are ‘thinking for themselves’ and piece-rate workers qualify themselves by showing they can fulfil sample tasks before they are assigned to paid ones. Neither requires a university qualification. If data work is the best most graduates can hope for – and many students and recent graduates are doing it already - they might as well learn-to-earn within the data engine.

Some elite universities will no doubt continue to thrive in this scenario, promising innovation (from research outcomes, machine learning PhDs) and ‘top experts’ whose know-how will be propagated through the databases and architectures associated with their respective fields. People with expertise will continue to be the source of economic value and the drivers of innovation. It is mass expertise that is under threat, the aspiration of the majority of the world to become specialised and qualified, to become fluent in particular knowledgeable ways, and to secure better life prospects from that.

Look out for more links in this series! And check back soon for a relaunch of my podcasts with more ways to listen.

I was inspired to submit a thinkpiece to this UNESCO call by a group of people whose own submissions (I happen to know) are wise, original and inspirational - so I’m truly grateful to have had their support and conversation. You can read all our contributions at this linktree, and catch up on a live panel we held recently to discuss our contributions (not organised or sponsored by UNESCO). Meanwhile, these are my own reflections - not my submitted piece which we are asked not to publish.

What is the right to education?

The ‘right to education’ is enshrined in the Universal Declaration on Human Rights (UDHR, Article 26). But there are many cultures and definitions of education, as other Declarations and Conventions emphasise. ‘Education 4.0’, according to the World Economic Forum, ‘places the responsibility for skill-building on the learner’, with teachers as ‘facilitators’ of this self-determining enterprise. This doesn’t sound like something ‘rights’ apply to, except perhaps the right of consumer redress if your facilitation-as-a-service teacher fails to satisfy. Nor has the AI industry been slow to promote its own version. Elon Musk encourages students at his privately funded Astra Nova school to address problems such as ‘How should Tesla build out a supercharger network in South America’ - so Musk’s money makes him free to determine the contents of at least some children’s education. But how many people in the world does this ‘right’ extend to, and with what consequences for other people’s rights? Meanwhile JD Vance, VP of AI, rails against ‘teaching’ as ‘brainwashing’ and ‘indoctrination’ and calls for ‘our kids’ to be ‘defended’ from it. In the world of Musk and Vance every self-governing homestead should be free to decide what their young people need to know: that is, after all, what patriarchs have always done.

But the liberal-democratic version of education that Musk and Vance dislike so much is exactly what the UDHR and associated Conventions intend: education is a right of all people as equals and a responsibility of the public bodies who represent them - states, nations and regions. The context can only be a democratic one. And despite the difficulty of accommodating so many different epistemic cultures and pedagogic traditions, the UN’s International Covenant on Economic, Social and Cultural Rights (ICESCR, Article 13) does have something to say about the contents of that education: ‘education shall be directed to the full development of the human personality and the sense of its dignity… to enable all persons to participate effectively in a free society, [and] promote understanding, tolerance and friendship among all nations and all racial, ethnic or religious groups’. The UN Convention on the Rights of the Child (CRC, Article 29) prefers: ‘the development of the child’s personality, talents and mental and physical abilities to their fullest potential’ so that the child may be ‘fully prepared to live an individual life in society’. The Convention Against Discrimination in Education (CADE) adds a host of further Articles reinforcing the equality and universality of these rights.

That expression ‘an individual life in society’ expresses a tension in these universal goals, one that all educators have to confront at some time in our careers. Education aims at the fullest possible realisation of the individual, but on terms given by society: what that society holds to be a life worth living, knowledge worth having, and work worth doing. By equipping young people with what is collectively valued and known, education shapes their structures of thought and developing personalities. But learners must also be granted the agency to respond uniquely, to transform their conditions as well as themselves, and to resist social norms:

‘Education changes people; people change the world’ (Paulo Freire 1972)

There is other tensions here: that while a child is a full human being from birth, endowed with human rights, still s/he is not yet a full person. It is the task of education - in the broadest sense - to hold that space of immanence and vulnerability, to give shape and expression to what is immanent, and care to what is vulnerable.

‘Education’ is not the only such ritualised space but it is how the United Nations have designated the support that children and young people need for their emergence into full personhood and social participation. Primary education at least must be ‘free’, secondary and higher education ‘freely available’. This collective investment means that education must be accountable to the rest of society: the purposes of education are held publicly, even if private interests might be involved in realising them. And there is a final tension. While respecting national and cultural determinations about what it matters for young people to learn, the demand that education should ‘promote tolerance and friendship’ between ‘all nationals and… groups’ is a strong sign that it should be outward-facing, inclusive of difference, and orientated towards tolerance and justice.

How can data and algorithmic platforms infringe these rights?

Data and analytics platforms, including so-called predictive AI, have been criticised for undermining the rights of young people in education through the capture and unregulated use of their data, and through opaque decision-making that exacerbates discrimination in educational settings and amplifies inequality of outcomes (Global Connectivity Report (2022) Ch.9; Williamson et al (2023) for the National Education Policy Centre).

UNESCO’s report An Ed-Tech Tragedy? (2023) found that during the Covid-19 pandemic ‘education became less accessible, less effective and less engaging when it pivoted… towards technology’ (p34); that this was accompanied by ‘a concerning transfer of authority away from teachers, schools and communities and towards private, for-profit interests… the censorship, data extraction, advertising, top-down control, intimidation and surveillance that so often characterize current models of digital transformation have made education less free’ (p.35).

On behalf of the Council of Europe, a Briefing on AI in Education through the lens of Human Rights, Democracy and the Rule of Law (2022) concluded that ‘We need appropriate, robust regulation, addressing human and child rights, before AI tools are introduced into classrooms’ (p.75) and ‘We need to ensure that children are not forced to accept being compulsory research subjects or being compulsorily involved in product development simply by exercising their right to education’ (p.76). Recognising these risks, the Global Digital Compact of the UN commands: ‘digital technology companies and developers to respect international human rights and principles, including through the application of human rights due diligence and impact assessments throughout the technology life cycle’.

None of these regulations, due diligence processes or impact assessments have been carried out. Far from submitting to new requirements, AI corporations have bribed or bullied governments to weaken their existing rules and tax regimes. And the threats to learners’ individual rights - the focus of most existing legisation - must now be set alongside the systemic risks of an industry that has grown vastly more wealthy and powerful than most nation states, let alone their education sectors. At current valuations, OpenAI, Nvidia and Microsoft are each worth as much as the GDP of the entire African continent. These corporations are powerfully aligned with the US state and its global interests, and the same is true of AI industries in other powerful, militarised states such as China, Israel, Russia and Saudi Arabia. AI systems create a seamless geopolitical infrastructure that can threaten the educational rights of whole populations (the use of AI to support school and university bombings in Gaza) as well as vulnerable individuals (the use of AI to identify and deport students protesting these issues in the US).

AI as a geopolitical infrastructure has the capacity to inflict global epistemic and educational harms.A rights-based discourse must therefore extend beyond classroom infringements of children’s rights to concern itself ‘with the preservation of life and the co-responsibility of AI harms to the majority of the planet’ Ricaurte (2022)

What are the specific risks from ‘generative AI’

The risks outlined in earlier reports on technology in the classroom were immediately amplified by OpenAI’s release in 2022 of its untested, unreliable, unsustainable data architectures in the GPT series and its ChatpGPT interface. Beyond building and commanding an immensely profitable new market, the intent was to capture educational and cultural assets and to do so partly by causing immense disruption to education and to knowledge cultures and moving in to take advantage.

Many experts consider general models to be unreliable in principle (Luciano Floridi, John Lanier, Ragnar Fjelland, Iris van Rooij for example). Certainly they have proved unreliable in practice. But while the technical capabilities of these models are overblown, their capacity to concentrate data, computation, know-how and capital in the businesses that own them is unprecedented. The AI investment surge has further consolidated economic power in a handful of global corporations, power they have used to challenge international laws, labour laws and human rights frameworks (this last link to an Amnesty International report, 2025: pp.23-25).

Generative AI presents a number of immediate threats to learners’ rights, amplifying the known threats from so-called predictive AI (though the two are increasingly used together in educational systems and dealt with in educational policy as though they were one and the same). Channelling the knowledge available to young people through anglo-centric, proprietary and normative data platforms is a risk to their cultural and epistemic rights. Releasing AI-generated content into digital platforms and attacking public sites with AI crawlers, agents and deepfakes deprives young people of free access to digital information and culture, while diminishing their own opportunities for cultural production. These harms affect young people of minority languages and cultures to a greater degree. And there is increasing evidence (see below) that the use of chatbots in school work is harmful to young people’s intellectual and social development, and so undermines their right to the fullest development of their potential as laid out in the UN CRC.

Young people in education are therefore subject to three distinct threats to their educational rights:

1. So-called predictive AI is used to collect their data, to make unaccountable and discriminatory decisions about their futures, and to surveil and discipline their interactions within educational systems

2. So-called generative AI is used to capture and monopolise cultural and educational content, to advance hegemonic languages and perspectives, to degrade learning and to replace pedagogical relationships with automated agents

3. The narrative of ‘AI futures’ is used to refocus educational outcomes around the automation of intellectual work, stunting the development of young people’s capacities, and preparing them for work that will be precarious, exploitative, and algorithmically disciplined

Beyond risk: documented harms

While the risks have been widely speculated about, there are many actual and documented harms from generative AI to the rights of young people in education. These include:

• Children and young people being subjected to harmful biases, hate, fakery and misinformation through the mass circulation of synthetic memes, including via platforms used in educational contexts

• Evidence of generative AI use harming young people’s critical, intellectual and personal development, and amplifying biases and poor judgement

• Evidence that dependence on AI chatbots impacts on young people’s socialisation and emotional wellbeing, undermines relationships in the classroom, and weakens social cohesion beyond

• Monopolisation of educational publishing, testing and tutoring by AI corporations, producing new forms of educational, linguistic and cultural disadvantage and locking students into extractive relationships with AI

• Loss of credibility of assessment systems, leading students to question the value of qualifications and of education itself

• Direct attacks on higher education, teachers and ‘expertise’ by AI interests and industry representatives

There is also evidence of harm to the knowledge systems that education and educators rely on.

• Degradation of search, news publishing and political communication, not only by introducing biases, inaccuracies and fakery as content/data, but by disrupting and then drawing ito the AI business model every organisation that deals in content/data. Educational content and scholarly publishers are key targets, and there are threats to the maintenance of libraries, archives, heritage resources and public knowledge projects.

• Wealthy governments rolling back copyright laws in order to attract AI business; minority cultures and less wealthy governments have even less capacity to protect educational and cultural assets.

• Threats to creative and cultural work in majority cultures; minority cultures have even fewer opportunities to support their cultural industries, practices and people.

• Disruption of long-established epistemological norms such as the transparency and replicability of scientific discovery; open peer review; and the accountable authorship of published works.

• Challenges to scientific method (‘science can advance without coherent models, theory, or really any explanations at all’ Andersen 2008) and scientific freedoms.

Children and young people are unequally and inequitably harmed by:

• The build-out of hyperscale data centres, threatening power grids and water tables, piling toxic waste into vulnerable communities, and multiplying the carbon footprint of the computing industry

• Risks to the global economy through the hype, over-financialisation, under-regulation and state capture associated with the AI industry

• Failing democracies, partly from the impacts of disinformation and media capture, undermining the right of young people to determine their own futures; lack of accountability and learner voice in the use of data platforms in their education

• AI-powered warfare and surveillance: schools, teachers and students have been targeted for violence by AI systems; learners are widely subjected to surveillance through edtech platforms without consent.

Risk table

This table of potential risks AI presents to the right to education is a work in progress - you can read and comment here.

How might education leaders respond?

So-called ‘AI’ is antithetical to the UN goals of free, equitable access to learning and cultural opportunity. Education leaders should take a human-rights based approach to AI, not only as a class of technologies with known impacts on learners’ data rights, but as a crisis for education systems and their role in global peace and democracy. This crisis has been engineered by a small number of the world’s most powerful corporations in alliance with their state militaries.

The global response should be led by those most immediately affected: people of minority languages and cultures; people suffering from epistemic injustice (particularly at the hands of the digital and AI industries); teachers and education workers threatened with poorer conditions of work; and young people who aspire to the full development of their personalities and intellectual powers.

The AI crisis both amplifies and draws energy from other crises afflicting education and learners around the world, and should be tackled alongside them. Only with solidarity among students and educators, and across education sectors and systems, can the power of AI corporations be resisted.

The ten principles of UNESCO’s (2021) Recommendation on the Ethics of Artificial Intelligence are admirable but have not been adopted into law, let alone enforced as law, by any of its member states. The AI industry has attacked its own ethicists and bullied governments into abandoning reasonable attempts at regulation. Without enforceability, good principles only amplify the harms by appearing to have them under regulatory control.

Similarly, UNESCO’s AI Competency frameworks for Teachers and Students are worthy aspirations that can only be achieved with the use of safe, ethical, reliable, non-extractive, non-exploitative AI systems. Framing these aspirations as individual competences puts an unfair burden of responsibility onto educators and learners, while absolving from blame the powerful corporations that released rights-violating technologies into educational and cultural ecosystems.

Educational leaders should develop a shared action plan, drawing on a wide range of experts (independent of the AI industry) and oriented not on finding new use cases for AI in education (the AI industry will take care of those) but on prevention, mitigation and resilience to rights-based harms. Expertise will not be enough: only international solidarity can prevent governments and educational organisations being picked off one by one.

Responses might be grouped into three areas: ‘repurpose, rebuild and refuse’.

Repurpose, rebuild, refuse

Repurpose AI technologies, where possible, for projects of authentic learning and human flourishing e.g.:

• Develop small scale models using open data and APIs, publicly or community owned, that meet real local needs for learning and cultural expression;

• Design curricula that support inquiry into AI, its inequities and harms and wider contexts as well as its opportunities for use;

• Deliver learning activities with/without AI that reveal its fragilities, fabrications and breaking points:

• Teach and actively support counter-hegemonic data projects: feminist, decolonial, minority-cultural, public and sustainable projects, projects of data sovereignty and data justice;

• Undertake independent research into the social, cultural, developmental and longitudinal impacts of AI in education;

• Encourage slow scholarship and slow learning with AI, refusing speed and productivity to focus on quality and interpretability;

• In all of these activities, refuse anthropomorphism: describe precisely how models produce their outputs and how human agency is involved;

• Be alert to the constraints and contradictions that become visible in these repurposings: notice, teach, discuss and make known the mechanisms and meanings of AI

Rebuild cultural systems, practices and archives with resilience to synthetic media, e.g.

• Train educators, librarians, archivists and culture workers to keep content secure from AI bots and crawlers, and from AI-generated slop;

• Respect culturally-specific practices and make time for them in the curriculum;

• Maintain valuable cultural artefacts in non-networked digital repositories: use and renew them in the curriculum

• Create and protect spaces for learning that do not involve data systems, such as hands-on making and investigating, live writing (not necessarily assessed), active experimentation, sensory feedback, discussing and debating, live (unrecorded) performance;

• Secure education policy and the training of teachers from the influence of AI lobbyists and AI-funded researchers;

• Redesign assessments away from instrumental outcomes and standardised rubrics; negotiate assessment tasks with students;

• Build and sustain communities of open AI practice with shared norms, standards and values.

Refuse the impoverished, unjust and unsustainable educational future being offered by the AI industry, e.g.

• Refuse to use systems or architectures that may be harmful, biased, unsustainable or unjust;

• Refuse to provide or collect data that may be used to build harmful data architectures or to make harmful discriminations concerning learners and educators;

• Only collect learner data that can genuinely be used by learners and teachers to support their development; ensure it remains under the control of an accountable educational organisation.

• Refuse to adapt educational workflows and conditions of work to the use of AI, particularly where this will render education workers overworked, more precarious or less skilled;

• Refuse professional tasks that require the use of or contribution to a proprietary data system unless alternatives are available;

• Refuse personal and organisational partnerships with or investments in the AI industry

As every digital platform is ‘enhanced’ with AI co-pilots and agents, refusing and rebuilding are acts of ingenuity as well as resistance. Far from being signs of ignorance, maintaining spaces of non-mediated dialogue and cultural expression are rapidly becoming signs of technical and epistemic skill. Educators can foster these skills, confident that their learners will not be ‘missing out’ on AI because AI will always be making itself more useable: indeed, it is already compulsively useable, and it not use but non-use that needs to be actively developed. These approaches also call for solidarity among people, organisations and linguistic and cultural communities.

The outcomes of ‘embedding’ AI into education remain uncertain but they will not be produced by a merging of common interests. Rather they will be the outcomes of a struggle between incompatible goals: mass intellect versus mass thoughtlessness; diverse cultures and ways of knowing versus the normative power of data; participation in communities of learning versus the consumption of bite-sized answers from chatbot companions. In this struggle, UNESCO should weigh unequivocally on the side of children’s and young people’s right to the full development of their humanity.

With thanks to Bryan Alexander, Doug Belshaw, Laura Hilliger, Ian O’Byrne, and Karen Louise Smith for conversations that helped motivate and inform the writing of this piece.

Some time ago I invited Katie (Kathryn) Conrad onto a discussion panel in the Generative Dialogues series, and while this covered some essential ground in AI pedagogies, I felt there was a lot more of her work to dig into. So I was delighted she agreed to come onto Imperfect Offerings for a deeper dive, particularly into her Blueprint for an AI Bill of Rights in Education.

Katie is a Professor of English at Kansas University and co-director of the AI & Digital Literacy project, in partnership with the National Humanities Center, as well as Associate Editor of the journal Critical AI. Her Blueprint was one of the first things coming out of the AI tailspin that really spoke to me - and I increasingly think a human rights based approach is critical - not only in the negative sense that the inequities of AI threaten so many legally enshrined rights (equality, non-discrimination, freedom of thought and expression for example) but also in a positive sense.

We need something in which to ground our ideas about humanity, in opposition to definitions of ‘the human’ emerging from the AI industry as a kind of host species for ‘intelligence’, and then as an inverse or supplement or deficiency in relation to whatever ‘artificial’ intelligence is supposed to be capable of. Human rights is a way of thinking about being human means that - for all its imperfections - starts from our common vulnerability and dependence on each other, and therefore our equality. Human rights have a long history of collective thought and action, and shared institutions that seemed robust until quite recently.

So it was the connection between education and rights, and Katie’s intentions in developing the blueprint, that I really wanted to talk about. As ever, we ranged well beyond our original brief. Here are links to some of the resources we touched on.

Links

The Blueprint https://criticalai.org/2023/07/17/a-blueprint-for-an-ai-bill-of-rights-for-education-kathryn-conrad/

NORRAG Policy Insights: AI and Digital Inequities (including a chapter by Katie and Lauren Goodlad) https://resources.norrag.org/storage/documents/NllPZ3GRhnWCbiMFcG0tUv5qxOt4snLAVxpOwgsN.pdf

Katie’s Critical Digital Literacy resources: https://docs.google.com/document/d/1TAXqYGid8sQz8v1ngTLD1qZBx2rNKHeKn9mcfWbFzRQ/

Katie’s blog: https://kconrad.substack.com/

Artificial Intelligence in Education: a critical view through the lens of human rights, democracy and the rule of law: https://rm.coe.int/artificial-intelligence-and-education-a-critical-view-through-the-lens/1680a886bd

Pedagogies of Generative AI: podcast with Helen and Mark Carrigan from May last year, with Katie and others:

Critical AI journal: https://www.dukeupress.edu/critical-ai

Marc Watkins’ Rhetorica: marcwatkins.substack.com

Thresholds in Education https://academyforeducationalstudies.org/journals/thresholds/

Maha Bali’s blog on critical AI literacies: https://blog.mahabali.me/educational-technology-2/what-i-mean-when-i-say-critical-ai-literacy/

Report to the UN General Assembly on AI in Education: https://www.ohchr.org/en/documents/thematic-reports/a79520-artificial-intelligence-education-report-special-rapporteur-right

Harvard’s AI Pedagogy Project: creative and critical engagement with AI in education: https://aipedagogy.org/

The Data Sovereignty CARE principles: https://www.gida-global.org/care

Melanie Dusseau’s Burn It Down piece for Inside Higher Ed: https://www.insidehighered.com/opinion/views/2024/11/12/burn-it-down-license-ai-resistance-opinion

Roderic N Crooks: ‘Access is capture’ - on how edtech reproduces racial inequality: https://www.ucpress.edu/books/access-is-capture/paper

Audrey Watters returns to the pod for a chat about ‘AI realism’, and how we are all trudging up the long, weary slope of the AI adoption curve whether we believe in the promises or not. What will we find when we reach the ‘plateau of productivity’? Less work and more smarts so we can spend our infinite leisure time in cultural pursuits? Or a slightly more datafied version of the workplace we started from? Is it easier to imagine the end of the world than the end of AI hype? Are young people vibing with it? And what is ‘vibe coding’?

Get all the answers - well, some of the answers and a lot more questions - in this week’s podcast. And look out for more themed conversations with Audrey, including an episode on AI’s bad boys and why we love to hate ’em. Coming soon, if DOGE doesn’t get to us first.

Links

Mark Fisher’s concept of Capitalist Realism: https://en.wikipedia.org/wiki/Capitalist_Realism

Rob Horning’s piece on ‘vibe coding’:

Internal exile

It mostly works

The term “vibe coding” has been around for only a week or so, but it already has a Wikipedia page. Credit for its coinage is assigned there to one of OpenAI’s co-founders, Andrej Karpathy, who defined it in an X post…

Read more

a year ago · 60 likes · 3 comments · Rob Horning

Referenced from Audrey’s Second Breakfast newsletter: https://2ndbreakfast.audreywatters.com/ai-against-democracy/

The ‘water we’re swimming in’ as an analogy for ideology is attributed to a commencement speech by David Foster Wallace: https://www.theguardian.com/books/2008/sep/20/fiction

Remi Kalir is Associate Director of Faculty Development and Applied Research at Duke University and Associate Director of the University’s CARADITE Centre, where many wise ideas about student writing and reading are developed. Since ChatGPT first emerged, he has been working alongside students to understand the role Generative AI has and could have in their practice. He is also the author of two books on annotation as a way of linking student reading and writing, and empowering students in relation to academic texts. He finds annotation to be a ‘participatory act [that] marks public memory, struggles for justice, and social change’.

Remi and I discuss the need for ‘brave spaces’ where the purposes of education and writing can be talked about. In Remi’s words, trusting young people to work with us means being open about our own states of ‘not knowing’, before we can find collective ways ahead.

Links

Remi’s blog Remi(x)Learning https://remikalir.com/

Centre for Applied Research and Design in Transformative Education https://lile.duke.edu/caradite/: https://lile.duke.edu/caradite/

CARADITE centre’s resources for students on learning with AI: https://ai.duke.edu/ai-resources/learn-with-ai/

ReMarks on Power (2025) from MIT Press by Remi Kalir: https://mitpress.mit.edu/9780262551038/remarks-on-power/

Blog connected to the book: https://www.readingremarks.com/

Annotation (2021) from MIT Press by Remi Kalir and Antero Garcia: https://mitpress.mit.edu/9780262539920/annotation/

The hypothes.is project and software: https://web.hypothes.is/

Links:

Alistair’s newsletter/blog Reclaimed Systems:

reclaimed | systems

We are as lichen and might as well get good at it. Technologies, ecologies and knowledges

By Alistair Alexander

His Web site: https://reclaimed.systems

Alistair’s recent piece in the Berliner Gazette, ‘After Progress’: https://berlinergazette.de/generative-ai-is-degenerating-human-ecologies-of-knowledge/

The course Alistair mentioned: https://www.schoolofma.org/programs/p/early2025-ecologies-of-technology

The glass room website: https://theglassroom.org/

‘Materialising the virtual’, art project mentioned by Helen: https://we-make-money-not-art.com/how-artists-and-designers-are-materialising-the-internet/

Some more recent creative works designed to ‘make visible’ the invisible labour of AI: https://berlinergazette.de/projects/silent-works/

A recent post by Edward Ongweso Jr detailing the capital investments being made by the ‘big four’ in building out data centres (notice that the boss of Nvidia sees inference, not training, as the major driver of demand):

You might also like my recent post about the UK Government’s plans to turn the UK into a data park

And this earlier post about the climate costs of AI: Saving the Planet, one cute animal video at a time

Finally, you might want to listen (again?) to Dan MacQuillan on the podcast, talking among other things about the need to ‘decompute’.

No sooner had the Starmer Government’s Action Plan for AI hit the news, than Elon Musk was demonstrating what the AI take-over of state power looks like. Although the Action Plan vibes more Third Way than Third Reich, the injection of AI capital into the UK public sector is intended to be just as radical. The Tony Blair Institute, among the few global cheerleaders for Musk’s DOGE project, has its fingerprints all over the Plan: its beneficiaries are the same ‘Founders’ and ‘Frontier’ corporations that the Trump government is currently mobilising, or being mobilised by, in the corridors of global power.

Just after the general election I ran through a report called ‘Governing in the age of AI’, penned by the Tony Blair Foundation, Faculty Science Ltd (a UK AI company we will hear more about shortly) and ChatGPT itself. I predicted that this AI junk would prove irresistible to a Government in need of quick fixes for the public sector. And here we are, straight to the vein.

In the six months since that first report, the tech-to-public transfusion has been nursed along by Matt Clifford, AI venture capitalist and entrepreneur, Effective Altrustism fellow traveller, and the man credited with introducing Rishi Sunak to the CEOs of OpenAI and Anthropic. While penning the Action Plan, Matt also is/was (current status unclear) a personal investor in Faculty and the Hakluyt AI investment fund. He is of course a properly disinterested expert - why else would he play a leading role in the UK AI Safety Institute? (We’ll also be hearing more about the work of the AISI.)

TL:DR

So what is in the AI pill for every public ill, from potholes to planning delays?

1. I look at the lack of evidence for public benefit from the deployment of ‘AI’, even in some of its most compelling use cases.

2. Cutting through grift, the Action Plan has three items: ‘sovereign’ compute that is under foreign control, murky data pools, and ‘world leading’ safety regulation that leaves the UK a playground for the biggest AI bullies.

3. Big tech’s hunger for public sector contracts can tell us a lot about its sources of power and its weaknesses. I look at why the ‘AI’ project needs public money and public data so badly, and why the UK is such a good place to get it.

4. A recent contract for educational data, awarded to Faculty, shows the continuities between generative AI (content as data) and predictive AI (people and their behaviours as data). When data of the second kind is acquired by powerful corporations, operating in what the Action Plan calls ‘deep collaboration with the national security community’, we should expect it to be used to discriminate against people where they are most vulnerable.

5. With the Starmer government fuelling the same politics of resentment that have put Trump into the White House - and put Musk’s engineers into the infrastructures of the US state - it matters more than ever who has hold of public data, and how people in the UK are exposed to data-and-algorithmic power, here and elsewhere, now and in the future.

AI Action Plan: nothing to do with the public interest

Let’s start our comedown with a report from the Joseph Rowntree Foundation, led by Imogen Parker of the Ada Lovelace Institute (ALI). Looking hard for evidence of productivity gains and better policy making, Parker concludes that there has been almost no real-world evaluation of public sector AI. That’s right: almost nothing is known about how 'AI’ is actually being deployed, or with what results, and there seems little interest in finding out. This

lack of monitoring and evaluation of AI within the public sector is compounded by the opaque market environment that exists outside of it… The ability of public sector decision-makers – from procurers in academy chains and NHS foundation trusts to senior leaders in the Civil Service – to make effective decisions about AI is limited by this one-sided market.

A separate review by Bristol Policy (2023) concurred that ‘public buyers are not adequately placed to confidently and responsibly procure AI technologies for the benefit of citizens’ and are ‘under a structural conflict of interest due to their operational interest in the deployment of AI’. And then there’s this:

Despite the widespread use of computational predictions in social settings… there’s remarkably little published evidence of their effectiveness,”

That’s from Narayanan and Kapoor, Princeton University computer scientists and authors of AI Snake Oil (here reviewed by Edward Ongweso Jr). They argue that so-called ‘predictive AI’ is not just unproven in public use: when it comes to making decisions about human lives, messy and emergent and complex as they are, automated systems are flawed in principle.

In practice, things also look bad. The Guardian reports that after a year of intensive work with OpenAI, Anthropic, Google and Microsoft, the Department for Work and Pensions has abandoned most of its potential AI projects. The Guardian also separately exposed that an AI system in use by the DWP is systematically biased in how it identifies claimants to be investigated for fraud. So while it’s reassuring that so many bad ideas are being dropped, it’s not at all reassuring to learn that ‘AI’ is already deeply embedded into public decision-making, and is about to be massively accelerated, despite the mainly negative evidence of its value.

There are more clues in this somewhat testy evidence from the (short-lived) Head of the Whitehall AI Incubator - formerly in defence intelligence, though we don’t know if she is returning there. Explaining why so few ‘promising’ applications have progressed to actual use, she has nothing to offer by way of lessons learned or lowered expectations. There is only dismay at the ‘blockers, frustrations and false starts’ that have hindered AI adoption. The point of piloting is not, you see, to test the promise against real-world conditions but to push on regardless: ‘if something fails, we try, try again and find another route to impact’.

Ah yes, impact. My deep dive into ‘AI in education’ research last year found a large literature that has education as its context. But it is overwhelmingly a literature of technical developments being evaluated against technical benchmarks. Such studies are already biased towards positive outcomes - technical systems often do improve against benchmarks over time. But actual use cases are only speculated about, or a token test case may be tagged on with the goal of proving ‘acceptability’ in some well-suited context of practice (‘acceptance’ and ‘trust’ are all that’s needed for the benefits of AI to be unleashed). Meanwhile, evaluating the effects on actual people… is constantly deferred. The same is true of swathes of studies in medicine and the care sector: benefits/impacts exist almost entirely in the conditional tense.

At least one of the first Actions of the Plan is to fund a serious trial into AI support for breast cancer scanning. If you thought the benefits had been proven long ago, that is not exactly the case. Diagnostic imaging has been a promising application for decades with hundreds of studies undertaken, but these efforts have only very recently shown that they can produce efficiencies in real-world diagnostic settings (and there is still some controversy - there’s a further summary below if you’re interested).

The trial looks at whether the use of AI could replace one of the two radiologists that is currently required to assess each scan. This could, on the face of it, save some precious human resources. But meanwhile, the shortfall in trained radiologists is projected to be 40% by 2028. Diagnostics has been a target for privatisation and outsourcing, not least because it is so technology-intensive, and the Society of Radiographers points out this does not increase capacity for the NHS but introduces new inefficiencies in the deployment of staff and the handling of patient data. A group of leading cancer specialists wrote an open letter to Wes Streeting as he took over at Health, begging him to stop looking for a ‘magic AI bullet’ and start training more radiographers.

So the best outcome of this lengthy trial is that a few radiographers can be moved from one part of the diagnostic process to plug shortfalls somewhere else. And we still will not know how the workflow might have improved if that money had been spent on training radiographers.

With trials like this being lengthy and uncertain, the Action Plan cannot seriously hope to pilot ‘hundreds’ of AI applications for public benefit. Luckily for would-be AI contractors, adoption does not have to wait on the evidence. Chancellor Rachel Reeves is determined to find public sector spending cuts, and in services that rely heavily on skilled workers, this can only be achieved through automation and deskilling. That is the hoped-for magic bullet.

Dozens of contracts for AI and algorithmic/data services have already been awarded, including £330million to Palantir to run the NHS Federated Database and - before the diagnostics trial is even under way - a £2m contract to Faculty to:

pilot the NHS Artificial Intelligence Deployment Platform (NHS AIDP)… [a] standardised infrastructure for the deployment of mature and certified AI medical imaging technologies at scale in the radiology workflow.

As the diagram shows, the contract gives Faculty the right to process image data and diagnostic information between private providers of ‘AI’ and NHS Trusts. As part of this contract, Faculty was also asked ‘to assist in the evaluation and moderation of tenders’ from ‘mature and certified’ AI vendors. So the data infrastructure is already built, power is already being ceded to the owners of the infrastructure, and vendors are already being plugged in.

Remember this model of value extraction when we come to education. For now, let’s return to the Ada Lovelace Institute report and its radical idea that, if you want to discover the public benefits of a new technology, you need 'qualitative, relational and experiential understanding from both publics and professionals'. That is, you should listen to people working in the system who understand the problems and bottlenecks that ‘AI’ is meant to solve. You should listen to people on the sharp end of the service that ‘AI’ is supposed to improve. (And perhaps, just as an aside, productivity isn’t the only purpose of the public sector). But the AI Action Plan strikes through woke nonsense of this kind. In its drive to ‘rapidly pilot and scale’, the real stakeholders are not the ‘people’ at all, but the technology companies.

‘In every element of the action plan, the government should ask itself: does this benefit people and organisations trying to do new and ambitious things in the UK? If not, we will fail to meet our potential.

Compute: the UK as data park

Yes, so long as it is new and ambitious, the UK Government is in. So long as it is doing a thing in the UK, the UK Government doesn’t want to own the thing, manage the thing, steer it, tax it, regulate it, or ask what tangible benefits the thing is going to bring to the people who elected it. The UK government is just happy - very very happy - that anyone with ‘AI’ wants to do a thing in the UK. Especially if that something is a data centre.

By focusing on computer power and data infrastructure, the Action Plan does at least recognise the business end of the AI juggernaut. The computer processing requirements for generative AI, at every stage of training and inference, are mind-bending. In fact, venture capital has piled into energy infrastructure as well as data centres, anticipating how much new carbon will need to be burned to access all that promised value. Though demand for AI compute is increasingly understood to be an accelerant to the climate crisis, for those on the supply end it is a massive accelerant to profit.

Chip-maker Nvidia became the largest and most profitable company on the planet in 2024. Boss Jensen Huang wants to make Nvidia data centres the ‘intelligence infrastructure of the world’. Amazon Web Services and Google Cloud (that is, providers of computational infrastructure) are the only divisions of their respective corporations that are making serious money from AI. Leaving aside the question of whether the world can sustain such an investment, it would seem a sensible plan for any nation with ‘growth’ ambitions to have its own infrastructure. The Ada Lovelace Institute (again!) in an April 2024 report on the Role of Public Compute, pointed out that:

As the compute intensity of cutting-edge AI research has increased… the participation of smaller firms and academic centres in AI research has decreased… allowing a small number of leading companies, often funded by venture capital, to effectively monopolise AI development. Venture-capital-funded development needs to provide exponential returns, which pushes towards building winner-takes-all products that can provide significant profits in a short period of time.

If the Government really wants to support UK start-ups and UK research, it needs to reduce dependence on these corporate giants. So welcome ‘StarmerGate’ - naturally on a more modest scale than Donald Trump’s $500 billion Stargate project - to:

‘Streamlined planning’ means preferential access to land and ‘accelerate the provisioning of clean power’ means preferential access to energy.

But here’s the problem: none of it will necessarily be owned by the UK public. Even the category of ‘sovereign compute’ (‘the smallest component’) is only meant to be ‘owned and/or allocated by the public sector’. The rhetoric is undoubtedly supposed to tickle post-imperial fantasies, talking up the UK as ‘an AI maker not an AI taker’. But in the press release boasting of the £millions of inward investment, the infrastructure companies are not from these shores. One UK-based cloud provider even complained in the FT (paywalled I’m afraid) that ‘by continuing to depend on overseas providers, the UK risks regulatory conflicts, security vulnerabilities and a fragile AI ecosystem dominated by external interests’.

The truth is these data centres will be ‘based within the UK but privately owned and operated’, with profits flowing to the host companies, overwhelmingly US tech giants. If Trump and his best buddies force Starmer to give up on the Digital Services Tax - and things are moving so fast, it may have happened by the time you read this - it’s hard to see what benefits will remain for the UK.

Because, aside from the empire-in-rags rhetoric, the UK is a puny player. The EU is investing 1.5billion euros in supercomputer sites, and already questions are being asked about the sustainability of this: chips need to be replaced every 8 months or so, and it’s doubtful any European AI company can generate enough revenue to justify the costs. The Sovereign wealth fund of Saudi Arabia recently entered a partnership with Google Cloud services worth hundreds of billions of dollars. But even in Saudi, where ‘sovereign wealth’ betokens some serious global power, Google will retain full control of the models and the data centres.

In a sign of just how craven a client state the UK has become, the Government recently caved in to demands by the AI industry to exempt AI model training from UK copyright law. Minister Patrick Vallance had the unenviable job of explaining to a posse of national treasures, from Kate Mosse to Kate Bush, why he wants to ‘support the development of world-leading AI models in the UK’ rather more than he wants to support the UK’s genuinely world-leading creative industries.

By contrast, when ‘Stargate’ crowds private and foreign state capital into US data infrastructure, there are obvious benefits to the US state. Massive subsidies for OpenAI, Oracle, Microsoft, Nvidia and Arm to continue their domination of the global AI stack. Support for the techno-dollar as the world’s reserve currency. Capture of data and content from the rest of the world. And all the geopolitical and strategic advantage of that accumulated data, capital and expertise. Even before the recent government-AI merger under DOGE, this Forbes article (approvingly) notes:

Trump is aiming to use the country's huge store of energy reserves to power growth, especially in areas where the U.S. is trying to secure global dominance. The infrastructure will provide crucial capabilities for national security applications, such as cybersecurity enhancements, autonomous defense systems, and AI-powered surveillance.

There is no obvious pay-off to the UK from parking those data-and-surveillance engines on our soil. Like other things the US has parked here - cruise missiles and Elon Musk’s politics for example - data centres are not a sign of strategic influence but of how far the UK must bend to accommodate its more powerful ally. Of course it is hoped that all this ‘compute will create spillover benefits in the form of jobs, investment and new, AI based, service businesses’, but ‘spillover’ acknowledges that it won’t be data centres themselves that create the start-ups - that will require entirely different kinds of investment. Most of the jobs data centres create are in their construction, because the whole point is that they are investments of capital: very few people are needed to keep the GPUs burning.

On the other hand, data centres have significant costs. Globally, they will soon outstrip the power demands of most nation states. Every kilowatt of that power requires 2 litres of clean water for cooling. These demands lead to local droughts and power outages, as well as toxic piles of e-waste as generations of chips are successively burned out or superceded. Data centres are being resisted by communities in many parts of the US, and by communities in Chile and Uruguay and Mexico where (naturally) the ‘frontier’ companies went prospecting for cheap land beyond their borders.

Closer to home, in Ireland, data centres have been using more power than urban homes since 2023 and are threatening to derail that country’s climate goals. The UK has been a small player until now, but The Raconteur still reports that:

In 2021, Slough effectively ran out of power. The construction of a new substation was required to meet [a] data centre’s growing electrical power demands. In 2022, the Greater London Authority warned that the building of new houses in West London had been hampered by a lack of energy capacity, as power was guzzled up by nearby data centres. And in 2023, Thames Water threatened to restrict the water services provided to data centres if they did not curb their consumption.

Let’s notice that the target areas for the new data parks are those ‘post-industrial towns’ where land and construction labour are cheap and there may be less organised opposition. At least some of them will be located in the Freeport Zones - free from local taxes, workers’ rights, safety rules and public transparency - that Starmer’s Government quietly inherited from its predecessor. And let’s consider what ‘coastal Scotland’ might have to offer. Coastal Scotland where, despite a complete lack of respect for energy transition agreements, the Government seems minded to approve a massive new oil and gas drilling field, and to honour every existing licence to ‘drill, baby, drill’. Coastal Scotland is where all that fossil fuel comes ashore.

It’s true that some of the new compute - the smallest ‘sovereign’ part - could be used for projects the Government deems to be in the ‘national interest’. This allocation, says the Action Plan:

could be modelled after the Defense Advanced Research Projects Agency (DARPA) or the Advanced Research and Invention Agency (ARIA) to quickly and independently provide large amounts of compute to high-potential projects of national importance, operating in a way that is strategic and mission driven.

The reference to DARPA - a give-away for the military origins and use cases of ‘AI’ - is also a gesture towards what ‘national importance’, ‘strategic’, and ‘mission-driven’ are likely to mean. But linking DARPA with ARIA, the UK innovation agency headed by one Matt Clifford, is hardly a sign of the UK’s equal status. ARIA receives less than £10million in funding from the UK state, while DARPA’s budget last year was over $4billion. So whatever Matt Clifford’s ambitions, there is not much ARIA can offer to counterbalance the demands of the real global hegemon here.

Murky pools: the UK as data source

Other nations have land and power supplies they are willing to trade for AI dollar, but the UK has something rather special. Our public services are among the longest and best established in the world, and this provides big tech with two resources: a captive market for data services, and a rich source of public data.

As the ‘best state partner’ for big tech, the UK certainly intends to be a captive customer.

So public sector ‘piloting’ should not be seen as providing an evidence base for the public interest but as a wedge for AI adoption. Feedback from the public may produce some improvements. but once adoption is locked in, the incentives to improve are small. Rather, public sector use cases drive adoption into more lucrative markets such as health insurance, education services, private security and private care homes.

At this point we say hello (again) to Faculty Science Ltd, the company that first rose to attention as the AI partner of the Vote Leave campaign. It has since built a sizeable portfolio in defense and security and other UK government data contracts, including - ironically - helping to tackle ‘disinformation’. We last heard of them in the context of Rishi Sunak’s Safer AI Summit, where for several hundred thousand pounds Faculty ran a two-day chat-athon about AI in schools. In their own write-up of this event, Faculty staff seemed confident of further funding to ‘try out’ some of the use cases they identified. And here in the Action Plan, as one of the ‘great examples of AI use-cases driving tangible benefits across the private and public sectors’, comes the news we have been waiting for:

the Department for Education’s generative AI and rules-based marking tool showed 92% accuracy in a pilot with teachers on year 4 literacy work when drawing from appropriately coded educational data and content ^{[footnote 15]}

As the only educational use case it does seem rather lame. And the embedded link goes to an entirely different report. But after diligent research I turned up the promised Use cases for generative AI in education: technical report, and was rewarded to discover that it was indeed Faculty that developed the ‘rules-based marking tool’ and wrote the elusive report. Though I’m afraid it is so bad I’ve left the details in a footnote, to be read only when you have a strong antidote to despair close by.

The point of these exercises in futility is not to build a viable product, and certainly not to find out what teachers need to help with their workload. It is to make the case that AI is essential, and can only be developed with huge piles of educational data:

it is important for [pupil] work to be assessed in both the context of other work from the pupil and, in comparison to the rest of the class or year group… collating these insights over a longer period allows teachers to more effectively monitor progress at both an individual and class level

It’s almost as if learner analytics never happened. Or rather, it’s as if learner analytics has not been promising exactly this for two decades, during which time teachers have gone from a few hours’ admin a week to incessant data work that is displacing time spent with pupils and driving many out of the profession. And doing all of this with AI will require even more data work:

including explicit structures and markers that AI tools can recognize and interpret is critical to enabling [documents] to be more effectively used by AI-powered tools. This could mean tagging learning objectives, error types, and correction suggestions in a standardised format that AI applications can process.

Of course Faculty is not alone in demanding more data, and more people doing more data work, so that they can make everything better ‘for people’ while keeping hold of all the data. We have heard the Tony Blair Institute demanding that every classroom becomes a ‘real time data environment’. Another public-spirited organisation, Google Cloud, recently paid Public First to assess the benefits of AI in the public sector, using the same method as the TBI (asking ChatGPT to analyse job classifications and running the results through Tony’s fantasy calculator).

And this bounty will only flow with proper homage to the data gods: ‘modernis[ing] Government IT infrastructure to ensure that AI is built on secure, efficient cloud computing foundations’ (like Google Cloud, perhaps) and ‘link[ing] together public datasets’ through common access points.

And this is where we come to the second big goodie bag the Action Plan is offering. Because if there is one area the UK does punch above its weight, it is in public datasets. The UK population is similar to the US in terms of health and disease, education systems, language and media consumption. But because - unlike the US - we have had a social state for much of the last 75 years, we have the largest bank of healthcare data in the world, and well-established record systems in education, social care, justice and publicly-funded research. All prized resources for private corporations that operate in those fields.

The Action Plan proposes a ‘National Data Library’, which sounds as though members of the public can pop along and take out some data in our lunch hours. And the Open Data Institute does in fact have a user-first vision for the Library with open standards, public governance, and ‘giving people and communities meaningful tools to have control over how their data is used’. But when the ODI asks, rather pointedly - ‘who are the key audiences for the Library - researchers, AI developers, public servants or members of the public?’ - only one of those groups, I’m afraid, is in the frame.

The Federated Data Platform for NHS data was awarded last year to Palantir - the company established by Peter Thiel to build military spyware and security tech. This contract has failed to win the trust of doctors, or of NHS patients, and The Good Law project contests that it was not given very ‘careful consideration’ either. The Faculty contract for medical imaging data is meagre by comparison, but the two companies have ongoing partnerships, as well as striking similarities in their military and security origins, and a shared model of playing their ‘security’ credentials to win contracts for sensitive public data.

And Faculty is now fishing in a new data pool. In August 2024 - just as its ‘use case in education’ report landed - the company was awarded a contract worth up to £4.5m, to create an educational ‘content store’:

including curriculum guidance, lesson plans and anonymised pupil assessments which will then be used by AI companies to train their tools. [Faculty will] develop the technical blueprint for a secure content store … with the capability for differential access and charging requirements should those be required.

Meanwhile, it is for other companies to develop the actual products:

You have to wonder what these companies think of being invited to Faculty’s data party, and asked to make tools to Faculty’s recommendations (for a fraction of the money). Still, we educators know nothing about the free market, so we will just have to trust that this is all in the interests of innovation.

Meanwhile, educators should be concerned about the ambition to include ‘pupil content’ and ‘anonymised pupil assessments’ in the data pool. There is at least a plausible reason for National Curriculum content to be used to augment the generation of lesson plans and marking schemes. What this specification seems to envisage is for pupil work to be assessed individually, by the same systems/data architectures that use that pupil work as training data. The error-in, feedback-out pipeline of Faculty’s proof of concept, despite its gaping flaws, may at least reflect a genuine ambition: that old dream of the personalised teaching engine.

In AI Snake Oil, Narayanan and Kapoor make a distinction between ‘predictive’ AI, that uses data about people and their behaviour, and ‘generative’ AI that uses cultural content. While cultural appropriation has many risks, especially for people of minority cultures, the impact on individuals may be indirect. Predictions based on personal data, on the other hand, have obvious and direct consequences on people’s quality of life: whether they can get insured, whether they are targeted for arrest or investigation, whether they are identified as ‘falling behind’ relative to the national curriculum standards.

You’ll have noticed that the Faculty content store is designed to manage both kinds of data. And there is a clue here. As discussed in other parts of the blog, generative AI use cases are not nearly valuable enough to justify the epic expense of training and maintaining the models. Student essays, boilerplate code and SEO pap are not going to cut it. Either specialist models must be trained or generic models must be retrained with codified expertise if they are to automate anything of actual value.

There is an older, simpler and more proven value proposition, and that is just to get hold of social data. If that data relates to people’s purchasing or political behaviours or media consumption habits, it can be sold on. If it relates to people’s interactions with health, education, care, the criminal justice system, it can be used to model risk within those systems. It can also be used to build alternative, shadow state services in which managing people as risk is the point rather than the method. The fast-fading sheen of generative AI is being used, it seems, as cover for social automation, social discrimination, and control of social data.

Regulation, deregulation, misregulation and safety testing: the UK as cheerleader

So we come to the third aspect of the Action Plan: ‘world-beating’ AI regulation. And this is where the doublespeak really goes full Orwell. Ignorance is knowledge. Regulation is deregulation. ‘Safe’ AI is ‘military’ AI. As if deregulated tax, deregulated planning and deregulated public data were not enough to propitiate the AI gods, the UK is doing its best to help deregulate the rest of the world. When I started this post, despite intense lobbying, the EU had shown at least some regulatory resolve in key areas such as data privacy, energy efficiency and AI safety. As I conclude it, threats of a military nature by the Vice President of the United States, a man who owes his entire career to libertarian AI Founder Peter Thiel, have led the EU to withdraw what was left of its AI Act: effectively to give up on regulating tech capital in the interests of European citizens.

Did it make a difference that the UK was the only other country to stand with Musk and Trump against the Paris declaration on open, ethical, secure and trustworthy AI? We will probably never know.

Again, when I started this post, the UK at least had something called an ‘AI Safety Institute’ that I was fully prepared to satirise. But reality outruns satire. Peter Kyle has just agreed to rename the AISI the ‘AI Security Institute’ and ‘shift[] the UK’s focus on artificial intelligence towards security co-operation rather than a “woke” emphasis on safety concerns’.

The most sympathetic reading of the Action Plan (as detailed here by James Smith on Mastodon) is that it promised to do nothing at all on regulation, pending further developments. But those further developments are very much under way, and Starmer has wisely observed that it is not enough to be a bystander when the bullies own the playground: anyone who does not join in is going to get their head stamped on.

In truth, the UK has dogged the US at every turn, allowing AI companies to sign up to voluntary commitments on safety, and ensuring the ‘AI Safety Security Institute’ was deeply embedded with industry interests. Remember the role of Faculty in helping to select those AI applications ‘mature and certified’ enough to play in their data pools? You wondered, I’m sure, who could provide this kind of certification. And here are two contracts for Faculty (worth £1.3million) to conduct safety testing for the AI Safety Security Institute, in close partnership with the much better endowed US AISI (it really doesn’t matter at this point if you can’t tell them apart).

As the Government’s own note makes clear, ‘safety evaluation’ is a purely technical exercise that compares the performance of new models with existing benchmarks. TheOpenAI o1 model, for example, outperforms the ‘reference’ model on cybersecurity tasks, but the report from UK/US AISI only notes that these superior capabilities ‘can be leveraged to strengthen cyber defences and can also be used maliciously to exploit a computer system’; so it’s not clear how ‘better performance’ translates into ‘better safety’ for users and data subjects. Faculty’s founder, Marc Warner, is not keen on doing even this level of testing. When the contracts were awarded he opined that there was not enough capacity at the UK AISI: rather than being ‘on the hook’ for doing the actual work, his goal was to ‘set standards for the rest of the world’: presumably the same standards agreed by the US AISI with the US AI industry. Since that august body seems now to have been abolished, it would be interesting to know whether the UK AISI is even in possession of its own rubber stamp.

Technical evaluations of model performance, in any case, say nothing about the harms that models may do in real situations. Here is a nice visual summary of the issue, from a recent paper by Gabriel Mukobi at Berkeley.

The gaps between technical evaluation and social mitigation may seem theoretical, but a quick read shows that we are all living through the contents of the first and second gaps, and many people are already plumbing the ‘Catastrophic’ depths of ‘Possible Harms’.

Standing with Elon Musk and JD Vance against ‘open, transparent, ethical and sustainable’ AI is one approach to world-leading regulation, though perhaps not the best one for avoiding Harms. No doubt eager to obscure this capitulation behind a more populist issue, Peter Kyle was sent forth (again) to trumpet that the UK would be the first nation to create a specific law against AI child abuse imagery.

Just to be clear, creating sexualised images of children (and other people) with AI is already illegal, but the legal system is unable to keep up with the tsunami of violent, degrading, fake and exploitative images that has been ‘unleashed’ by synthetic image generation. In the last decade of relentless technological upgrade there have been cuts of 22% in public funding for justice. So the problem is not a lack of laws. Not even the Internet Watch Foundation, that campaigned for legal changes around AI-generated images, thinks these particular changes are going to make a big difference. How will ‘optimised’ AI models be defined, when a generic model can be optimised with a prompt injection, or a database of private images? And has anyone written a ‘manual’ for doing technical stuff since, I don’t know, the 1990s? If accessing the relevant threads on Telegram, Reddit etc etc is too difficult, ChatGPT might be able to help, now that OpenAI had agreed to ‘ease up on AI paternalism’.

I notice that people who spent 2023 reassuring us about guardrails have pivoted to guardrails being childish nonsense. I notice that, once again, it is the ill-defined activity of ‘building AI tools’ that is prohibited and not using ‘foundation’ models to do exactly the same thing. Because, what with OpenAI’s new ‘grown-up’ approach to safety, Meta dissolving its Responsible AI Team, Musk launching an ‘even more uncensored’ Grok, and the Trump administration throwing out Biden’s already weak-as-piss AI safety rules… foundation models don’t really have much left to stop bad actors using them in the worst possible ways.

This is not a drill

Though the media class on this side of the Atlantic seems to be in a state of fascination / dissociation about events over there, the parallels are striking. In Starmer we have a weak, technocratic leader whose tech projects will fail to deliver any service improvements, but will put oodles of public data into private hands. The profits from managing this data will be committed to the kind of politics that keep tech capital ahead: tax and legal loopholes from the state, and the micro-politics of resentment, ignorance and white entitlement, amplified through deregulated social networks. Musk-backed or self-propelled, Farage is waiting in the wings.

It does not take a paranoid imagination to join the dots. There are direct links between AI in the public sector and projects of military and security control: between Palantir having access to your sexual health data and trans people being harrassed in US jails; between OpenAI writing lesson plans, and helping to target civilians in Gaza, Syria and Lebanon; between Musk enshittifying twitter, Musk withdrawing Starlink support from Ukrainian troops when the mood takes him, and Musk denying medical services to tens of thousands of people unfortunate enough to depend on USAID. Google Cloud and Azure are running your university services today, and tomorrow they are doing whatever Pete Hegseth decides they should do, in the interests of US billionnaires.

There is still time for the UK to wake up and clean up from AI. None of this is in the public interest, and it represents massive opportunities for vulnerable people to be harmed. No more public data should pass into private hands without robust, transparent and well-funded safeguards of public accountability. Institutes like the ALI and ODI have practical solutions for this, independent from the AI industry. And it should be obvious from events over the pond that this is not a drill.

Subscribe now

Aside on two UK figures that link several parts of this story

Dominic Cummings started out in AI, famously used Faculty’s services to engineer public opinion behind the Vote Leave campaign, and moved seamlessly on to being the power behind the Boris Johnson throne. After a rapid fall from grace over Covid 19 scandals and lies, these days he can be found advising Elon Musk on UK politics and small Government, and apparently telling Starmer’s Labour party how to crush crime and build infrastructure in partnership with AI.

Lord Peter Mandelson, whose company Global Counsel manages PR for Palantir, is reported in the FT to have advised Starmer to ‘build relations’ with Elon Musk using Nigel Farage as a ‘bridgehead’. As UK Ambassador to AI, sorry, the USA, Mandelson claims to have brokered the ‘signature policy’ of making the UK Safety Institute an offshoot of the US Security State, because the ‘backbone of the special relationship between the US and UK lies in its defence, intelligence and security partnerships’. This is a good fit with his previous adventures spearheading UK profits from the reconstruction of Iraq after another US-UK defence partnership devastated that country.

Aside on public sector productivity

If we cared only for productivity in the public sector, we should still pay attention to the productivity paradox - known and studied since the 1980s, recognised by workers everywhere - that computerisation does not improve productivity. This has been confirmed with AI applications in industries as diverse as construction and agriculture; a recent Upwork study found that 77% of employees in companies deploying AI felt they were working harder. Recent work by economist Daron Acemoglu and colleagues has found that across industries, automation produces modest productivity gains in some areas, but large increases in inequality and precarity across the board.

So when Pat McFadden opines that ‘AI could be the key to improving public sector productivity, which has been flat for two decades’ (actually it has fallen for three) either he has missed all the data and algorithmic management that has been injected over that time, or he really does think AI is something completely different. No relation at all to Tony Blair’s first e-revolution, including the ill-fated electronic patient records system that has still to be delivered, or the plan for digital identity cards that Blair is still energetically touting. No relation to any of the botched public IT projects of Blair’s first tech revolution that the Independent reported in 2010 had cost the tax payer nearly £30 billion.

But perhaps we should not see productivity as the main goal of government. As Eryk Salvaggio argues in his Anatomy of an AI Coup, the entire rationale for Elon Musk’s DOGE :

is based on a productivity myth that the goal of bureaucracy is merely what it produces (services, information, governance) and can be isolated from the process through which democracy achieves those ends: debate, deliberation, and consensus

The public sector is supposed to contribute to people’s quality of life, and (within the many limitations of a mass system) the capacity of those same people to have a say over how ‘quality’ is defined and produced. Quality is always debatable, in the best sense of that word, but efficiency (speed, scale, standardisation) might well be antithetical to it. Caring attention is not scalable. Respectful interaction can’t be standardised. And public accountability is clearly wasteful, compared with the kind of tech start-up / monarchy / reich in which the Founder gets to decide how everything is run. Politics is like that.

It’s the opportunity to be involved with the quality of people’s lives that draws most people to work in the public sector. It’s hard to know why else anyone would put up with the low pay, long hours, and being on the front line of social break-down. Doing public sector work is not ‘efficient’ from the perspective of its workers’ own economic self-interest or personal thriving: what if ‘care’ is, precisely, the surplus you bring, over and above the transactional demands of the job? If you have been involved in community work for a long time, as I have, you will know that providers and recipients are not two sides of a transactional arrangement but roles that everyone plays at different times, and often at the same time in different ways (Covid would not have been survivable, for many, without mutual aid). Of course a public service is also a ponderous bureaucracy and a state enterprise, an instrument of social control and a risk management facility, in which this kind of reciprocal community is a distant ideal. But I suspect it is the ideal that keeps people working for it, and the whole thing tottering forwards.

Aside on diagnostic imaging

Diagnostic imaging '(‘cancer scans’) is the use-case of choice for every AI champion looking to tug on the nation’s heartstrings - and I apologise for returning to it again. I’ve no expertise (obviously) in radiology, but I can read research. And researchers in quality journals like Nature express frustration that real-world clinical testing lags so far behind technical development; that improved ‘accuracy does not equate to efficiency’ even in highly specialised workflows where AI has been carefully integrated; and that bottlenecks in cancer diagnosis and care could be fixed far more quickly by investing in the workforce than by ‘novel’ technologies that take years to develop, refine and integrate.

A recent review in Nature should be required reading for everyone who believes that ‘AI is curing cancer’ (Peter Kyle, for one example). The reviewers looked at 48 clinical trials, of which only one did not show bias in the study design. Then they undertook a meta-analysis of groups of comparable studies and concluded there was ‘no evidence of AI tools reducing the time on imaging tasks’ in real-world use. Of course data can support clinical decisions - it has been doing for decades - but only as part of a complex workflow involving many decision points, many kinds of expertise and many patient interactions. Machine learning data is costly to bring into use, sometimes requiring specialist scanners as well as specialist data training, and that all comes at the cost of other investments - the actual cancer treatments, the ongoing human costs of care, or the training of highly expert oncologists and cancer nurses.

Commenting on the new £11m NHS trial, the president of the Royal College of Radiologists recognised the potential but:

“while AI holds great promise, it also carries major risks”, she added, acknowledging concerns that AI algorithms would need to be very carefully designed so they produce equally reliable results for different groups of women regardless of their ethnic background. [The issue here is not only the data that AI systems are trained on: if AI detection leads to a few early cancers being identified even earlier, that may only shift resources of treatment towards patients who already present earlier.]

She also voiced concern that, while the trial is not due to produce results for several years, the longstanding shortfall in consultants in the NHS is expected to reach 40% by 2028.

Questions for journalists

I’m not a journalist, I’m an exhausted academic, but I do hope journalists who cover these issues are asking questions like:

• What does ‘allocated by the public sector’ actually mean in terms of national computing infrastructure?

• How will public interest uses in education, health and infrastructure be ring-fenced and prioritised?

• How will profits be taxed?

• How will any AI solutions developed with UK compute be regulated and made accountable to the UK public? Under what legislative regime will data be processed?

• Will UK AI researchers be able to access public compute without compromising their independence through partnerships with big tech?

• Will the National Data Library be managed and governed under the kind of open and accountable rules set out by the Open Data Initiative? If not, who will manage and govern public data in the interests of the public?

• Does the UK civil service have the expertise to commission and manage such infrastructure projects in the public interest, or will management continue to be outsourced to the tech sector itself?

• What investments are being made in the UK water supply and power infrastructure to protect the public from the impacts of increased demand?

That User Research from Faculty: a Titanic fail

Use cases for generative AI in education: technical report is cited by the AI Action Plan as showing ‘92% accuracy in error detection with teachers

• The 92% success rate refers to the best error detection achieved in a technical trial. This was with a synthetic dataset of single sentences, to each of which a single grammatical or spelling error had been added.

• No real student essays were involved, or real teachers marking them.

• The report is a classic of the genre I’ve already described - the ‘technical development, technical benchmark, token use case’ pilot - but there was no technical benchmark. Rather:

‘the process for inserting errors from the NC [National Curriculum] into the essays [sentences] is essentially the error detection pipeline in reverse. Both pipelines were developed simultaneously.

• So, essentially, the developers have prompted ChatGPT to find a small number of basic errors in short sentences - that they introduced themselves by the same process - and this was done with less than 100% accuracy.

For contact with actual teachers (possibly as many as 18 of them) you have to track down an even more elusive Faculty report, Use cases for Generative AI in education: User research. This is so bad, I’ve left the details in a footnote. Please only read it if you have a strong antidote to despair close by. I finished the technical report and user research report from Faculty very confused about who had participated in their research, how many of them, and what they had participated with - whether it was ‘examples of feedback the tool had generated’, wireframe screens (as included in the report itself) or a chance to ‘interact with the tool directly’ in some way.

Before anyone got to interact with the tool there was a survey (respondent groups not clearly defined) that started out like this.

If I was being picky, I would suggest that introducing ‘a magic tool’ for saving time and then asking about AI doesn’t really avoid respondents ‘being biased by preconceived ideas’. But no further mention is made of the survey, so let’s move on. The use case chosen to progress to Proof of Concept is ‘giving feedback on student essays’, and the PoC seems to be the same technical development described as giving 92% accuracy in error detection using synthetic data and a two-way error/detection pipeline. In the technical report, 6 teachers or ‘super-users’ get to interact with the tool, though the 3 ‘responses’ they each generate are reported as though there are 18 teachers involved. In the user report (try to keep up), the number is 8 teachers (the same? different?) while 5 ‘AI leaders’ and a further 12 teachers are interviewed, some but not all of whom have encountered the tool. Nine students are just told about the tool and asked if they think it is a good idea: on the whole they don’t.

Confused? Don’t worry, because the generated outputs included in this report are so bad that we don’t really need to worry about what a cherry-picked handful of AI-inclined, teaching-adjacent participants made of them.

The PoC uses OpenAI’s GPT4 to provide the interface and language model, and injects additional data in the form of the National Curriculum specifications for Year 4 literacy (that can be found here), and a number of documents about feedback practice. Two essays from the NC specifications have been used to generate sample essays and feedback on them: a short descriptive piece about the Titanic and an informative piece about pointe ballet shoes. The Titanic piece in the original is vivid and imaginative:

Wealthy people smiled at their beautiful and expensive accomodation while the poor threw their bags on their bunk bed all squashed.
While you look down on the floors and look on the walls and chairs you could see a sparkly clear reflection of yourself. It was so clean you could kiss the floor.

While there are a couple of grammar and spelling issues here, the NC specification does not really focus on them. Instead it provides (or models) detailed, perceptive feedback on the writer's choices and skills. Faculty’s version of ChatGPT generates this:

Having found the errors inserted for it to find, the model concludes that what this student really needs to practice is the formation of regular plural nouns by adding an -s. Even in year one, few pupils have problems with forming regular plurals in English, and this student can correctly use the adjectives ‘well-spoken’ and ‘impeccable’. So a real teacher and not a chatbot with an error pipeline would have concluded that any missing ‘s’ was a slip of the pen.

There’s also a suggested exercise for this student, which is the function of this PoC that supposedly goes beyond what ChatGPT could provide through naive prompting with student work.

You’ll spot that there are a few problems with this as an exercise. In a. what’s actually wrong is the number, not the tense of the verb (‘the wealthy people smile’ is fine, and so actually is ‘the wealthy person smiles’, unless from surrounding context you know that the sentence should be in the past tense: but you don’t, and anyway, why confuse the tense with the number issue?). There’s also a problem with the number of a different noun (‘accommodations’) if you are a speaker of British English. In b the problem is not with the verb but with the adjective/adverb. And c. is not incorrect at all.

Meanwhile the writer of the pointe essay, who also has bizarre basic errors introduced to their text, is directed to spelling practice:

Even if these were all real errors (only the last is in the original) and even it were plausible that a student capable of writing at this level could not spell christmas, and even if ‘beautifully (used as an adverb)’ were incorrect… presenting wrongly-spelled words and asking students to guess what they are supposed to spell is not a recognised approach in literacy teaching.

Remember that these are the two examples that the developers actively chose to present as a proof of concept and to publish in their report. As they admit themselves, even this quality of response:

derives from the inclusion of education-specific feedback guidance within the training data of these models, possibly sourced from across the globe. It should be caveated that this can only be said confidently for feedback on Year 4 English essays, and quality is highly likely to decrease for older year groups and other subjects.

Of course there are far, far better applications for correcting spelling and grammar - you are almost certainly using one now. And that may explain why both developers and participants assume the quality of these outputs is acceptable: everyone knows that computers can do this stuff and no-one is paying much attention. But the proof of concept actually has nothing to do with spelling and grammar: the proof is supposed to be of helping teachers to give personalised feedback on student work. The fact that it doesn’t work for the most favourable use case for automation, further reduced to basic error production/detection pipeline, does not seem to get in the way of it being cited as the basis for the Government’s strategy on AI in schools.

The most punishing irony of all is that it was Dom Cumming’s friend Michael Gove who, as Education Secretary, drove through sweeping reforms of the UK literacy curriculum towards a rigid focus on ‘correct’ grammar and spelling. Gove did away with the ‘woke’ nonsense of having children acquire grammar naturally by using language - in ways that make sense and stimulate and are enjoyable to them - to insist on learning ‘the rules’ of such things as ‘frontal adverbials’, an approach that even Gove’s original advisers agree was utterly mistaken. The evidence remains as it always was, that teaching grammar does not improve students’ writing.

But meanwhile these reforms have meant that access to creative subjects has been ‘devastated’ for those children subject to the National Curriculum’s demands. Children’s enjoyment of reading has plummeted. And the Gove project of restoring ‘grammar’ schools and faux grammar schools as the gold standard of English education has amplifed educational inequalities through the school system.

A belief in grammar as the ‘logic of language’, as representing ‘intelligence’, and as having one correct formulation regardless of context - the dream of a context free language that could be parsed automatically - was the explicit project of computational linguistics up to and until the emergence of machine learning as a viable alternative. Its problems proved fairly intractable, due to what this recent MIT review of automated grammar correction names ‘the inherent difficulties in defining an error according to the perceived communicative intent of the author’. Yes, ChatGPT and its derivatives can perform grammar correction tricks only because they have ingested vast amounts of grammar-teaching and grammar-correcting text, examples of which we can see swilling around half-digested in the Faculty ‘pipeline’. But ‘perceived communicative intent’ is what literacy teachers work with.

ChatGPT can spit out a correct sentence, but it is hopeless at helping students to understand their own inaccuracies and idiosynchracies, or inspiring them to write in more powerful ways. And yet, with AI’s insatiable demand for profitable use cases, this focus on what can be automated in the production of language will no doubt be ramped up. Teachers and children alike will be subjected to regimes of linguistic correction, and made to feel stupid even when the ‘smart’ bot gets it hopelessly wrong. The most privileged schools, of course, are exempt from Gove’s curriculum, as they will be exempt from AI ‘assistance’ to their teaching staff. But 100 years on from the Titanic, ‘the poor’ are still being ‘squashed’ into their linguistic bunks.

It seems the tech news has become the news, and whatever madness that brings into the world in the coming days and weeks, you’ll want to get your sanity check here.

Limited show notes this week, but you might like to check out:

Some recent commentary on the Elon Musk moment (sure to be out of date by now) from the UK Guardian: https://www.theguardian.com/us-news/2025/feb/08/elon-musk-doge-team-staff

And from the Washington Post: https://www.washingtonpost.com/business/2025/02/05/elon-musk-federal-technology-takeover/

Up-to-date takes on tech history-in-the-making are often posted here: https://futurism.com/.

Daniel Greene’s book, mentioned by Audrey: The Promise of Access: Technology, Inequality, and the Political Economy of Hope (MIT Press): https://mitpress.mit.edu/9780262542333/the-promise-of-access/

Feminist critiques of AI from the 1980s and 1990s, mentioned by Helen (most of these require a log-in):

• Alison Adam: https://journals.sagepub.com/doi/10.1177/135050689500200305

• Lynette Hunter: https://www.jstor.org/stable/10.1525/rh.1991.9.4.317

• Donna Haraway: https://www.jstor.org/stable/3178066

• Lucy Suchman (still writing brilliantly on this topic today): https://journals.sagepub.com/doi/full/10.1177/20539517231206794

Audrey’s newsletter, Second Breakfast: https://2ndbreakfast.audreywatters.com/

Audrey’s book Teaching Machines https://mitpress.mit.edu/9780262546065/teaching-machines/

Simone Brown on the origins of surveillance in the management of plantation labour: https://journals.kent.ac.uk/index.php/klr/article/view/1100

Emily Bender and Timnit Gebru (et al’s) famous paper: On the dangers of stochastic parrots: https://dl.acm.org/doi/10.1145/3442188.3445922

Recent critique of this paper from a posthumanist perspective, referenced by Helen: https://posthumanism.co.uk/jp/article/view/3287

Meredith Whittaker on Babbage, computers and plantation labour: https://logicmag.io/supa-dupa-skies/origin-stories-plantations-computers-and-industrial-control/

Reid Hoffman ‘AI will empower humanity’ in the NYT, referenced by Audrey: https://www.nytimes.com/2025/01/25/opinion/ai-chatgpt-empower-bot.html

The article is paywalled but there is an interview with similar takes here: https://techcrunch.com/2025/01/26/why-reid-hoffman-feels-optimistic-about-our-ai-future/

A recent Guardian UK article on the ‘Paypal Mafia’: https://www.theguardian.com/technology/2025/jan/26/elon-musk-peter-thiel-apartheid-south-africa

Peter Thiel’s argument that freedom and democracy are incompatible, referenced by Audrey: https://www.cato-unbound.org/2009/04/13/peter-thiel/education-libertarian/. This is also referenced by Curtis Yarvin and Nick Land in support of their Dark Enlightenment neo-reactionary movement: https://en.wikipedia.org/wiki/Dark_Enlightenment

Links between Palantir (Peter Thiel’s company) and the US military: https://www.palantir.com/offerings/defense/air-space/

Helen’s original substack post on Faculty AI (a new one follows shortly): https://helenbeetham.substack.com/i/139080460/safer-ai-round-two

AI Snake Oil, blog of the book by Arvind Narayanan and Sayash Kapoor, discussed by Audrey and Helen:

AI Snake Oil

Debunking AI hype. The book gives you foundational knowledge and the newsletter covers new developments.

By Sayash Kapoor

Here’s a quick take on why I think the launch of DeepSeek, a Chinese hedge-fund development that has evolved into a model that can out-reason OpenAI’s o1, has so rapidly undermined the stock value of Nvidia, what this reveals about the current AI narrative, and why in the end I think Nvidia (and US AI) will recover. That is, why the build-out of hyperscale data centres, alongside the concentration of all kinds of social and cultural value that can be rendered as data, will continue unchecked.

I’m sorry to say it again, but AI was never going to live up to the hype. This was obvious to anyone with even a passing cynicism about venture capital and the tech hype industry that shares its blood supply. It was widely discussed by tech industry insiders and by experts in (yes) artificial intelligence and cognitive science. It’s almost embarrassing to have spent so much time arguing for ideas that can now be found on fridge-magnets: here’s tech rag The Byte, just days before the current share-dive:

‘This [economic] forecast brings into question the logic underlying the incredible economic, environmental, and human costs of AI development… When, for all their advancements, all these half-baked AI rollouts can offer are "leaked photos of heaven", it needs to be asked: who are these advances for? If AI developers are incapable of producing socially compelling use cases for their models, then burning through trillions of dollars in a never ending AI race isn't anymore practical than a one-way trip to Mars.’

Put it on a t-shirt. The story of hype is not a new one, of course. People who have spent the last two years demanding or proclaiming or actively colluding in the reorganisation of public life around this new technology are now nodding wisely and renewing the Gartner Hype Cycle cartoon on their office door. So that’s OK. We all move on. From the carbon that can’t be unburned, the jobs that can’t be made secure again, the knowledge economies that can’t be detoxed.

But there is something puzzling about this hype cycle. Even Gartner is puzzled:

So while GenerativeAI promised virtual persons, spending on it has been mostly on brute force hardware. On cloud credits by business customers, and on hyperscale data centres by AI providers themselves. And this spend is going up despite the ‘disillusionment’ and the lack of ‘functionality’. How to explain?

In my own feed I’m hearing two takes that make sense. One is Brian Merchant’s. He says that the DeepSeek launch has undermined a story about scale. The story is that:

‘Generative AI is the future, it’s improving rapidly, and thanks to that we’re on the road to Artificial General Intelligence. AGI, in turn, will enable enterprise software to “outperform humans at most economically valuable work”… but the models need to be scaled first. Those models need ever more data, they need more compute power, and they need more energy to run.

DeepSeek undermines this story because it was built with a fraction of the dollars and training compute required for the equivalent OpenAI model (it claims ‘only’ 10,000 Nvidia chips were needed to train R1, though this is disputed). Leaving the geopolitics aside for a moment, Merchant argues that the US innovation sector will be running to miniaturise its own offer, and that it may no longer need all the chips and data centres, the demand for which has hyperscaled Nvidia to the largest and most profitable company in the world.

Yes, and no.

I think the story about scale was already on the way out. Altman and Hasabis have both gone on the record in the past year to say that scale ‘is not enough’ to provide the next breakthrough. (Their hands may have been forced, since nothing in the last 14 months has shown anything like the step-up in capability that GPT4 showed from GPT3, while scale has continued unabated). By November of last year the talk among developers was no longer of improving outputs in any significant way, but of matching existing capabilities with models that are smaller and cheaper. It’s a long way down from ‘liberating humanity’ to ‘making the product cheaper’, but venture capital was always going to demand commercially viable product for its noble investment in ‘humanity’ at some point. In those terms, DeepSeek has jumped ahead and US companies will have to respond, because they can’t command as much for their own product as they could before DeepSeek strolled past ChatGPT on the Apple App store.

But the US companies are so powerful, their reserves of capital so immense, their funding from the US state so secure, they can easily weather what amounts to a slightly longer lead-in before their products become profitable. In Altman’s justificatory statement on DeepSeek, he isn’t stepping down at all from the need for ‘more compute’, even though ‘compute’ is (a significant part of) what makes his product so expensive.

One thing Altman may be relying on here is what is called Jevons’ paradox, where making a resource more efficient only increases demand for it. When he says ‘a LOT of AI’ he is assuming that his company in particular, his narrative and the reorganisation of business platforms around it, has made ‘AI’ such an essential commodity (despite its lack of ‘functionality’) that making it cheaper will only accelerate demand. In terms of sustainability he may well be right: even before the current chat-fuelled boom, emissions from ICT outpaced the general growth in global emissions every year, despite efficiencies in production. People just can’t get enough of those chips.

We should not forget that along with nVidia, stocks in major energy companies have also taken a hit (this headline in UK investing explicitly tells us that ‘efficiencies’ like the ones apparently available with DeepSeek are a source of ‘concern’ to energy investors, because the more carbon gets burned, regardless of whether it needed to be burned or not, the richer they get).

This is where I think we need another story, perhaps a supplement to the first one about the US losing out on miniaturisation (not itself a novel one). What if building out vast amounts of compute and using it to concentrate, organise and govern vast amounts of data, actually is the project?

What if it’s not about AGI at all? Not even about generative AI and whatever commercially-viable use cases it spits out on its way up and down the hype curve? (Obviously the investors and CEOs and business customers will lap up whatever they can get in the way of cognitive automation, and thank you kindly for a lifetime of service down the use-case mine.) What if it is all about data and compute, and who controls them?

Controlling data, controlling the systems and architectures it flows through, digitising the world’s many cultures and cultural products and privatising them, monetising access to them. This is the promise of generative or cultural AI. Owning data about people and their behaviour, predicting and managing ‘deviance’ from the norm, turning what’s left of the social state (in the wealthy world) into private data-driven services, and the rest of the world into a data farm. This is the promise of predictive AI. There may or may not be profit in these today, but they are surely the route to squeezing the last juice out of capitalism in the years ahead.

I have more on the ways that generative and predictive AI are, in fact, convergent in my next post.

So-called ‘AI’ or data intensive military surveillance is also being used to acquire people-as-data and territory-as-data for targeting and rapid decision making at war. Some of the same companies involved in generative AI have been active in this space also, and the alliance between Silicon Valley and the US military state is one we are already seeing strengthened under the current US regime.

From this perspective, so-called ‘AI’ is just the circus, more effective than crypto at inflating speculative bubbles of private capital, more effective at capturing cultural and social data, more effective at mobilising build-outs of data and energy infrastucture - and more effective in allying with the militarised state. (China, of course, has taken a different path to social governance through data, but one that increasingly relies on indistinguishable technologies and privacy violations.) Yes, in the short term it is a blow to US AI capital that genuinely open models can be built for a fraction of the cost and compute. But that is only the short term. In the medium term, it’s all about the concentration of power and data by vast corporations in alliance with the most powerful military states.

That is why AI has ceased to be a story about the commercial viability of some overhyped product, and become a story about geopolitics, and the battle (literally) for alternative futures.

Please like, subscribe and share.

Links

Eamon’s LinkedIn profile: https://www.linkedin.com/in/eamoncostello/

Call for Papers, JIME special issue, Metaphors of AI: https://account.jime.open.ac.uk/index.php/up-j-jime/libraryFiles/downloadPublic/8

Education After the Algorithm: Symposium at City University Dublin: https://www.hackthiscourse.com/symposium/

Eamon’s co-authored article ‘Speculative Practicescapes of Learning Design and Dreaming’ (2024): https://link.springer.com/article/10.1007/s42438-024-00465-5

Eamon’s co-authored article ‘Information and Media Literacy in an age of AI’ (2023): https://www.mdpi.com/2227-7102/13/9/906

Eamon’s ‘AI is destroying education’ post (back in 2023): https://www.linkedin.com/pulse/ai-destroying-education-we-have-one-chance-stop-eamon-costello-hl7xe/

Emily Bender’s talk ‘Synthetic text extruding machines’ (2024) referenced by Eamon: https://linguistics.ucla.edu/event/colloquium-talk-emily-bender/

Helen’s ‘provocation: the unconscious is structured like a language’, referenced by Eamon:

Liz Jackson’s article ‘The Manliness of Artificial Intelligence’ (2024) referenced by Eamon: https://www.tandfonline.com/doi/full/10.1080/00131857.2024.2409739

Althusser’s concept of ‘interpellation’, referenced by Helen: https://en.wikipedia.org/wiki/Interpellation_(philosophy)

Dirty Work (2023) by Eyal Press, book referenced by Eamon: https://www.bloomsbury.com/uk/dirty-work-9781801107235/

Ghost Work (2019) by Mary L. Gray and Siddharth Suri, referenced by Helen: https://ghostwork.info/

Ruha Benhamin’s article on Eugenics, referenced by Eamon: https://lareviewofbooks.org/article/the-new-artificial-intelligentsia/

Links

Catherine’s home page: https://catherinecronin.net/

Laura’s blog: https://czernie.weebly.com/

Higher Education for Good: Teaching and Learning Futures. The book is freely and openly available to read or to download from the publisher, OpenBook: https://www.openbookpublishers.com/books/10.11647/obp.0363

Feminist Special Issue of Learning, Media and Technology journal (2022): the open version from FemEdTech https://femedtech.net/special-issue-of-learning-media-technology-feminist-perspectives-on-learning-media-and-educational-technology/

Helen’s review of HE4G in Postdigital Science and Education (2024): Where is the University worth fighting for? https://link.springer.com/article/10.1007/s42438-024-00511-2

Mark Fisher’s Capitalist Realism (referenced by Helen) https://en.wikipedia.org/wiki/Capitalist_Realism

Walter Ong’s Orality and Literacy: the Technologizing of the Word (referenced by Catherine and Helen) https://monoskop.org/images/d/db/Ong_Walter_J_Orality_and_Literacy_2nd_ed.pdf

Eddie Glaude Begin Again (referenced by Catherine) https://en.wikipedia.org/wiki/Begin_Again_(book)

Links

Dan’s home page: https://www.gold.ac.uk/computing/people/d-mcquillan/

Resisting AI: and Anti-Fascist Approach to Artificial Intelligence from Bristol University Press: https://bristoluniversitypress.co.uk/resisting-ai

Dan’s ‘other’ podcasts on Resisting AI: https://www.transformingsociety.co.uk/2023/07/17/the-extensive-and-unconventional-reach-of-dan-mcquillans-resisting-ai/

On Arendt’s diagnosis of ‘thoughtlessness’ as a feature and an enabler of fascism: https://danmcquillan.org/arendtandalgorithms.html

On AI colonialism and the likely impacts on the Global South: https://foreignpolicy.com/2024/12/17/ai-global-south-inequality/ or https://www.technologyreview.com/supertopic/ai-colonialism-supertopic/

On algorithmic states of exception: https://research.gold.ac.uk/id/eprint/11079/

Wikipedia article on the Situationists: https://en.wikipedia.org/wiki/Situationist_International

And on Guy Debord’s Society of the Spectacle: https://en.wikipedia.org/wiki/The_Society_of_the_Spectacle

“All that was once directly lived has become mere representation”

Welcome to 2025, subscribers old and new. I’m happy to start the year as one of 100 women and non-binary writers on AI who are considered worth a follow on substack. I’m less thrilled to be writing this in a week when Meta has officially sacked the truth and the mainstream media of Europe, or what is left of it, has become an echo chamber for the politics of Elon Musk. But this is only what enlightenment institutions can expect when they cosy up to the counter-enlightenment projects of big tech, and hello higher education, what are your plans for 2025?

I’m sorry you have heard so little from me in the last few months. Blame the conjunction of a new full-time post and a descent into despAIr. MalAIse. MiserAI. At some point in 2024 it all became horribly clear. No amount of skepticism from the business sector is going to stem the supply of AI-ready graduates, whether they are wanted or not. No amount of research evidence or critical analysis of the claims about AI is going to be read in a university sector that is determined to turn research and critique over to privatised data architectures, and reading along with them.

No, it doesn’t matter how destructive generative AI turns out to be for the environment, how damaging to knowledge systems such as search, journalism, publishing, translation, scientific scholarship and information more generally. It doesn’t matter how exploitative AI may be of data workers, or how it may be taken up by other employers to deskill and precaritise their own staff. Despite AI’s known biases and colonial histories, its entirely predictable use to target women and minorities for violence, to erode democratic debate and degrade human rights; and despite the toxic politics of AI’s owners and CEOs, including outright attacks on higher education - still people will walk around the herd of elephants in the room to get to the bright box marked ‘AI’ in the corner. And when I say ‘people’ I mean, all too often, people with ‘AI in education’ in their LinkedIn profiles.

I believe I have been right about the elephants. But I’ve been wrong about the people. Our willingness to ignore harms when they are happening to other people is less surprising, given human history etc, than our capacity to put up with crap. It takes a lot of cope to deal with the banality that is AI in the real world. To find every tool you reach for has already started off on its own chattering journey like a wind-up toy. To feel every online interaction being stripped of tone and style and personal meaning and to tell yourself that’s fine, it’s good not to care too much, it’s probably ‘just an AI’ reading it the other end. To give up on finding reliable information or sharing family news because that environment is now a bin fire, but it’s OK because your grammar is never going to let you down. To forget what was promised last year, or last month, because ‘the AI’ does a new thing now that’s also ‘meh’ but it’s new and it promises to be amazing. To pay the subs, to not worry about the student assignments, to pay the higher subs, to do the academic integrity training, to paper over the cracks, to call papering over the cracks ‘AI literacy’, to realise your career and reputation and working future depend on not finding any of this problematic. All of that. It is exhausting. And human agency is exhaustible.

If I had to diagnose my own malAIse more precisely, I don’t lack energy for the big issues, I just can’t deal with the daily thoughtlessness, the anti-intellect of the AI narrative. And mounting evidence about the impact of using generative AI suggests that thoughtlessness is one thing that will definitely be scaled up.

Luckily, just as I was running out of juice, the wonderful Audrey Watters - original Cassandra of edtech - re-opened her blog for business, and AI has been relentlessly in her sights. You could do worse than start the year with this post from her about reading, give her a follow, and read on.

Meanwhile on imperfect offerings you can expect some short posts about issues that poke me too hard to ignore. I have some longer pieces in development on AI as interface, AI at war, and what universities might be doing better - when the day job gives me time. And starting this week, an imperfect podcast for your listening pleasure. This turns out to be a fabulous way of having interesting people say interesting things about AI and posting them to my own credit and fame. The podcast started out as a syndication of Generative Dialogues, my series with Mark Carrigan, which continues alive and kicking whenever we have something to kick off about.

So, precious readers, if you have managed to read this far without synthetic support, I bring you interviews with Dan MacQuillan, Eamon Costello, Catherine Cronin and Laura Czerniewicz, on topics from the fascist histories of AI to the capture of voice data, neo-Luddism and zines. And there are more in the pipeline. If you’d like to suggest someone I should talk to, please get in touch. Meanwhile as 2025 shambles into focus and the spaces of imperfection become a little less tenable, please do like, subscribe, share, comment, read and now listen to this one.

In this post I recap on what I have written in the past about generative AI and working futures. If you have been following for a while, I have new evidence and some new thoughts. If you’re new, I hope you find what you’re looking for. Follow the subheadings for the tl:dr version, and dive in for more detail, track-backs to previous pieces, and an assessment of how my earlier thoughts are playing out. I conclude that we need AI resilience and suggest what kinds of educational research might help.

In this piece:

1. Generative AI has few really valuable use cases

2. Employers are already falling out of love with generative AI

3. Generative AI might still restructure work - but not in a good way

4. Generative AI is not getting ‘better and better’

5. The carbon, power and water costs are frightening

6. Employers are also concerned about the legal and ethical issues

7. In the end, it’s all about student learning

8. Why AI resilience is so urgent

1. Generative AI has few really valuable use cases

Long ago, in ‘what is it good for?’ I defined generative AI as the automated production of digital content, optimised to a norm. Digital content production was to a large extent pre-automated before the arrival of generative capabilities, thanks to the ubiquitous use of certain productivity software (for example Adobe and MS suites) and standardised workflows around them. The content that was scraped to train large language models (for example from wikipedia, reddit and Common Crawl ) reflected the norms of the particular people and cultures that were over-represented in digitised content, and much of it was optimised for search algorithms, attention rents and click-throughs (that also have cultural biases).

The probabilistic methods that are used in training large language models, and that predict the most likely next element (word, phrase, cluster of pixels) during inference, weight its outputs even further towards cultural norms and biases embedded in the training data.

So pre-automation creates contexts in which large language and media models can seem to produce meaningful outputs. One obvious such use case is the production of search-optimised, clickable content. A second is short-cuts in writing code (highly standardised: already well documented via sharing sites such as HackerNews and StackOverflow). A third is providing natural language front-ends (chatbots) on standard interactions such as customer service. These uses have already been trialled extensively. None has so far produced a revolution in productivity, though staff have certainly been laid off in copy production (or have been transferred from direct production to improving the ‘AI’ output) and some boilerplate code is undoubtedly being outsourced to CoPilot, though with variable results.

In chatbots, the much-touted ‘success’ of OpenAI partner, Klarna, in laying off 700 staff turns out to be a little less than it seems. (Klarna outsourced customer support in 2023, leading to major problems with backlogs and customer satisfaction. The new ‘AI’ chatbot replaces an older, presumably phone-tree style system, but it is not clear whether the improvements claimed are in relation to the original service, or the degraded one, or whether they include the earlier chatbot system in the comparison.)

In none of these cases is the impact on work ‘transformational’ for the workers involved, except in a negative sense. The introduction of generative ‘AI’ is making work that was already repetitive and highly automated even more stressful and precarious. Its use has led to more quantity of copy and code, but as a result the overall value of search results and the quality of online content is degrading; same with the quality of code and its ease of maintenance (something we should all worry about in the immediate aftermath of the CrowdStrike outages).

Unfortunately for higher education there is another use case, and that is the production of student assignments. and similar kinds of writing to a rubric (cover letters, for example). This has many implications for the relationship students have to their learning, and to their teachers and universities and courses of study. It’s a far from simple problem that I and many other educators have commented on, and I will return to at the end of this post. But here, I want to make a distinction between the reasons students use generative AI and the reasons we are told that educators should actively promote its use. The rationale for the second is: ‘they will have to use AI in the workplace, so they (we) had better get used to it’.

This argument is flawed. Universities are not helpless bystanders to the economies of professional work but key stakeholders and policy makers. A university education is supposed to empower students to shape their futures, including how they relate to different techno-social configurations of work. And universities have responsibilities beyond employability - to justice, equity, the disinterested pursuit of knowledge, and having a planet to live on - that demand a more critical assessment of how these technologies might reshape work, and what harms they might inflict in the process.

But there are now more pragmatic objections to this argument. If by ‘AI’ you mean specialist applications of machine learning, these are being developed and adopted gradually, with expert input and as part of existing expert workflows. Students can get on with developing the expertise demanded by their field, confident that will then be able to participate in future ML developments (where they add value) as experts, rather than as data workers. If by ‘AI’ you mean the generic shitshow that is ChatGPT, there is no evidence that it is revolutionising work in general, or even making work significantly more productive. Here’s why…

2. Employers are already falling out of love with generative AI

Congratulations if you are so uninterested in the celebrity romance between big AI and big corporations that you haven’t noticed the cooling off that is in the air. So let me tell you that Goldman Sachs no longer sees AI as a game changer for business productivity. Cognizant, an IT consultancy that has just bet $1bn on generative AI infrastructure so really, really wants it to work, published a survey that found ‘up to 13% of businesses will have adopted the tech in the next three to four years’. Sorry, how many? The Harvard Business Review has also just dropped a study that shows using AI at work makes people unhappy (so less productive).

Companies are beginning to pull out of the enterprise versions of MS CoPilot because its outputs are ‘middle school’ level at best.

Other high profile AI projects that have been quietly dropped include Meta’s clutch of celebrity chat interfaces, an AI chatbot for Los Angeles school students, MacDonald’s AI interface for drive-thru customers, and Elon Musks’ Grok chatbot, shut down for spreading election misinformation. Actually it hasn’t been shut down, as the article makes clear, but it has spread misinformation. The Guardian recently listed other walk-backs, many from the creative industries.

Hedge fund managers are now telling investors privately that AI has been ‘overhyped’, and that its proposed use cases are ‘never going to to be cost efficient, are never going to actually work right, will take up too much energy, or will prove to be untrustworthy’. And the Guardian quotes a senior analyst at Forrester saying that:

a lack of economically beneficial uses for generative AI is hampering the investment case. There is still an issue of translating this technology into real, tangible economic benefit.

Tech stocks are falling, as analysts have realised there is a $600bn hole in revenue forecasts. Some of the biggest companies involved - Meta, Amazon, and Microsoft - now admit it will be ‘years’ before any of their AI products are showing a profit. Heck, Microsoft’s own Chief Financial Officer has just told investors that the company’s $56billion investment in AI data centres is not expected to make a profit from actual, sellable AI applications for more than 15 years.

The pillow promise in this love affair was that AI would make workers more productive, therefore capital more profitable. Back in March, Gary Marcus (one of the most prescient insiders) predicted that any productivity gains would be modest:

The same pre-automation of content production that makes generative AI quite useful, if you don’t care too much about quality, means that the contexts in which AI is quite useful were already quite efficient. Generic generative AI may provide shortcuts to simple tasks (think of MS Copilot as a Clippy update with some enterprise apps) but it makes some complex tasks harder. No doubt the potential will be talked up for a while longer, but no-one serious is talking about a productivity revolution any more.

3. Generative AI might still restructure work - but not in a good way

We should not see this moderation of the hype as a sign that AI will be less bad for workers in creative and ‘knowledge’ sectors. At every point in the hype curve it is possible to worsen working conditions and cheapen the price of labour. Brian Merchant puts it like this:

The design and build of large synthetic media models embodies a particularly toxic approach. As I wrote in ‘Labour in the middle layer’, models are built on the unacknowledged, unpaid work of content creators in the past (training data). But they also depend on the hidden labour of thousands of data workers in the present. First to clean, curate and prepare the training data; then to evaluate, annotate and refine model outputs to produce the desired norms. This process took eight months in the case of GPT4, and although it is a closely guarded secret how many human hours this actually represented, eighty percent of the paid hours on any AI project are estimated to be on data work.

In ‘Luckily we love tedious work’, I argued that graduates won’t necessarily find themselves occupying one of the two ‘expert’ groups in this labour sandwich - producing the original, valued content for models to extract, or using models to support them in their expert roles. In the sandwich economy, productivity and profit depend on paying as few experts as possible, and maxing out their value in the middle layer: this is what produces the model itself as a source of value for its proprietors. Growing at an estimated rate of nearly 30% a year, this work is typically outsourced, precarious and badly paid, but it is far from unskilled. Data outsourcing companies now specialise in sectors or industries; even large general crowdsourcing platforms prefer workers with specific expertise (observations from Muldoon et al’s 2024 Typology of AI Data Work).

Tasks such as these from ScaleAI may relate to generic data, such as images of public streets, or highly specialised data, such as artisitic works, medical or military images, engineering drawings or business spreadsheets. As this layer of labour becomes more specialised and segmented, it has even more use for graduates.

In researching the links between ‘AI’ and the military for another post (coming soon), I discovered that US Project Maven uses Community College students to annotate satellite images for potential military objects ‘for $15 an hour, with course credits thrown in’. These students are definitely ‘AI-ready’, and can move seamlessly from their college degree to the kind of work available through platforms such as ScaleAI. By the way, the UK Government recently celebrated ScaleAI choosing the UK as its centre of European operations, as though it were the next DeepMind instead of a gigwork company highlighted by the Oxford Internet Institute for failing to meet basic standards of fair work.

One way that AI companies are trying to make up for the hallucinations, biases, privacy risks and running costs of their product is by telling businesses to plug in their own data infrastructures. Companies are running MS Semantic Index on the heaps of documents they have accumulated in MS 365, or annotating valuable company IP and documentation to make it ‘AI-ready’. Generic AI models can then be ‘plugged into’ these Retrieval Augmentation Generation (RAG) sources to produce business-specific insights with greater accuracy and fewer privacy risks. It’s a neat way of getting businesses to do the data work that produces most of the actual value, while still extracting rent for the generative front end. But it also means that work is being restructured towards data accumulation and management inside the businesses that are buying the generative solutions.

RAG is basically Knowledge Management for the 2020s. In case you were not there at the time, KM was a business improvement process that was very profitably touted around by management consultants in the 1990s and early 2000s. It urged businesses to capture every thought and gesture of their staff in the form of digital records, supposedly allowing the business to leverage all that knowledge without the expense of employing the knowledgeable staff who produced it (sound familiar?). Mostly it failed, though arguably today’s integrated MS systems are a hangover from the fantasy of total knowledge capture leading to vast productivity gains. Only you can judge how happy, creative and fulfilled the MS panopticon makes you in your job.

In fact, many of the conditions that led to the failure of KM also pertain to Generative AI. A recent study by Upwork (as summarised by Cory Doctorow) found that:

96% of bosses expect that AI will make their workers more productive;

85% of companies are either requiring or strongly encouraging workers to use AI;

49% of workers have no idea how AI is supposed to increase their productivity;

77% of workers say using AI decreases their productivity.

Employers who want to make their workers more productive are almost comically susceptible to hype of this kind. Once they are bought in, the onus is on their staff to make the miracle happen, even though the idea of documenting everything you do and then being expected to do more of it in less time (or gracefully resign) is not an attractive one. But like any ponzi scheme, there is no benefit in telling the person above you that the promise they have just been sold is an empty one. The only option is to keep pushing the promise downhill, until it arrives with those least able to push back.

The irony that attempts to automate can actually hamper productivity is not a new one. It has been researched for many years in diverse industries. In ‘Ironies of Generative AI’, Simkute et al. revisit earlier findings concerning:

a shift in users' roles from production to evaluation, unhelpful restructuring of workflows, interruptions, and a tendency for automation to make easy tasks easier and hard tasks harder…

and find them playing out in full force in the GenAI adoption crisis. But the belief that they can be ‘replaced by AI’ is profoundly depressing and potentially undermining for graduates and professional workers, and only makes it more likely they will accept working conditions that are more precarious, less fulfilling, more isolated, less remunerated.

Educators should already have been asking: ‘is it really our purpose to prepare graduates to be the most productive humans-in-the-loop they can be?’ and even ‘whose productivity, for what greater human good?’ We can now add to that the question: ‘what productivity are you actually talking about?’.

4. Generative AI is not getting ‘better and better’

AI models carry on not getting exponentially better - not even very much better - exactly as I predicted here. There is no fix for hallucinations, as the CEOs of bigAI admit here, here and here. The sheer size and inscrutability of the trained data structure makes it impossible to fix all the connections that might spit out the ‘wrong’ answer, though this does not stop big AI spending $millions on data workers to keep patching up the worst examples. OpenAI has still not delivered GPT5, despite many promises and trailers. The people who are saying that we should be grateful for each iterative improvement are the same people who were saying a year ago that we were ‘only at the start’ of AI’s incredible capabilities, and that they would have improved beyond recognition in a year’s time.

Many cognitive scientists have argued from the beginning that generative AI has computational and theoretical limitations: read, for example: The Cognitive strengths and weaknesses of modern LLMs, Reclaiming AI as a theoretical tool for cognitive science, or Intelligence without reasoning. All of them find it intrinsically unlikely that there will be a major breakthrough in performance. There may of course be iterative improvements due to further scaling up of parameters, or improvements in training and post-training human reinforcement learning. But these are not only costly (and investors are losing faith): they are also pushing against other real limits.

For example, AI is fast running out of quality human-generated data. These projections do not include the impact of synthetic text flooding public sources of information, poisoning the culture of text production and diluting the quality of any human text that still gets out there. (A timely article in Nature also finds that training AI on AI generated text leads straight to model collapse - a shame because that is precisely the solution to a lack of data being touted by people who never thought that human content was up to much anyway).

AI is also running out of compute. There is a world-wide shortage of GPUs (AI chips) and Nvidia has delayed the launch of its next generation Blackwell chip due to production flaws. But it is probably a good thing there are not enough chips to power up the ambitions of big AI, because the world just can’t afford to power them. What Sam Altman calls an ‘energy breakthrough’, required to produce the ‘next generation’ of AI, most analysts are calling an ‘energy crisis’. Because…

5. The power, carbon and water costs are frightening

The full cycle carbon costs of generative AI models are only just starting to be seriously researched. But all the early indications are that they are far, far higher than anyone thought when I first wrote about this issue. There are the enviromental costs of chip fabrication, including the mining of rare metals and the use of largely coal-fired power in Taiwan. These costs are accelerating, as one of Nvidia’s strategies for staying on top of the market is to rapidly obsolesce and replace their highest end GPUs. There is the power required to run the data centres that the models are trained and housed on, recently predicted to rise by 160% before the end of the decade, and there is the fresh water required in all these processes.

Above all there is all the additional compute required by the whole connected world now that power-hungry inferential processes have been embedded into so many basic operations (Luccioni et al explain here why inference is so disastrously expensive). Emissions have soared at Microsoft and at Google, threatening any gains they have made in carbon reduction. Google alone is pouring $billions into new data centres. But most of the additional compute, and therefore carbon costs, is being fired up not by the big AI companies themselves but by the companies buying into their AI solutions:

Since the same companies that sell ‘AI’ also sell IT infrastructure, aka cloud computing and cloud services, and since they have an interest in hiding ‘distributing’ all that additional carbon cost away from their own balance sheet, we will almost certainly never know the true impact. But what is really sick about all this is that finance is now piling into power and utilities stocks. The AI boom they created by piling into AI stocks may not last, but the people convinced by it are still buying chips and cloud credits. So its win/win for finance,so long as the little people just keep believing, and so long as no-one needs a planet to live and breathe on.

I wrote that the greatest climate threat from ‘AI’ was its ability to divert attention, financial power and political will away from from the economic transition we urgently need, and into tech non-solutions like nuclear fusion or some climate engineering miracle designed by a future AI brain. But the capacity of ‘AI’ to delay the real solutions to climate crisis is even more dangerous if in the intervening years - and there’s a phrase to make any climate scientist despair - ‘AI’ is ramping up the carbon output to the max.

6. Employers are also worried about the ethics and legality of generative AI

Employers are concerned about the environmental costs, and also about the toxic biases of generative AI and the negative implications for their DEI agenda. There is plenty of evidence that the use of AI leads to bias in recruitment, for example, and companies are worried about the legal implications. The ‘big four’ accounting firms - that offer some of the top graduate opportunities in the UK - have ‘prohibited’ the use of AI in job applications on equity grounds, and on the grounds that it might select candidates who are disposed to ‘cheating’ - not the best start in an accounting career. Of course ‘humans are fallible’ in recruitment, but few human HR managers have the reach of an AI system, and they can always be held to account. When it comes to gender and race discrimination caused by AI, there is currently no clear legal framework, something that puts people at risk immediately, but also puts companies at risk of future claims.

The issue of potential copyright infringement in the training of generative AI models is still not resolved. Major content owners are rapidly being bought out by AI companies, and copyright law itself is being undermined by campaigns like this one from OpenAI, to the effect that if a government insists on upholding laws that are inconvenient to AI, they won’t get access to the sweetie jar. Still, all this is another source of uncertainty (and therefore legal expense) to any company contemplating whether and how to adopt.

And finally, never mind the factual errors and productivity fails, generative AI is becoming a toxic product. From deepfake image-based abuse to undermining democracy and manipulating elections, from polluting science to ‘cancelling emotions’ (especially useful in those new AI-supported call centre roles), from racially biased image generation to new forms of online addiction, companies are having a hard time persuading their customers that the ‘AI’ in their own products is squeaky clean. It doesn’t help that the biggest players are often the biggest offenders. Nvidia, for example, has just been caught scraping youtube videos for facial data and Meta garnering clicks from AI images of extreme human suffering. But, hey, celebrity chatbots everyone!

7. In the end it’s all about student learning

It is far from clear that employers want or need (generative) AI-ready graduates. All the things that corporations are concerned about right now are things that critics - many based in universities - have been saying from the start. So where are universities in this debate? At least they could be engaging with it as a contested zone that they might still influence for the better, rather than as a predetermined ‘AI future’ they most provide for.

Existing uses of machine learning and generative AI at work show that they can automate the routine parts of tasks. But you can only know how this automation will be useful if you are already an expert in that task. You can only initiate and guide the generative component if you are already an expert in that task. You can only correct for errors and refine the outcomes if you are already an expert in that task. You can only participate in the design and development of new workflows if you are already an expert in that task. This is true whether the task is writing prose or diagnosing cancer. So universities should continue to produce graduates with expertise, confident that they will be able to accommodate any efficiencies that computation may offer down the line. Technologies are designed for ease of use: expertise is hard to acquire.

As I said at the start of all this, the real issue is not what students might be doing with generative AI in some possible future, but what they are doing with generative AI now and how that shapes their individual learning and development. Also what they aren’t doing when they are using generative AI that might be more valuable to their learning (opportunity costs). And finally, what collective cultures and practices are being shaped by their use/non-use of generative AI that students will take forward into work and life.

I am confident there are uses of generative AI by teachers that are helping students to learn. The question is whether they need to be learning in this way. Are they learning in ways that our theories of learning tell us will be beneficial to them in the long term, as workers and as people with an intellectual life and culture beyond work? Are they learning in ways that accord with our values for university learning as a set of cultural practices? I have not seen evidence about this, but then I don’t think many people are asking these questions. What I have seen is a vast amount of work being done by teachers to devise tasks with generative AI, often ingeniously, and always with commitment to student learning. I believe that many of these tasks could be at least equally beneficial to student learning if they did not pass at any point through a synthesis engine.

To give some examples.

1. Prompt engineering, a skill that is already obsolescent. Generative AI is basically evolving into an interface, and interfaces need to be frictionless, so most of the ‘work’ involved in crafting prompts is already being absorbed in further layers of automation (think CoPilot suggestions, and any ‘AI-based’ app that is essentially poking prompts into a foundation model and serving you the results). What will never be obsolete is the skill of asking the right question, or engaging in a dialogue with genuine curiosity. So go ahead, have students devise questions (call them prompts if you like) to clarify their thinking. These can be used in discussion, or to carry out a literature search or some other iterative data query, or to frame the reading of a paper or the watching a video. Want to stage interesting debates and scenarios? Let students devise and enact them. Or have students write their own test and revision materials. Let them write detailed prompts to an imagined language model to ‘test me on this topic’, but then complete the instructions themselves. Because they know what it is like to be a student of this topic (and they have theory of mind), students will do this far better for each other than a chatbot can, but more importantly they will learn from every part of the process.

2. Breaking down writing tasks into component parts. This is always helpful and UK students do not get enough support with it IMO. But the components do not need to be practiced with the aid of an auto-complete word-generator. Ingenious prompts can be prompts for actual writing, not for synthetic production of text. Also, the components still have to be put back together again. At that point students realise that ideation is not separate from gathering evidence, and gathering evidence is not separate from summarising/annotating it, and notes are not separate from ideas and opinions, and ideas and opinions are not separate from the words and images they are expressed with. Writing involves iterating among all of these, and testing the results against facts in the world and with other thinking people. So to invite students to outsource all the parts while they take responsibility for ‘bringing it together’ is to misunderstand as well as to confuse students about the nature of writing. And the nature of thinking. Also, I notice that some guides tell students to outsource the planning and overview but be sure to write in the detail themselves, while others tell students they must own the planning but can use generative AI to help with the detail. Just who is most confused about writing here?

3. ‘Doing research’. As everyone knows, generativeAI is prone to mistakes, hallucinations, non-referencing and false referencing, and bias to the norm. Many students now use ChatGPT/GLM etc for basic research, that is ‘finding stuff out’, and happily most research-based tasks designed by educators are meant to illustrate the problems with doing this. But we are relying too much on students already thinking like experts in their field? If students can identify a problem, what resources are they using to do this, and how are they judging the reliability of those sources? If students can provide a better answer, what knowledge and expertise are they drawing on? And if what they really need is to develop those alternative resources of knowledge and expertise, any search or ‘research’ they do with ChatGPT must have an opportunity cost. So ‘spot the AI mistakes’ might be a cautionary exercise, or a one-off revision test, but I don’t think it provides the motivation or structure students need to develop information skills or disciplinary practices or foundational concepts or epistemic judgement for themselves. The experience of many students in this situation, that the AI is ‘often right but sometimes wrong but you can’t tell when’, may lead to the worst possible combination of dependency and anxiety. Instead of focusing on specific errors, such exercises are surely better directed to revealing the generic mistake that is relying on an autocomplete engine at all, and explaining the biases, obscurities and injustices that make them so unreliable. And that leads naturally into a discussion of research strategies and methods that are trusted in the subject discipline, and why they are trusted (and so into epistemology, even if that word is never used).

If universities believe in the value of learning at university, they will offer spaces for collaboration and knowledge-building. They will provide students with models of self-development, and a culture of respect for the finite planet, and the different epistemic cultures and traditions that enrich it. They will be places where public knowledge and specialist expertise are actively being produced, not passively accumulated. Then they will develop not only ‘good workers’ but people who can build good workplaces.

8. Why AI resilience is so urgent

As I argued in ‘writing as passing’, student work is almost by definition the production of content, optimised to a norm. This makes the effects of generative AI on learning and assessment profoundly disruptive. Assessment is a performance, but we make a contract with students that by jumping through the hoops (usually content production of some kind) they will in the process be developing some practice or understanding useful to them beyond the scope of that performance. In other words, they will be learning something of value.

If a chatbot can jump through the same hoops, to something like the same normative standards, and if we can’t reliably detect the difference, every part of that contract breaks down. Students don’t have to go through the process we designed for them. Worse, we can’t follow what their process is. The models are black boxes that refuse to give up their secrets. How (for example) do the patterns they encode correspond (or not) to the schemas that experts use to organise their thinking? (Remember constructivism? This first concept of teaching 101 is that learning involves actively building conceptual schema, and patterns of meaningful activity. Learners have to do this for themselves, on their own terms, based on their own prior experiences. The schema and practices of experts can be valuable as models. The parametric structures and hyperparametric weights of a transformer model? Not so much.)

Because of the secrecy engendered by ‘academic integrity’ scares, students’ process with these models is a second black box on top of the first. We see what comes out, but we have very little idea (unless I am missing some new, more subtle research) what students are putting in. Worse still, students can no longer be sure that whatever process they are learning has value, if a computational process can (apparently) do the same. If they have followed guidance and only used generative AI for some parts of the process, what grade might they have got if they had used it for the whole process? What does that mean for the value of the part they did for themselves? What does it mean for the value of the part they didn’t do for themselves?

In February, a survey of nearly 500 students found:

preliminary evidence that extensive use of ChatGPT has a negative effect on a students’ academic performance and memory; [therefore] educators should encourage students to actively engage in critical thinking and problem-solving by assigning activities, assignments, or projects that cannot be completed by ChatGPT,

Tempting though it is to wave at these results and retire, I am not convinced they show ChatGPT is a cause of poor performance. But they do identify - at least for this sample - the kind of students, and the kind of learning situation, for which the use of ChatGPT becomes compelling, and they show that for these students the academic outcomes are poor. Another recent study on student perspectives found that the more confident students were in their own writing, the less they approved of the use of ChatGPT for writing. These studies are useful, but they raise more questions than they answer. We need more studies of this and more qualitative kinds; studies of different uses, different students, and different settings; we need far more longitudinal studies; we need more critical studies that look beyond immediate contexts of student use. But mostly we need collaborative work with students that go beyond surveys, inviting them to explore openly their processes of production, and their experiences, both personal and socio-cultural. (What are the pressures on students to which ChatGPT/ChatGLM (etc) is the solution? How do they value their own processes of reading, writing, note-making and knowledge construction? How does the belief that ‘everyone is using it’ change their perceptions of academic work? Such experiences will be complex and differentiated.)

Instead of the AI-ready graduate, this might give us a sense of how the ‘AI resilient’ graduate comes into being. How do they develop the expertise to critique and mediate the generative output, not just in class-based activities but in their own study and intellectual life? What skills and practices of production might help them to withstand future cycles of planned deskilling and automation? How can the promise of instant productivity and performance be mediated by other values and hopes, for their learning and for their working futures?

In the short space before we learn what that strategy priority will look like, I am taking my soundings from two main sources: Tony Blair’s piece in The Times on Saturday, much of it reiterated in his speech to the Future of Britain conference, and Peter Kyle’s visit to Silicon Valley in February of this year. There are other influences on the Starmer government, such as the more measured approach of Labour Together, but these two seem most likely to set the tone on technology policy.

It’s an open secret that the Tony Blair Institute for Global Governance (TBI) has been advising Starmer for years, and its 1000-odd staff will be filling key advisory roles in his government. TBI’s relentlessly millennial vibe is technology on drum and privatisation on bass, with backing vocals from a band of states with dodgy human rights records, and Larry Ellison (tech billionaire) on the decks. So it’s no surprise to hear Blair crooning for Starmer to feel ‘the full embrace of the potential of technology’. AI is ‘the only game changer’ that will ‘turbo charge growth’ and ‘save the government tens of billions of pounds’. That’s a bold prediction when there is growing scepticism from investors and push-back from almost every commercial sector against the hype around AI productivity. But the public sector has always been a place for big tech to find acceptance when the commercial world turns harsh and judgy.

Specifically, Blair proposes to use AI to ‘cut workforce time’ by 20% across the board, amounting to around 1.15 million public sector jobs by the end of the next parliament. Expected redundancy payments of £24billion will be offset by an AI productivity bonanza, and if trade unions are pushing back against these proposals, it is surely a sign that they must be on the right track. Working with Faculty AI (spoiler alert - Faculty will appear more than once in this piece), TBI analysed roles in the Department for Work and Pensions and estimated that over 40% of tasks here could be automated. Ironically or not, ChatGPT was used in the task analysis. DWP is twice as nice for AI, it seems, because so much of its work is ‘citizen-facing interactions’, and because pensioners, people with disabilities, and people in situations of desperation and poverty are especially keen to talk with chatbots about their circumstances. (TBI/Faculty admit that the study was theoretical and did not actually interface with any public servants or service users).

I found most of the TBI/Faculty report difficult to follow, repetitive and badly referenced. Perhaps ChatGP was let loose on more than just the task analysis. In one of its more lucid suggestions, AI was:

analysing streams of real-time data to produce actionable insights at an otherwise impossible pace and scale: for example, identifying fraudulent or erroneous claims in progress and nudging citizens away from proceeding with these claims.

If only this gripping scene had not been cut from Minority Report.

TBI’s proposals for AI are essentially the same as its proposals for public service reform (and at this point you might want to refer back to a piece I wrote last year: AI and the privatisation of everything). For every problem of infrastructure the answer is digital infrastructure, and the shibboleth for digital infrastructure is currently ‘AI’. ‘Almost everywhere, AI can help us reimagine the state’. Public transport? ’Mobility as a service’ ‘ (‘combining data and ticketing’). Unemployment, or under-employment? ‘A digital employment assistant for every claimant’. Crumbling schools? ‘A secure, functional and interoperable digital learner-ID system’. People dying on waiting lists and hospital trolleys? You’ve guessed it: ‘personal health accounts’. And most of this infrastructure will naturally be based in California, even if the people whose jobs it displaces and whose data it aggregates are living in the UK.

Let’s turn now to Peter Kyle, the new SoS for Science Innovation and Technology, and Matt Rodda’s boss. He has shown an interest in digital inclusion and promises more government support for digital skills. If that is channelled through organisations like the Good Things Foundation with its local hubs and public sector ethos, and if it means lifelong capabilities rather whatever tasks mTurk is assigning this week, that will indeed be a Good Thing. Kyle also promises a longer term approach to R&D funding, so there is a chance to build national capacity in relation to the challenges and opportunities of AI: research labs, models, development tools and data sets, and regulatory initiatives to meet the particular needs of the UK (ideally open and publicly owned).

But there are reasons to think that building these public capacities may not be Kyle’s priority, and to understand that we need to look at his relationship with Silicon Valley. After a think-tank-funded visit in February with MS, Amazon, Meta, Apple, OpenAI and Anthropic, Kyle seems to have been given a free go on the Tony Blair fantasy calculator (the one that works in denominations of ten billion), declaring that: ‘The growth opportunity from AI [ ] could see the exchequer receive £60 billion more in revenues’. On his return to the UK, he lost no time reassuring big tech that a Labour government would make few new regulatory demands: a requirement for ‘frontier labs to release their safety data’ still sounds like marking their own homework. Instead, a Labour government would ‘unblock the tech barriers’, ‘stepping out of the way to allow large technology companies to build critical infrastructure like data centres in the UK’.

How will it serve the UK economy to become a lorry park for big tech’s data juggernaut? Kyle does not address this question. Instead, he picks out two examples of AI from which he extrapolates many opportunities for the public sector: ‘faster cancer scans’ and ‘personalised lesson plans for children’. The promise of faster cancer scans also comes with a personal story that makes it very hard to ask more probing questions. Kyle tells us:

‘I have seen AI tools which I believe would have caught my mum’s cancer earlier. It is personal for me to get this technology used in a way which keeps families together for longer.’

Cancer scans and lesson plans

Cancer scans and lesson plans are recurring motifs in the TBI reports as well. So what do these two applications of AI have in common? Regular readers may recall my deep dive into the claims of machine learning models to be revolutionising certain fields of science. I think the claims of AI to be revolutionising cancer diagnosis and lesson planning are rather similar. You could almost call it a playbook.

In the case of cancer scans, it was 2016 when ‘godfather of AI’ Geoffrey Hinton demanded that the world should stop training radiologists because AI was making them redundant. Thankfully, the world did not listen. In fact the NHS is experiencing its worst ever shortage of radiologists right now, as training fails to keep pace with the demand for skilled diagnosticians. Hinton was right that radiological imaging would account for the majority of ‘AI’ applications in healthcare, since diagnosis and prediction are what machine learning is (sometimes) good for. But he was entirely wrong to think that radiology can be reduced to those aspects of the imaging-to-diagnosis workflow that machine learning (ML) can support.

ML systems are specialised to certain kinds of image, and sometimes even to specific imaging equipment, and even these highly specialised systems lose efficiency when exposed to real world data. So there is no generic ‘AI radiologist’ that can detect cancerous signs from a multitude of image types. Expertise is needed to interpret even algorithmically-enhanced images, and to check and sign off on any diagnosis that may be suggested. In 2021, a review of ‘AI for clinical ontology’ found that:

While there have been thousands of published studies of deep-learning algorithm performance (Kann et al., 2019), a recent systematic review found only nine prospective trials and two published randomized clinical trials of deep learning in medical imaging (Nagendran et al., 2020).

And three years further on, a major synthesis review still found that:

even though the potential of AI software to impact radiology is large, little is known about how it is changing the quality, efficiency and costs of health care…
it has been proposed by multiple parties, including the FDA [US Federal Drugs Agency], to change the way AI software is regulated from a standalone evaluation to a more systemic approach where the context of the clinical and human interaction is taken into account. This would require more real-world monitoring contributing to the available evidence on the actual clinical impact of AI on health care.

At the sharp end of health care, I was recently put on a two-week cancer pathway and waited eight weeks for surgery. I’m fine, and my care was good when I got it, but the strains on NHS resources were obvious. Most people in the UK can tell a similar story about themselves or a loved one - we all have the equivalent of Peter Kyle’s mother we’d like to be treated faster. But even if diagnosis is improved in some measure by new imaging tools, making a difference to a cancer patient’s outcomes still requires investment in oncologists, specialist nurses, lab technicians and surgeons, in operating theatres, hospital beds and therapies.

Just this week, a group of experts writing in the Lancet have begged the incoming government to stop seeing AI as a ‘magic bullet’ for cancer and invest in some of the critical infrastructure that has been so neglected.  The UK is currently one of the worst of the developed nations for cancer survival rates, and unnecessary cancer deaths align closely with indicators of deprivation. But these are much bigger problems than can be dealt with by machine learning in some diagnostic workflows.

The lack of rigorous evidence for ‘AI’ in real-world cancer outcomes will remind regular readers of the lack of rigorous evidence for ‘AI in education’ that I covered in a previous post. Melissa Bond and her team concluded a meta-synthesis review in January 2024 with ‘a compelling call for enhanced rigour’. Even hotter off the press, a systematic review of the literature of ‘trustworthy and ethical AI in education’ in July found that:

‘the complete absence of pedagogical, design-based as well as empirical, evidence-based studies on experiments from practice is notable, and underlines the starting point of a scientific debate that is still lacking practical developments and implementations’.

But TBI’s Future of Learning report refuses any doubts about practical implementation. AI is set to raise national attainment and GDP by 6% ‘at the most conservative estimate’. This figure seems to have been arrived at by taking the mean average benefits reported in 14 meta studies of ‘pre-AI-era ed tech’ and multiplying by the effects on lifetime earnings found in a single study of pupils completing their GCSEs in 2001. Readers might want to check out the first few ‘references’ in this report to get a sense of the academic rigour involved. (I hope Larry Ellison thinks his funding is being well used here, though he may be distracted from considerations of learning gain by the recent collapse of his AI-for-cancer-detection start-up.)

The roadmap to achieving these incredible advantages with AI is ‘a single digital ID for every learner’, collating every source of data about their progress and achievements. Labour may have ruled out Tony’s plan for digital ID cards to manage migration, but none of the uses they have for ‘AI’ in public service is possible without data collection and integration on a national scale. In education, TBI claims that this will allow the classroom to become a ‘real-time data environment [that] would mean AI could mark a class’s work in less than a second and provide personalised feedback’.

In a familiar use of what I like to call the ‘extended conditional’ tense, the report goes on to anticipate that:

adaptive learning would become the norm in every subject, using data collated from different apps to create assignments that challenge, stimulate and engage the learner without leaving them behind. Each learner would have access to a personal learning map, with AI-supported tuition, and could revisit any content when needed’.

In Teaching Machines: the history of personalised learning Audrey Watters records the promise of individualised learning over more than a century, showing how it has often accompanied the introduction of new technologies. In this real history – the ‘extended imperfect’ if you like – educational benefits have proved elusive, inequalities have often been exacerbated, and a great deal of money has been spent. ‘Personalised learning’ almost always turns out to mean creating efficiencies in admin systems by collecting a lot of personal data, efficiencies that could be translated into benefits for learners if the savings were all invested back into teachers and learning support. But (and this may come as shock) organisations tend to have other uses for efficiency savings, and tech companies always have other uses for the data.

Still the promise refuses to die. Just last year, FacultyAI’s ‘hackathon’ for education came up with the question:

What if teachers could use GenAI to create lesson plans, homework exercises, personalised worksheets and assignments for students based on the curriculum and to support content covered in class?

‘What if?' indeed. The Oak National Academy had in fact just received several £million to develop ‘AI lesson planning’ capabilities for UK schools (nine months on, the web site shows that this is ‘coming soon’).

Data subjects and risky futures

In the use of cancer scans and personalised lesson plans to advocate for a general ‘AI revolution’ in public services, there is more going on than an ignorance of contextualised research. There is an ethos of diagnosing needs and risks instead of addressing people’s conditions of life. TBI envisages‘citizen-centric digital identity’ systems interfacing between people and services at every touch point: managing migration, policing protests, tracking qualifications and skills, monitoring healthy activities, or assessing claims for benefits. ‘Centric’ suggests empowerment, but it’s clear that the person in the middle of these converging data systems can’t be recognised as a citizen or a service user unless they submit to this systematic surveillance.

And what is government going to do with all this data? Provide more and better services? Target services differently? Collectivise the identified risks? It isn’t clear. Nor is it clear who will own, manage and enable access to citizen-centric data. Part of Blair’s thesis is to treat all data as a marketable asset. In this interview, he cites ‘bioscience and AI’ as the key drivers of UK growth and argues that the country’s greatest asset lies ‘with the NHS, because it's a single payer system, and you have got data that is of immense value to creating your bioscience industry and to letting it flourish’. So the best way for the UK to participate in the next technological revolution is to turn over its NHS and Biobank records.

The second part of Blair’s plan for data is to use it – via personalised technology – to give individual service users more ‘informed choice’. Personalised Health Accounts and genomic sequencing are classic TBI projects, as well as personalised learning accounts. They provide major new market niches and procurement opportunities for private tech, while contributing in a deeper sense to privatisation by making health or learning a matter of personal self-knowledge and self-investment. Something to be read off from expensive wristwear rather than guaranteed by a democratic state.

The subject/citizen is now a body of data, invited to know themselves as a unique bundle of desires and needs (and genetic codes), but known by the state and its corporate partners in terms of quantified risks. Whatever can be ‘personalised’ in a public service is almost by definition non-essential. So individual users are always right about what they want, but the needs people have in common can be refused any reality. Tech capital in particular has no capacity to build the foundational services people need, but can help to manage those needs, providing the interface between citizens and what remains of the common good in the form of ‘choices’, ‘customer services’, ‘personalised plans’ and ‘diagnoses’, chatbots and AI-based apps. Meanwhile the state can use all that data to provide a kind of risk management service or insurance back-stop to private capital as it moves into the public sphere. Calculating risk is exactly what deep learning is good at.

Regulatory capture

When it comes to AI, there is also the question of its own risks and how a Labour government might seek to manage them. The manifesto promised to criminalise the production of deepfake images, as the last government did. End Violence Against Women has argued that this would require thousands of individual prosecutions to achieve any deterrent effect. There is no suggestion that the tech companies supplying the means to create and spread harmful images should be subject to any legal sanctions. But ‘outlawing nudification’ is at least a statement of intent.

Beyond deepfakes, Peter Kyle has criticised the last government for being ‘too slow’ on AI regulation overall. And it’s true that Sunak was reluctant to place any constraints on the sector that he rather obviously hoped would employ him in the very near future. His Frontier AI Taskforce was so stuffed with SV insiders that a 2023 House of Lords Inquiry into Generative AI went so far as to express concern about regulatory capture.

(What is regulatory capture? A recent article in Policy and Society traces the deep influence of big tech/big AI across multiple domains of public policy, describing the biggest corporations as ‘super policy entrepreneurs’ :

semi-autonomous and semi-sovereign entities enjoying considerable global authority. Sovereign states are also beginning to treat Big Tech like sovereign actors. Governments worldwide have started assigning diplomats to work exclusively with Silicon Valley. 

Silicon Valley, of course, has long been assigning its own diplomats (lobbyists) to ‘work with’ state legislators. A new report from CommonWealth identifies other mechanisms of influence: using venture capital to monopolise R&D; funding thinktanks; capturing AI talent; and influencing (keynoting, hosting, sponsoring) major conferences and policy forums.)

Surely these efforts will continue under Labour. But will the outcomes be any different? Chris Bryant, who now has responsibilities for cyber and digital, said in opposition that: ‘

A Labour Government would introduce binding regulation of the most powerful frontier AI companies, requiring them to report before they train models over a capability threshold, to conduct safety testing and evaluation and to maintain strong information security protections. 

‘Binding’ sounds tougher than the laissez-faire approach of the last government. But it still asks AI companies to self-report their approaches to training and to safety. It still limits regulation to ‘the most powerful frontier AI’. ‘Frontier AI’ is a term invented by leading AI companies and defined as ‘highly capable foundation models that could possess dangerous capabilities’: both models and dangers are therefore still in the future. The ‘frontier’ companies have lobbied hard for self-regulation with some kind of government oversight, creating strong partnerships between the current biggest players and any future regulatory bodies, and making it much harder for models to legally be developed by anyone else.

For these companies, the Labour manifesto could not be more reassuring:

We don’t seek to disrupt the voluntary code, but we will certainly make sure [the standards] are maintained and that any new entrants into the market will know that there’s a legislative foundation that must be adhered to.

Labour also seems happy not to disrupt the AI Safety Institute, that advocates an extremely narrow and technical approach to safety, ignoring risks to the environment, knowledge economies and human rights. The Institute’s safety regime looks exactly like the kind of red teaming that already goes on in Silicon Valley, and Matt Davis of AI Now argues that this risks the Institute ‘essentially becoming the provider of voluntary services to large incumbent companies’. So it may be significant that while the Institute is still ‘acquiring the necessary technical expertise’ in the UK, it has already opened an office in San Francisco for ‘conducting joint evaluations of AI models’ with key players over there.

This surely is what CommonWealth meant by the capture of expertise.

Meanwhile Peter Kyle has given numerous speeches to business leaders promising to ‘cut red tape’ on AI start-ups and innovators. The ‘Regulatory Innovation Office’ that he proposes to ‘speed up’ the Government’s response to AI developments is meant to accelerate innovation, not regulation. Nor is Labour immune from more direct forms of capture. Faculty AI have donated £36k this year to ‘work on AI policy research’ in Kyle’s personal office. Yes, it’s Faculty again! Developed for their core business in defence, border control and cybersurveillance, their Frontier AI platform (nice name) has been adapted to support operational decision making in the NHS.

Faculty received £350k to consult on AI use cases in education, as previously covered here, and the company now provides safety testing services to the AI Safety Institute, since its role as data partner to the Vote Leave campaign gave it unique insights into the interface between politics, privacy, and data safety.

So, far from challenging the previous government’s trajectory, Labour seems even more keen to open up UK public sector to AI based services. People who have objected to AI spy firm Palantir running the NHS data service are chided by Labour’s new Health Secretary, Wes Streeting, as ‘the tinfoil hat brigade’, though they include health unions, patient groups, privacy campaigners and well-known tin-hatters the British Medical Association. Streeting promises to ‘fight’ such nonsense. Better use of data will be ‘equivalent to hiring thousands of new doctors’, he claims. (Readers, I checked the 2022 report he referred to, and the main takeaway is that doctors are fed up with NHS IT). In the same speech he chides patients who ‘still wait on the phone at 8 am, or even queue up in person in the cold on a frosty morning just to see a doctor’ when they could be using the NHS App. I use the NHS App myself, and I can’t help feeling that the problem is a lack of doctors at the other end of whatever tech you are using, and not a lack of tech. (Can I resist mentioning here that Google Deepmind tried and failed to build an AI-powered NHS app some years ago? No, I don’t think I can.)

It’s true that public procurement of corporate IT systems has not been an unqualified success in the UK, perhaps most spectacularly in health (oops, that was one of Blair’s). And it’s true that data systems are not at all well integrated, one consequence of splitting up what was once a national health service into competing entities, much as local authority managed schools have been split up into competing academy chains, and public providers into private water, power and rail companies. But is the solution bigger data management contracts? Is the solution ‘AI’?

What to build?

Perhaps it’s time to admit that the only thing keeping these disparate state services together are streams of data. Not a universal ethos of state care ‘from cradle to grave’, and certainly not a commitment to accountability, with representative bodies deciding how services can better meet the priorities of today and of the future. You could point instead to the plethora of regulatory bodies that have been set up in the wake of privatisation, bodies that have the tricky job of promoting public interest and safety with companies whose first statutory responsibility is to their shareholders. But they too rely on data to monitor what is decreed to be the public interest. They do not require an actual public with an actual voice.

So Labour is promising a new National Data Library, bringing together government, industry and university data sets, and using them to deliver its vision for ‘data-driven public services’. Whether this can work in the public interest depends very much on who gets to build, own and manage the magic mountain. If Kyle wants to ‘reduce red tape on AI innovation’ to ‘support the next 10 DeepMinds to start up and scale up here within the UK’, he might want to start by asking why DeepMind has been owned by Google for the past decade. Or why MicroSoft has just hoovered up InflectionAI, or why the UK’s most innovative chip maker, Arm, is now owned by SoftBank and traded on the NASDAQ. One reason the EU is so keen to regulate hard on AI is to ensure home-grown businesses can compete with the US giants, and to keep on shore some of the employment opportunities, intellectual properties and tax take. This is only possible with the kind of public investment that Labour seems to have ruled out, relying instead on deregulating in ways that incentivise the private sector.

And so we return to the one concrete promise in the manifesto, to ease planning restrictions on big data centres so that Californian tech companies can capitalise on the relatively cool, well-watered environment of the UK. Will these data centres be part of a sustainable energy transition? Will they power up UK businesses and public services? Or will they, like Amazon warehouses, benefit from low regulation and low-paid workers while taking the power, profits (and tax liabilities) back home? It’s worth noticing that the decision by ScaleAI to site its European headquarters in London has been trumpeted by both incoming and outgoing governments as a sign that the UK is an AI world leader. But ScaleAI is not another DeepMind, pushing the boundaries of research and development. It is a data services company, supplying cheap labour from India, the Philipines and other parts of the Global South to annotate training data. It was named by the Oxford Internet Institute in a study into poor labour practices. The new government will quickly have to decide whether it is an asset or a liability that the UK is at time of writing the least regulated nation in Europe for labour rights and AI safety.

If it is time to build, who is building, and how does it benefit workers and citizens? As Matt Davis argues later in his excellent article for AI Now:

Instead of assuming that any and all types of AI will produce economic growth and societal surplus with minimal state intervention, government needs to develop a clear articulation of what “public benefit” looks like in the context of AI and what sort of AI sector will deliver it. It also needs to understand how AI… impinges on other long-term priorities such as environmental obligations and the concentration of power in the digital economy.

Only governments can do this. But if governments are captured by big AI, they are less likely to take action on risks such as environmental degredation, the concentration of power, or labour rights. And regulatory capture provides a different lens through which to view Kyle’s visit to Silicon Valley in February, a visit funded, hosted and largely scripted by big tech. We should perhaps stop asking what policies the Labour government has in mind for AI, and ask more pertinently: what policies does AI have in mind for the Labour Government?

Last month I gave a seminar as part of the IoE Academic Writing series called ‘Student writing as “passing” and the role of generative AI’. You can find the recording here if you prefer to read with your ears. Thanks to the wonderful Ayanna for organising this event and for her ongoing investigations into student writing alongside synthetic text. Thanks also to the participants whose insightful questions and contributions helped to shape this post, and to Ayanna and John Hisldon for generous comments on it.

In this post I cover:

• The Turing Test, and what it has in common with student writing, at least as writing is often experienced by students

• Student writing as ‘passing’ in a world of deepfakes and generative possibilities

• Beyond passing 1: student writing as identity work

• Beyond passing 2: student writing as social activity

• Beyond passing 3: student writing as expression and dialogue

• Some practical ideas for writing beyond passing, many drawn from the seminar participants

In preparation for the webinar, I also wrote a post all about the Turing test, its implications for contemporary ‘AI’, and how it relates to theories of learning. That might be a good place to start if you want a deeper dive into those issues. Otherwise, here’s a brief summary on the Turing test before getting down to thoughts about student writing.

The Turing test

The Turing test is a thought experiment that remains at the heart of the ‘artificial intelligence’ project.

The classic version can be represented as a system of three players. The role of Player C – the ‘judge’ or ‘interrogator’ - is to question the other two players, who are a computer (Player A) and a human responder (Player B), and to decide which is which. Turing’s prediction was that by the end of the twentieth century:

‘an average interrogator will not have more than 70% chance of making the right identification after five minutes of questioning’

It is this capacity to fool people, or at least some of the people, some of the time, that is the goal and standard for artificial intelligence. It is a standard that is regularly claimed to have been ‘passed’ or ‘broken’, and since the first public generative transformer models were unveiled in 2022, those claims have come and gone as regularly as OpenAI has changed its ethics team.

The equivalent positions of human and computer, the insistence on disembodied ‘outputs’ and the comparative nature of the judgement are all used as signs of the test’s objectivity. The players are behind some kind of screen. Their bodies and voices are hidden, their responses limited to the passing of text messages. The human beings, then, are made to seem as much like automata as possible. Yet, as I show in my parallel post, the test is a maelstrom of misdirection, identity politics and hidden desires.

Student writing as ‘passing’

In my talk I outlined some similarities between the Turing test and student writing for assessment, at least as it is typically understood by students themselves. For example, the dynamics of the situation place students on the other ‘side’ to their academic assessor, visible only through a screen. Text is the preferred medium, reducing opportunities for interactions of care and concern that may previously have existed among the players. The judge can ask anything they like, and their decision is highly consequential. Ultimately, ‘passing’ as a graduate really does confer a new identity. But meanwhile, every time a student offers up work for judgement, they are being compared with other students who may appear to be more credible, more ‘passable’, less of a fake.

There are many reasons why students may feel that they are not credible as ‘real’ students, or ‘real’ producers of text. Synthetic text promises, on the face of it, to allay these fears.

Yet while it offers to help students ‘pass’ each assignment, overall synthetic text only adds to the anxiety about passing. The norm for comparison – in students’ minds at least - is now an AI-augmented student, perhaps one with access to better models or AI writing services, or with more confidence to use AI and not get caught. Students may feel they are contending directly with text automata, their faultless grammar and authoritative tone becoming the new norm against which ‘real’ student work can only look inadequate.

And students may justifiably feel concern about failing the Turing-equivalent test of ‘AI detection’. Having your text identified, rightly or wrongly, as emanating from a bot may mean becoming a non-student, or at least being subjected to investigations that further undermine your sense of validity. In the absence of any certainty, on either side, about how ‘AI text’ can be detected, the best strategy for passing may not be the straightforward one – to write unmediated – but rather to second-guess what ‘unmediated’ looks like from the other side of the screen.

The dilemma of writing authentically in a world of synthetic text is the same dilemma facing every content producer in a world of deep fakes. Here’s Rob Horning in a brilliant recent substack post on the tension between realism and ‘the real’:

Horning goes on to define the ‘deep real’ as a representation that aims ‘to subordinate events to their potential status as content’. The real (event) gains currency only when it enters circulation, and it is the content economy that defines its realism. If we think about student writing through this frame, all writing as process (event) is sooner or later subordinated to its status as an assignment (content), offered into the system for submission, grading, and validation. The status of writing as ‘authentic’ is not given until it has ‘passed’ these hurdles, which like realism also have their modes and fashions.

None of this invites deep attention to the process or event of writing. Indeed, unlike in the US, UK students receive little or no attention to writing as a process. They may well have come through a school system that promotes the most instrumental kinds of text production, expecting a mark for every correct fact, and throwing in as many key words as possible. School students in the UK are trained to write for a highly automated marking system, even if teacher labour has not been fully automated. And this is true of education systems around the English speaking world. As a result, large language models are trained disproportionately on text that has already been optimised for such systems.

For students, university writing can seem an unassailable mystery even before the mysteries of assessment. It is a black box all of its own. You learn stuff. You write stuff. Who knows what happens in between? It must not be cut and paste. It should not be auto-summarise, or auto-generate. And yet these processes work, in a way that is not at all guaranteed by the recommended methods of note-making, observation, reflection, discussion, planning, analysis, ideation, argumentation, position-taking, translation, disputation, editing and re-editing, and all the iterative adjustments between the whole and the parts that constitute thinking in text. However, we can try to open up the black box of writing, and later sections explore how some teachers of writing do this.

Unlike Turing, academics also try to open up the black box of judgement and explain to students the criteria by which their writing will be assessed. But in being made explicit, qualities of writing can be further instrumentalised, and can even become data models of the kind used in automatic essay scoring. (It seems worth noting here that, after decades of investment, AES systems remain barely useable, unless their purpose is to skew student writing permanently towards those surface features that automation can measure, and away from all other considerations of value. One good thing to be said for AES, however, is that ChatGPT is worse.)

Today’s large language models are evaluated against qualities that sound a lot like marking rubrics. ‘Readability’, ‘insightfulness’, ‘completeness’, even ‘metacognition’. Every measure in the FLASK evaluation set (shown above) is a statistical model of a quality that was originally judged by human assessors. As you can imagine, among model developers there is a strong incentive to integrate these benchmarks into the model training process, ensuring that outputs are already aligned and the model is likely to perform well on benchmark tests. Just like marking rubrics, benchmarks skew performance towards values that can be measured, and towards norms that have already been standardised.

Retraining is a long and expensive process. As a short cut, model engineers also use system prompts to adjust performance. System prompts are natural language instructions that are typically added to the user prompt to direct outputs. Versions of ChatGPT’s system prompt have been widely circulated, and they read in parts like the kind of general guidance that might be given to a student writer, perhaps in a slightly more instrumental tone:

• Organise responses to flow well, not by source or by citation

• Do not regurgitate content.

• Make choices that may be insightful or unique sometimes.

• Never write a summary with more than 80 words.

• Integrate user suggestions.

So synthetic text arrives into a regime of writing and assessment that is already instrumental and transactional, tuned to measuring what can be measured rather than supporting personal development or diverse voices. Without being fully synthetic, student writing has already been pre-automated in order to pass through university platforms: submission systems, plagiarism detection, automated marking and feedback, progress/grade dashboards. Student writing might already feel a lot like passing textual tokens through a screen. Particularly as rising teacher/student ratios and reduced teaching time mean that the different players, just like Turing’s, may hardly know each other at all.

Student writing as identity work

How can we enable students to experience writing as more than transactional?

A text that influenced my thinking on this subject was Mary Lea and Brian Street’s (1998) Student writing in higher education: an academic literacies approach. (The link is to a great retrospective article on the book and its impact.) Lea and Street see ‘disciplinary practices of reading and writing’ as constantly evolving and being contested over, as well as becoming authorised and passed on. The academic literacies that students develop are not just skills with text but are ways of relating to their subject of study, and negotiating a place in that discourse community. Literacies are ‘concerned with ...identity, power and authority’.

In Writing as Identity (also 1998), Ros Ivanic considered how the process of writing asks us to manage several different versions of ourselves. There is the ‘autobiographical self’ through which we tell our ongoing life story, and which has brought us to the point of writing. There are the various sociocultural roles we are offered to adopt as writers, for example in relation to the university and its arcane processes for authorising texts and people. In writing, Ivanic distinguishes also the ‘discoursal self’, which we might think of as writing style or voice, from the ‘authorial’ self, which we might think of as position or stance – how we construct the ‘I’ of the text, and take responsibility for its arguments, observations and positions.

I’m not convinced Ivanic’s different selves can easily be untangled, but what her work does help me to think about is how identity is at stake in writing. How there is never only one identity, or only one outcome of the reflective dialogue between them: ‘In any institutional context there will be several possibilities for selfhood… [but some] of these will be privileged in the sense that the institution accords them more status’ (Ivanic 1998: 27).

How and what we write confers power and privilege as well as identity. Like spoken accent, writing acts as a sign of social capital, and a key to accessing further opportunities.

The magic of writing at university is that it really can produce a new identity. Graduation literally confers this. The hope is that by learning to account for themselves in writing, students produce new resources for autobiography as well as for accreditation. But writing is exposing. It is a bid for credibility that is public, contestable, ‘out there’, and may fail. Although universities have changed in many ways since Ivanic was writing, and are on the whole more diverse and more welcoming of diverse forms of expression, still their role remains to uphold the status of certain kinds of writing. And today’s students, used to the intense scrutiny and self-scrutiny of a life online, are more anxious than ever to maintain credibility when they express themselves, and more anxious than ever about comparisons with a norm.

I think most students experience academic English as a profoundly ‘other’ discourse. If they don’t feel completely excluded from it, they may feel they are writing in a voice that is not their own. All the more so if (some variant of) English isn’t their first language. One way of gaining a sense of ownership is to bring more personal or personally significant material into student writing, so that they have something to say (so that they feel like ‘authors’ of their own material) before they start to write. Hannah Ashley and Katy Lynn, in a lovely essay ‘Ventroloquism 001: How to throw your voice in the academy’, report that this approach gives students greater confidence for writing.

Some students will feel more comfortable than others working with personal material. And there is always a source of tension between the personally available and the authorised. But Ashley and Lynn see this, too, as a source of interest and identity work:

Practices that bring in the "I," like memoir and service [community-based] learning, provide students with an opportunity to see how their own private/community discourses are part of a particular set of popular/commonsense notions which get called into question when they butt up against a different community discourse. Ashley and Lynn 2003

What I like about their work is that they avoid calling student’s private/ community discourse ‘authentic’. They talk about students learning to recognise themselves as writers in dialogue with a host of speakers, writers and discourse communities, some of them authorised by the academy, some personally available, but all of them contestable. In this teaching practice, there is no ‘correct’ way of writing – not even one way of ‘writing as oneself’ – but rather a range of possibilities that students can explore.

Bringing the ‘I’ to writing does require students to take up a stance, even if it is a provisional and partial one that can be revised. In the context of a reflective classroom, ventriloquy or passing as a writer of one kind or another can be made more playful, but it can’t be made completely safe.

Today, as well as the spaces of higher education, students are negotiating their identities in digital networks and communities. So it does seem to me important that research into students’ use of generative AI also looks at how students are being constructed as writers by the AI industry and how they are also managing these constructed selves when they come to write. AI tools and services do not typically address students as producers of knowledge, or as negotiators of different perspectives, however we may encourage these approaches. They do not suggest identity as work, even of a playful kind. What AI offers is a magic cloak, a trick, a fully achieved writing identity. And this in a setting where other ideal selves are also magically available.

Student writing as social activity

I don’t think we can counter the allure of the magic cloak without looking a bit deeper into the theories of language that stand behind the AI industry – and they are different to the ones that have mainly informed practices of academic writing.

Computational linguists have often tangled with the linguistic ideas of Ludwig Wittgenstein, at least as they were expressed in his early Philosophical Investigations. In 1939 there was even an argumentative encounter between Wittgenstein and Alan Turing: his subsequent development of the Turing Test as a kind of language game is rumoured to have been a kind of riposte from Turing to Wittgenstiein (a reminder that I have written more about the Turing Test here.)

It was a student of Wittgenstein’s, Margaret Masterman, who first proposed that words could be represented as (mathematical) vectors describing their relationships with other words in a corpus of text. Her work made language computable without having to start from general rules of grammar or syntax. Each word can be expressed in terms of its relationships to other words. Unlike in formal grammars, new relationships can always be added on, making these models flexible, extensible, and particularly suited to data at scale. Vectors formed the basis of the first working search engines and information retrieval models. Today’s large language models are essentially massive vector databases.

Masterman’s approach can be traced back to Wittgenstein’s insight that definitions can be ‘by example’ or by family resemblance, rather than by logical equivalence. So Wittgenstein is held up (correctly, in my view) as supporting the conceptual shift within AI, away from propositional rules and towards probabilistic modelling. A useful recent article about this shift is called From rules to examples: machine learning’s type of authority. Another very accessible piece is Remembering Ludwig Wittgenstein in an age of AI.

However, neither of these essays, nor Masterman’s original work, take account of another insight of Wittgenstein’s, against which his idea of exemplary language needs to be set. ‘The speaking of language is part of an activity, or a form of life’, he wrote. It is the form of life, the ordinary activities or games people enter into with language, that give language its meaning. GPT4.o, for example, can respond to input text in a ‘context window’ as long as 128k elements. But however massive and inter-related the corpus may be, however impressive the ‘context window’, they are still just text. And Wittgenstein clearly intended by ‘use’ something more than ‘place in surrounding text’ or even ‘web of textual relations’. Wittgenstein meant the social, cultural and practical situations in which people make language work for them: ‘to imagine a language is to imagine a form of life’.

(Arguably, the to-and-fro dialogue between a user and a large language model constitutes a new kind of language game. I find this an interesting idea: but it is still one that would need to be investigated in terms of users’ emergent understanding and meaning-making, and not – in my view – as if the language model itself is participating in a form of life, in Wittgenstein’s sense.)

Margaret Masterman realised Wittgenstein’s first insight through the development of neural nets and machine learning. The Wittgenstein of ‘language games’ is found in other lines of linguistic thought, lines that throw into question whether language models can be said to ‘mean’ at all.

Critic Toril Moi sees Wittgenstein as a philosopher of ‘the exemplary, the public and the shared’. The exemplary in this sense is not the same as the ‘normal’, statistically calculated, but almost the opposite: a space for coming to terms with differences, for encountering the otherness of other people and their meanings. While statistical AI no longer tries to ‘fix’ language in grammatical and syntactic rules, it still insists on a fixed structure. A structure that expresses probabilistic rather than deterministic relationships. A structure that emerges from reams of public data rather than being constructed by self-appointed experts. But still, it is a structure that fixes text in the past, that is supposed to generate meaning while completely resisting any challenge or change.

And it is not the quantifiable differences among words that constitute, for Wittgenstein, a language game, but rather the qualitative differences between people who share a linguistic space, who try to make sense of and with each other. Language is social activity. Meaning is not derived from the relationships among words in the texts of the past, but by exchanging words in the present, in order to make something (socially) happen. In a recent interview Moi summarised her insights like this:

[writing is] action and expression. [We] need to think about what we stake ourselves… this will involve trusting our own experience, although our experience might at the same time need educating.

Student writing as expression and dialogue

According to Moi, then, experience is what education offers. New experiences, new ways of experiencing, and new opportunities to express and share experiences with others. So key elements of student writing are activity - doing things with words – and dialogue – engaging with others, using words.

My friend John Hilsdon is an academic writing specialist who works with students on the basis that academic writing is always doing something. His ‘functions of writing’ framework draws on theories of functional linguistics (that also owe a debt to Wittgenstein). But what matters to students is that by asking simple questions about the role of each piece of text, they start to worry less about word choice and ‘academic style’ and to focus more on communication. For example: how does academic writing produce credibility? It gathers and presents evidence, credits and values other people’s ideas, constructs arguments, reflects on experience, acknowledges situations and limitations, forms judgements. Credibility in writing is an active verb – in fact a whole Bloom’s taxonomy of verbs.

This contrasts with the credibility of synthetic text. Even when it isn’t making factual and interpretive errors, synthetic text can only borrow credibility as a style from the words and word orders in its lexicon. It is poor at making extended arguments, distinguishing good evidence from bad, or taking firm positions. In Ivanic’s sense of the term, it cannot be an author: it cannot have a purpose that is realised through the production of text. I have argued elsewhere that students benefit from accountable assignments that put the purpose of writing first. That taking responsibility for a position is an attribute of an embodied social person who can follow through on their commitments, whether in writing or in speech.

So then we come to dialogue. At the most basic level, talking about writing in class is an important way of mixing codes and discourses, expressing ideas in different registers, and ‘talking back’ at those authoritative writers of academic texts. Diverse varieties and registers of English as well as diverse viewpoints can be welcomed as legitimate resources. This is work that bell hooks spent much of her life doing and advocating for:

Embracing multiculturalism forces educators to focus on the issue of voice…To hear each other, to listen to each other, is an exercise in recognition. bell hooks (1994)

Many European higher education systems use oral assessments, and not only for PhD theses or for discursive subjects. For example, at the end of high school, the European Bacchaleureate requires an oral examination in three subjects, often including maths. In the UK, the idea of oral tests as a way of minimising the generative AI advantage has been derided as a step ‘back to the middle ages’, but it might just as well be seen as a step out into a broader culture of university learning as developing both written and oral communication skills.

‘Talking about writing together’ can become ‘writing about writing together’, live in class, on personal devices or on paper, on post-it notes and whiteboard screens, in shared digital documents and design boards. Again, there has been a reaction against the idea of dealing with generative AI by encouraging students to write live, as though this can only mean an invigilated live exam, heads down, no conferring. As a teacher of creative writing I have made many different uses of classroom time for writing. For some students the instruction ‘just to write’ is liberating. For others it is a chance to share and discuss strategies.

From bell hooks again:

Learning and talking together, we break with the notion that our experience of gaining knowledge is private, individualistic, and competitive.

In breaking with this notion, I believe we come to see dialogue, beyond particular acts of talking/writing together, as part of the nature of writing. In the thinking of Mikhail Bakhtin, all writing is part of a ‘chain of communication’ in which meaning depends on who is being addressed, for what purposes, in what social setting. A piece of writing is never complete or self-contained, since it responds to earlier writings and genres of writing, and because it anticipates a response. Writing exists on a boundary between previous and future texts, self and other perspectives, private expression and public forms.

A focus on academic writing as talking back at someone or writing forward to make a difference can liberate students from a narrow focus on technique, and help them accept their writing as incomplete (imperfect) but engaged. I’ve encouraged this way of thinking myself by asking students to write a postcard or email to an academic author, to review and self-review academic writing (this can be in the style of a film or product review, if it helps students express their opinions more naturally), and by imagining ‘what happened next’ in response to a piece of academic or professional writing.

Bakhtin’s contribution to understanding student writing is explored in another great essay from the early 2000s, by Theresa Lillis: Student Writing as Academic Literacies: drawing on Bakhtin. She writes:

‘Bakhtin’s emphasis on the encounter between difference, on communication and knowledge making built on a dialogic both/and rather than a dialectic either/or, stands in sharp contrast to much academic meaning making. Dialogue within this frame is not just the process of meaning making, but is rather the goal: difference [must be] always kept in play.’

So while dialogue is a given, writing can always strive towards particular qualities of being in dialogue. And this is where I find another fascinating commentary on today’s language models. Bakhtin contrasted two modes of writing. One he called monological or monophonic writing, oriented on achieving authority within the text. The other he called dialogical or polyphonic writing, which acknowledges the diverse and contesting realities of alternate texts and voices. The engagement with other voices need not be a friendly one: parody, irony, pastiche, refutation are all examples of writing that allow another text/author to be heard but kept at a distance. Large language models are, on the face of it, rather good at repeating elements of a genre or style. But they have no sense of distance. They are unable to show, through markers of irony for example, different attitudes towards the style or genre or content that they reproduce. They are unable to bring alternative possibilities forward.

Here, I want to quote from an article on Bakhtin that I read in a rather obscure radical magazine Ceasefire. Published in 2011, it comfortably predates generative AI and so could not have been written in any sense against its logics. And yet I think its relevance is remarkable. I could not quite resist making my own dialogic annotations [in square brackets] on the original.

In monologism, one transcendental perspective or consciousness [the generative transormer model] integrates the entire field, and thus integrates all the signifying practices, ideologies, values and desires that are deemed [statistically] significant… Monologism is taken to close down the world it represents, by pretending to be the ultimate word.

In monologism, ‘truth’, constructed abstractly and systematically from the dominant perspective, is allowed to remove the rights of consciousness [the rights of original authors, especially from marginal perspectives]. Each subject’s ability to produce autonomous meaning is denied. Qualitative difference is rendered quantitative. This performs a kind of discursive ‘death’ of the other, who, as unheard and unrecognised, is in a state of non-being. The monological word ‘gravitates towards itself and its referential object’ [the model refers only to itself.]

We are back, here, with a similar critique of language models to the one I derived from Wittgenstein: that language is not representational form, however complex and inter-related, but action, interaction and expression, and in this:

There is no single meaning to be found in the world, but a vast multitude of contesting meanings. Truth is established by addressivity, engagement and commitment in a particular context.

Writing beyond passing

How can we bring these insights into support for student writing? How can we enable a variety of student ‘others’ to co-exist with the authoritative voice of the academy and its assessment practices? These are key questions generative AI poses to educators. And some of the answers may be quite simple.

We can support the same diversity and freedom of expression in academic writing that has traditionally been allowed in classroom talk and in non-academic genres of expression. We can invite students to make diverse commitments to telling it how it is, or how it seems to them. To express their experience in whatever media they choose – video, audio, graphics, presentations, code, multiple genres of text, including the popular and the marginalised. Being able to translate academic and professional concepts into different genres is something we can treat as a sign of a deep engagement, rather than a departure from the ‘norm’ of authorised expression.

This will mean coming up against students’ own aspirations to communicate in a privileged register. After all, it is not only universities that reward conformity to these codes. But students understand that the game has changed. Anyone with an internet connection can produce writing that ‘passes’ as undergraduate level work, at least to most readers. Recruitment practices are beginning to reflect employers’ understanding of this change too. Students will quickly see the value in developing a repertoire of means of expression, including some that do not pass through a synthetic editor or indeed through a screen at all.

We can also commit to forms of collaborative and dialogical writing, that universities always claim to value but almost never assess. One strange twist in the generative AI story has been a flowering of creative ideas for writing – role plays, parodies, peer review, annotation, self-explanation, argumentation mapping, writing for different audiences. The tragedy is that these ideas have all been put forward to support dialogue with a generative agent. How much better for student writing if this same ingenuity and playfulness was applied to supporting dialogue and collaboration in the writing classroom. This need not exclude generative agents, but it would prioritise the encounter with other students, who are at a similar stage of writing development to one another, and who have the same to gain (and to risk) from taking a more playful approach to their writing.

The expertise that resides in academic writing centres has never been more critical to universities than at the present time, and goes far beyond my own personal interest in the topic. I was able to tap into that expertise at the webinar, and I’ve summarised some of the participants’ suggestions for writing practice in the rest of this post. Even so, it can only be a limited and partial view of what is possible.

In the 'writing-to-learn' tradition, students are encouraged to write actively, collaboratively, and live in class. The idea of producing a finished text is let go. Instead, students write in short bursts, interspersed with peer conversation where active listening and mirroring are encouraged.

Several participants were using generative text in the context of highly scaffolded guidance. Students undertake separate activities such as: devise research questions, define terms, generate literature search criteria, summarise key texts, produce a research plan, develop a research instrument. Generative capabilities are then used to refine each stage. I welcome anything that makes the writing process less mysterious, more accessible, and available for reflection and review. As with my observations about dialogue, however, I have still to see evidence that a generative agent is offering more than an opportunity to reflect would offer, or (better) to engage with peers. There may be reasons why students find a generative agent more available and less challenging than other student writers – but this is a preference we might also challenge.

One webinar participant described giving students an essay question that they responded to in class with an outline. A copy was handed in as a first submission. Students then went away, did additional research, added references, and were able to use text generators to ‘get feedback and edit’ if they wanted to. The final submission included a reflection on what they had done and why. ‘It turned out that despite being able to use as much AI as they wanted, they absolutely could not do a good job unless they already had a good grasp of the basics of structure and argumentation and synthesis. If they used AI productively to enhance their own ideas they tended to do well.’

If students are to focus on the process of writing, assessment practice needs to change. So students might be invited to submit several drafts, or samples of writing in several stages (planning, reading-to-write, note-making, drafting, annotating). Students might also submit intermediate materials such as notes, annotated sources, concept maps, and responses to feedback. One participant described this as a ‘process-folio’; another as a ‘work book’.

Many suggestions involved making writing ‘live’ and ‘in person’, or at least engaging in live talk about writing. When these approaches were first suggested as a means of countering ‘AI cheating’, there was a fairly hostile reaction. Wasn’t this a step back into the medieval age? If by ‘live writing’ you think only of a traditional, closed-book, invigilated examination, and if by ‘talking about writing’ you think only of a formal viva, it’s easy to see these as backward steps both for equity and for the diversity of ways students have come to be assessed. But low- or zero-stakes live writing is entirely different. It can integrate writing more easily and naturally into other forms of expression (talking, note-making, doodling). Talking about writing brings this often intensely private activity into the light of day, and into the zone of proximal development, where students can be exposed to different strategies and possibilities beyond those they would naturally reach for themselves.

After all, aspects of university learning that involve live practice or performance, such as lab and field work, professional and creative practice, are often highly motivating for students and they engage different pedagogies on the part of teachers too. In my own teaching practice (founded in creative writing) I have encouraged students to treat academic writing as a foreign language, to parody it, to ‘write on from’ a particular academic paper or chapter as a form of fan fiction, to engineer a paragraph as a machine with moving parts, to cut and paste with literal scissors and glue. (A recent workshop that I ran with three colleagues, ‘generative alternatives’, explored these and other creative possibilities.)

Other suggestions in the webinar offered a twist on the classic Turing set-up: invite students into the ‘judge’s position, and ask them how well synthetic text ‘passes’ as good student writing. This calls for both critical engagement and a sense of agency. Both are possible, I think, if students have some understanding of the subject matter and of writing practice. However, as soon as students leave the classroom where this agency is being offered, and where the position of judge is being held for them, they are likely to slip back to the more familiar side of the table.

The authority vested in academics as assessors is real. And the messages pushed at students by the AI industry are all about academics as enemies of ‘passing’, implacable judges, interested only in catching students out. So while I think this critical move is essential, I’m not sure it’s sufficient without ongoing support for judgement as part of the process of writing, as something students are entitled to do already but that teaching can actively support (remember Moi’s ‘experience might at the same time need educating’?)

The webinar also provided an opportunity for debate on how universities are responding to the challenge of generative AI. Many participants noted that departments were giving students very different guidance, ‘which recognises the complexity of the technology and how it relates to different subjects, but can be really confusing for students (especially on joint degrees)’.

The conversation touched on theories of language: ‘we are emphasising to students (in some disciplines anyway) just what Helen has been talking about—learning and self-expression through writing’; ‘might it all come down not in the register of language used but in the ability of the author to express and indicate stance and criticality?’ Participants debated to what extent Ivanic’s idea of ‘authorship’ (‘the ability of the author to express and indicate stance’; ‘organisation of ideas’, ‘reasoning’) can be separated from her idea of ‘discourse’ (‘vocabulary and grammar’, ‘accuracy of expression’). While this question was not resolved, all agreed that the register students speak and write in makes a difference to their perceived credibility as authors, and that this is a connection no academic exercise or course of study can entirely break. The reality is that: ‘Global Englishes are not accepted as “passing” yet.’

There were, not surprisingly, contrasting views on how students experience generative AI. Some reported that students are anxious not to use or to overuse it: ‘because it takes away their power of thinking’; ‘it completely erases the writer's voice, the writer's “accent”’.  ‘Perhaps it’s about getting students to think about what will be effaced or muted (their critical voice, their intellectual identity) if they ask a chatbot to impersonate them.’

Others felt that students benefited from access to a more ‘formal’ voice for expressing themselves. ‘AI opens up so many new multimodal writing processes, I wish I had time to get my students to experiment with them all’. ‘AI makes them start off a step further, since structure/grammar and the lot are given. They need to delve deeper and attempt criticality much earlier in their careers as academic writers’.

But there was consensus, finally, on the need to support writing beyond ‘passing’, recognising that the main losers from a transactional approach are students themselves. Academic writing can provide opportunities to develop their thinking, to find out what they think, to engage in thoughtful dialogue with people and texts, and to know themselves as different to who they thought they were. These are all more possible when assignments are oriented towards dialogue and expression, as Toril Moi defines them, and when, as bell hooks demands, writing classrooms are spaces ‘to hear each other, to listen to each other… an exercise in recognition’. By which we have to mean something more profound than an exercise in identifying who passes as (the right kind of human), and who fails.

Subscribe now