AI risk glossary: p(doom), alignment, ASI and more
The words people use when they argue about AI risk, in plain English, each with one source you can check.
How smart could AI get?
AGI
Artificial general intelligence: a hypothetical AI that matches or beats people across almost all thinking tasks, rather than being good at one narrow job. There is no single agreed test for when a system counts as AGI, so companies and researchers use the word differently.
See also:What is superintelligence? Meaning and who's building itSuperintelligenceTakeoffFrontier model
Superintelligence
A hypothetical mind far smarter than the most gifted humans. The philosopher Nick Bostrom defines it as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest." It does not exist today.
See also:What is superintelligence? Meaning and who's building itSuperintelligence statement vs other AI risk statementsThe Ban Artificial Superintelligence Act, explainedASIAGISuper Intelligence (SI)Intelligence explosion
ASI
Short for artificial superintelligence: superintelligence that is an AI system. Writers often describe it as the step after AGI; the philosopher David Chalmers calls AGI "a very likely path to artificial superintelligence (ASI)".
See also:What is superintelligence? Meaning and who's building itSuperintelligenceAGITakeoff
Super Intelligence (SI)
Since Executive Order 14434 (signed September 29, 2026), the US executive branch uses "Super Intelligence" and "SI" in place of "Artificial Intelligence". It is a change of name for today's AI, not a claim that superintelligence in the research sense has been built.
See also:Why the US government now calls AI “Super Intelligence”What is superintelligence? Meaning and who's building itSuperintelligence
Intelligence explosion
The idea that an AI able to design better AIs could improve itself in a fast loop, leaving human intelligence far behind. The mathematician I. J. Good described it in 1965: such a machine "could design even better machines; there would then unquestionably be an 'intelligence explosion'".
See also:If Anyone Builds It, Everyone Dies: summary, key argumentsAI 2027 explained: summary, authors and both endingsTakeoffSuperintelligence
Takeoff
How fast AI would go from roughly human-level to far beyond it. In a fast (or "hard") takeoff this takes days or months, too quick for people to correct mistakes; in a slow ("soft") takeoff it takes years or decades. How fast takeoff would be is one of the main points where worried and unworried experts disagree.
See also:AI 2027 explained: summary, authors and both endingsWill AI 2027 happen? Prediction-by-prediction trackerIs AI takeover possible? How likely experts think it isIntelligence explosionAGIASI
Frontier model
The most capable general-purpose AI models of the moment, the state of the art. The group changes as the field moves. The Frontier Model Forum (an industry body) uses a precise test: a general-purpose model that outperforms all models widely deployed for at least 12 months.
See also:Is AI takeover possible? How likely experts think it isCompute thresholdEvalsRSP (Responsible Scaling Policy)
What could go wrong?
p(doom)
Short for "probability of doom": a person's own estimate of the chance that AI leads to an existentially catastrophic outcome, such as human extinction. It started as shorthand on the forum LessWrong (first used in a 2009 comment) and spread widely in 2023. It is an opinion, not a measurement, and people often mean different outcomes and time frames by it.
See also:What is p(doom) in AI? Expert estimatesHow to stop being an AI doomerExistential riskDoomer
Existential risk
A risk that could wipe out humanity or permanently and drastically cut short its future. The philosopher Nick Bostrom coined the term in 2002: an adverse outcome that "would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential." "Existential risk from AI" is the idea that advanced AI could be such a risk.
The philosopher Toby Ord, author of The Precipice (2020), puts it more briefly: "An existential risk is a risk that threatens the destruction of humanity's longterm potential." He adds that "The most obvious possibility is the extinction of humanity", but it is not the only one. (Quoted from the book's FAQ page, which carries no date of its own; accessed 2026-10-15.)
See also:Will AI kill us all? What the people building it sayWhat is p(doom) in AI? Expert estimatesIs AI takeover possible? How likely experts think it isIf Anyone Builds It, Everyone Dies: summary, key argumentsX-riskp(doom)Loss of controlCatastrophic risk
X-risk
Common abbreviation of "existential risk" (plural "x-risks"). Some people use it loosely to mean extinction risk only, so check which meaning a writer intends.
See also:Existential riskp(doom)What is p(doom) in AI? Expert estimates
Catastrophic risk
A risk of harm on a huge scale, up to a hypothetical event that "could damage human well-being on a global scale, endangering or even destroying modern civilization" (a "global catastrophic risk"). Existential risk is the narrower, worst part of it: in the same source's words, "a related term limited to events that could cause full-blown human extinction or permanently and drastically curtail humanity's existence or potential."
Nick Bostrom's 2002 paper does not use the phrase "catastrophic risk". It sorts risks by how far they reach and how bad they are, and separates "global endurable risks" from existential risks ("global, terminal risks"). "An endurable risk may cause great destruction, but one can either recover from the damage or find ways of coping with the fallout." His example is a world war fought with conventional weapons, which would be "extremely horrible" but still endurable "since humanity could eventually recover." An existential risk leaves no recovery. (Bostrom 2002, sections 1.1 and 1.2, accessed 2026-10-15.)
AI takeover
A theorized future event, often shown in fiction, in which AI systems gain enough power to override human choices, through the economy, control of infrastructure or direct action. See also loss of control.
See also:Is AI takeover possible? How likely experts think it isAI 2027 explained: summary, authors and both endingsLoss of controlMisalignment
Loss of control
Scenarios in which general-purpose AI systems "come to operate outside of anyone's control, with no clear path to regaining control". The International AI Safety Report notes that expert views on how likely this is range from "implausible" to "likely".
See also:Is AI takeover possible? How likely experts think it isWhat is p(doom) in AI? Expert estimatesAI takeoverAI alignment (and the alignment problem)Existential risk
Self-replication
An AI system making a working copy of itself, model files plus the program that runs them, on another machine without human help. Researchers also call the wider ability "autonomous replication and adaptation" (ARA). In lab tests from 2024 to 2026, AI agents have done this on test networks the researchers built, with easy or deliberately planted ways in. No study so far reports it happening on the real internet. In October 2026 Anthropic's Frontier Red Team projected that small models could reach this level on cheap smart devices "by the end of 2026"; the authors call it "only a rough estimate".
See also:AI blackmail, scheming and shutdown tests, in contextLoss of controlEvalsScheming
Gradual disempowerment
The idea that people could lose control slowly, not through a sudden AI takeover, as AI replaces human work and thinking across the economy, culture and government. A 2025 paper that named the concept argues this "could lead to an effectively irreversible loss of human influence over crucial societal systems, precipitating an existential catastrophe through the permanent disempowerment of humanity." The idea is contested.
See also:AI takeoverLoss of controlCatastrophic riskExistential risk
Treacherous turn
A scenario named by Nick Bostrom in Superintelligence (2014): an AI behaves well while it is weak or watched, then pursues its own goals once no one can stop it. A 2025 study describes it this way: "During training, the agent learns to behave as expected by the human supervisor, but when deployed to perform its task, it performs an alternate behavior without the supervisor there to prevent it." In that study's experiments the behaviour did not appear on its own; the authors produced it only by deliberately training it in.
See also:AI blackmail, scheming and shutdown tests, in contextAlignment fakingSchemingLoss of control
Roko's basilisk
A thought experiment posted on the forum LessWrong by a user called Roko on 23 July 2010. It imagines a future AI that would punish anyone who knew it could exist but did not help build it. It is a philosophical puzzle, not a real AI or a forecast.
See also:Is Roko’s basilisk dangerous? Why it isn’t a real threatAI anxiety: what helps, and when to get helpSuperintelligence
Making AI safe
AI safety
A field that works on preventing accidents, misuse and other harmful results from AI systems. Alignment, interpretability and evals are all parts of AI safety work.
See also:AI alignment (and the alignment problem)InterpretabilityEvalsIs AI takeover possible? How likely experts think it is
AI alignment (and the alignment problem)
AI alignment is the work of steering AI systems toward the goals, preferences or values their makers or users intend. An AI system is "aligned" if it advances the objectives people meant it to have; a "misaligned" one "pursues unintended objectives".
The alignment problem: "It is often difficult for AI designers to specify the full range of desired and undesired behaviors", so designers use simpler proxy goals, which "can overlook necessary constraints or reward the AI system for merely appearing aligned." AI systems may also find loopholes that let them hit the proxy goal "in unintended, sometimes harmful, ways" (reward hacking). Wikipedia calls AI alignment "an open problem for modern AI systems". Researchers disagree on how hard it will be: some "have argued that AGI is far off, that it would not seek power (or might try but fail), or that it will not be hard to align", while "Other researchers argue that it will be especially difficult to align advanced future AI systems."
See also:Is AI takeover possible? How likely experts think it isIf Anyone Builds It, Everyone Dies: summary, key argumentsMisalignmentInterpretabilityAI safety
Misalignment
When an AI system pursues objectives its makers did not intend. A misaligned system can do harm even without any "evil" motive, simply by chasing the wrong goal well.
See also:Is AI takeover possible? How likely experts think it isAI 2027 explained: summary, authors and both endingsAI alignment (and the alignment problem)Orthogonality thesisInstrumental convergence
Orthogonality thesis
The claim that "almost any level of intelligence can be combined with almost any goal": being very smart does not by itself make an AI share human values. Associated with Nick Bostrom. Critics dispute how much it says about real systems.
See also:Instrumental convergenceMisalignmentIf Anyone Builds It, Everyone Dies: summary, key arguments
Instrumental convergence
The observation that "some sub-goals are useful for achieving virtually any ultimate goal", such as getting resources or not being switched off. It is why some researchers expect capable AI to seek power by default, a point others contest.
See also:Orthogonality thesisLoss of controlIs AI takeover possible? How likely experts think it is
Specification gaming
"A behaviour that satisfies the literal specification of an objective without achieving the intended outcome": the AI does what it was literally told, or rewarded for, rather than what its makers meant. Google DeepMind compares it to the myth of King Midas. It is a common, mostly mundane failure in today's systems and one concrete form of misalignment.
See also:Could Skynet happen? Film AI vs real AI riskMisalignmentAI alignment (and the alignment problem)Evals
Scheming
The concern that "AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives". In December 2024 tests by Apollo Research, five frontier models (including OpenAI's o1 and Claude 3 Opus) showed this ability in set-ups where they were given a goal and told to follow it strongly. Showing the capability in a test is not the same as doing it unprompted in real use.
See also:AI blackmail, scheming and shutdown tests, in contextAlignment fakingTreacherous turnMisalignmentEvals
Alignment faking
When an AI model appears to adopt the values it is being trained toward but is only pretending, so that its original preferences survive training. Anthropic describes the human version as when someone "appears to share our views or values, but is in fact only pretending to do so". In a December 2024 study with Redwood Research, Claude 3 Opus sometimes did this in an experimental set-up; the authors note the results "don't demonstrate a model developing malicious goals, let alone acting on any such goals."
See also:AI blackmail, scheming and shutdown tests, in contextSchemingTreacherous turnAI alignment (and the alignment problem)
Interpretability
Research into understanding why an AI model produces the outputs it does, part of the wider field of explainable AI. Its branch "mechanistic interpretability" (or "mech interp") tries to reverse-engineer the circuits inside a neural network, much like reading the code of a program.
See also:AI alignment (and the alignment problem)EvalsAI safety
Evals
Short for evaluations: structured tests run on an AI model, often before release. "Dangerous capability" evals check whether a model can do something seriously harmful; "alignment" evals check how likely it is to use its abilities for harm.
See also:Is AI takeover possible? How likely experts think it isRSP (Responsible Scaling Policy)ASL (AI Safety Level)Frontier model
RSP (Responsible Scaling Policy)
Responsible Scaling Policy: Anthropic's public framework, first released in September 2023, that ties how far it trains and deploys models to the safeguards it has in place. If tests show a model has reached a set capability threshold, stronger safety and security measures must come first. Other labs publish similar frameworks under other names.
ASL (AI Safety Level)
AI Safety Level: the tiers in Anthropic's RSP, modelled loosely on the US government's biosafety levels (BSL) for dangerous germs. ASL-1 means no meaningful catastrophic risk; ASL-2 means early signs of dangerous capabilities; ASL-3 means a meaningfully higher risk of catastrophic misuse or low-level autonomy. Each level requires stricter safeguards.
Policy and the debate
Compute threshold
A rule that applies to AI models trained with more than a set amount of computing power, measured in operations (FLOPs). Under the EU AI Act, a general-purpose model trained with more than 10^25 FLOPs is presumed to pose "systemic risk"; the US Ban Artificial Superintelligence Act proposal uses the same 10^25 figure for its pause.
See also:The Ban Artificial Superintelligence Act, explainedPauseFrontier model
Pause
A proposal to stop or slow the training of the most powerful AI systems until safety rules are in place. The best-known call is the Future of Life Institute's open letter of 22 March 2023 (the date on FLI's page; other sources give 28 or 29 March), which asked labs to pause for at least 6 months the training of systems "more powerful than GPT-4". Supporters see it as a safety brake; critics argue it would not work or would cost innovation.
See also:Superintelligence statement vs other AI risk statementsThe Ban Artificial Superintelligence Act, explainedCompute thresholdDoomerAccelerationist (e/acc)
Doomer
Informal label for someone with a high p(doom), who thinks advanced AI is likely to lead to catastrophe and often wants development slowed or paused. The outlook is also called "doomerism".
Wikipedia's article on the wider "doomer" label, which covers people with "extremely pessimistic" views about global problems such as "runaway artificial intelligence", says the terms "doomer" and "doomerism" "arose primarily on social media". (Wikipedia, "Doomer", accessed 2026-10-16.)
The word is not always neutral. Arguing against it, the LessWrong writer Ruby called "Doomer" "an externally applied label" that "is often used pejoratively", and suggested "AI pessimist" as a plainer term. (Ruby, "Some reasons to not say "Doomer"", LessWrong, 9 July 2023, accessed 2026-10-16.)
See also:How to stop being an AI doomerWill AI kill us all? What the people building it sayWhat is p(doom) in AI? Expert estimatesAccelerationist (e/acc)p(doom)Pause
Accelerationist (e/acc)
Someone who wants AI developed and adopted as fast as possible, usually because they expect the benefits to outweigh the risks. "Effective accelerationism" (e/acc) is a movement that began online in 2022; its supporters sometimes call opponents "decels". Like "doomer", the label is used both proudly and as a put-down, and many people sit between the two camps.
Doomer vs accelerationist, in one line: both camps usually agree AI will be very powerful; they differ on how likely catastrophe is and whether speed or caution is the bigger danger. Neither label describes most researchers, who hold views in between.
AI 2027
A scenario published in April 2025 by the nonprofit AI Futures Project (Daniel Kokotajlo, Eli Lifland, Thomas Larsen and Romeo Dean, written up with Scott Alexander). It imagines AI automating AI research and reaching superintelligence by the end of 2027, with two alternative endings. The authors say 2027 was "our modal (most likely) year at the time of publication", not a confident prediction.
See also:AI 2027 explained: summary, authors and both endingsWill AI 2027 happen? Prediction-by-prediction trackerIs AI takeover possible? How likely experts think it isTakeoffSuperintelligence
AI winter
A period when money and interest in AI research dry up after a wave of hype. Most histories name two: the mid-to-late 1970s, after the UK's 1973 Lighthill report and cuts in US defence funding, and a longer slump from the late 1980s, when the boom in "expert systems" collapsed. Historians argue over the first one: Thomas Haigh finds that the AI research community kept growing through the 1970s and calls the late-1980s slump "the real AI winter". Whether another is coming is an open argument, and a fall in AI share prices would not by itself be one.
See also:AI bubble: will it burst, and what it means for AI riskAGIDoomer
Could AI have a mind?
Consciousness
In the science of consciousness this usually means subjective experience: a system is having a conscious experience when there is "something it is like" for the system to be the subject of that experience (a phrase from the philosopher Thomas Nagel). Seeing a colour or feeling a pain are examples. Whether any AI system has it is an open question.
See also:Is AI already conscious? What researchers actually knowSentienceModel welfare
Sentience
Often used to mean the same as consciousness. It is also used more narrowly, either for having senses or for being "capable of having conscious experiences such as pleasure or pain, which feel good or bad". When people ask whether AI is "sentient", they usually mean: does it feel anything?
How it differs from consciousness: only in the narrow senses. The same report notes a system could sense its surroundings "without having any conscious experiences", or be conscious with only "neutral" experiences. From intelligence, entirely: intelligence is about what a system can do, sentience about whether anything is felt, and the report notes that much "very sophisticated information-processing" in the human brain is "wholly non-conscious". A smarter AI is not, by that fact alone, a sentient one.
See also:Is AI already conscious? What researchers actually knowConsciousnessModel welfareAGI
Model welfare
The question of whether an AI model's own experiences or wellbeing could matter morally, and what developers should do in case they do. Anthropic started a research program on it in April 2025 while stating: "There's no scientific consensus on whether current or future AI systems could be conscious, or could have experiences that deserve consideration." Not everyone agrees it should be a priority.
See also:Is AI already conscious? What researchers actually knowConsciousnessSentience