The term for the idea that AI could improve itself indefinitely is “recursive self-improvement” or RSI.

A related term for technology having achieved RSI is “Artificial General Intelligence” or AGI.

RSI’s impossibility stems from epistemological limits (no self-bootstrapping without grounding), computational bounds (diminishing returns, uncomputability), and philosophical subjectivity (“improvement” as human-relative, not machine-quantifiable). For instance, if “the good” is an abstraction requiring nuanced values, pure recursion can’t capture it without human-like qualia or external input.

Related reading: The Illusion of Self-Improvement: Why AI Can’t Think Its Way to Genius