Oracles and Recursive Self Improvement (opens in new tab)

I've been thinking about recursive self improvement. But especially the likelihood that it will be important soon. Soon means with current LLMs that might just give up or delete the test set or otherwise detach from reality. Call the probability of that happening P. You can estimate P by looking at research tasks and how much they need to get helped along by humans to stay on task.