I'm really doubtful of this generalising to other subjects areas. I feel like the field is pretending hallucinations don't exist anymore but they do, and in most disciplines AI is generating massively more research with incorrect information, bad citations and outdated, cliched arguments.

I would argue that AI is actually decelerating research and innovation in the vast majority of subject areas. In my discipline, it's extremely rare now to see a fresh argument or new innovation, in what was extremely common two years ago. Instead, academic journals are overwhelmed by poor quality AI papers, to the point of the research system itself being at risk of collapse.

I get that people here are mainly focused on coding, but when you bury your head in the sand about the broader R&D space it just reads as completely out of touch.

I definitely think you're right that vague prompts will be the main scenario. It's possible that the same prompt could have a legal and nonlegal answer if written too vaguely or without specifying certain boundaries or methods. I've seen some research on misalignment that suggests this too.

The other situation is users who do in fact intend a crime. Increasingly, they will have to jailbreak the AI, because the AI companies will put in more safeguards over time.

So dealing with both vague prompting and jailbreaks could help mitigate some of this.

In terms of examples, I gave the example of insider trading because I think it's the easiest to imagine. If you get an agent to trade on the market and it does so via gaining data from a company server, then that alone might be enough. Various white collar crimes are quite small in terms of what you actually need to do to commit them.

Fraud is another example that we are seeing already. Because AI hallucinates and makes things up, it's kind of already vulnerable to fraud scenarios, and in particular misrepresenting the user it works for. So an AI could say their user is a professional (X) when they're just a student, or something similar. In many jurisdictions claiming a licence you don't have is already a crime.