I suspect that the way most people (not just people with high leverage) interact with LLMs today is more relevant to catastrophic risk than it first appears to be.
I worry that the way most people interact with LLMs today is writing the script for how we will interact with increasingly intelligent AIs going forward.
If ~ASI is close, we should be identifying a rough optimal idea of what a human’s relationship with an LLM should look like (eg preserving critical thinking, thinking things through oneself, aiming to understand something rather than have the LLM do the thinking/understanding for you) in order to avoid ‘locking in’ dynamics that could enable gradual disempowerment.
Although how people with the most leverage are interacting with current AI systems is probably the most important factor here (eg people with decisionmaking power in society), I think that it's important we foster a culture more broadly which values human autonomy, especially since each insitution in society likely faces unique risks around how it interacts with AI.
To be clear, if alignment and superhuman intelligence becomes proven over time then I do think some degree of gradual handover of agency on important issues is likely overall value-positive. But I don't think we're at that stage.
I suspect that the way most people (not just people with high leverage) interact with LLMs today is more relevant to catastrophic risk than it first appears to be.