Part of what I'm struggling with is that "existential risk" isn't a claim, it's a concept. So saying "existential risk is an extraordinary claim" makes no sense. "Teleportation will be invented next year" is a very specific claim I suppose you could call extraordinary. I'm not sure what specific claim you're evaluating. Maybe it's the claim that "there is a realistic chance of an existential catastrophe occurring in the next 5 years" or something like that?

I also don't know what criteria you use to call something "extraordinary" or not. What is it that actually makes general relativity extraordinary?

Are you essentially saying our prior belief in an existential catastrophe occurring should be very low because something so drastic has never happened before, and the evidence for it is not strong enough to shift us to a position where it should be a priority for us? This would be a framing that makes more sense to me.

In this context I don’t understand the difference between x-risk being an extraordinary claim and the risk being low.

You say that you’re not arguing if existential risk is high or low, but that’s it’s an extraordinary claim. But your arguments to me just seem to be arguing that the risk is low. Which is fine, I’m just confused by the distinction you’re trying to draw here.

You say we need to have extraordinary evidence to be concerned about it, but I don’t really get this. I think the amount of concern we should have about x-risk should be proportional to both the risk level and the amount that we think would be lost if a catastrophe occurred. There are people who think the risk is 1% (which is pretty low) and who are still extremely concerned because there’s just so much at stake. 

Basically I just see your post as arguing that the risk seems low and presumably therefore that you think there are higher priority things we should be doing. That’s fine, you’re just framing your argument in a confusing way. 

I wonder what the mechanism is here. If I had to guess, I’d say EA is most useful when you’re uncertain about how to do the most good, but many highly-engaged EAs have now decided reducing AI extinction risk is the overwhelming priority, and there are better forums to discuss that.

I agree AI is the most important thing to think about, but I don’t know that the community has thought deeply enough about the priorities we should have to make a transition to super intelligent AI go well. So I share your sadness at the decline of the forum.

I'm not sure this is as obvious as you're making out. If you give a misaligned ASI a goal to maximize its own wellbeing it could still do that very well, it may also just do other things you didn't intend for it to do (like wipe out humans).

A paperclip maximizer does its job really well after all, and we wouldn't call it aligned.

I think the article gives the impression that EAs are generally happy for humans to go extinct if they are replaced by happy AIs. But this isn't my impression. EA, at least outwardly, is very concerned about human extinction! Almost obsessively so!

I think a utilitarian can be in favor of humans staying in control and surviving. They might not think AIs will be capable of wellbeing, or might think there is a good chance they will have negative wellbeing, or might not be confident they will create a flourishing future.

What annoyed me most about the Hendryck's article is that he essentially assumes impartial utilitarianism must be wrong because it can lead to a conclusion he (and others) don't like. Impartial utilitarianism also would have suggested to slave owners they shouldn't keep slaves - a conclusion they and others wouldn't have liked!

I would have liked more philosophical discussion about if impartiality is or is not a good moral principle. But I suppose he wasn't interested in getting into that.

Yeah I think that people with suffering-focused views could accept RC but not VRC.

RC doesn't consider anyone with a negative life, but VRC considers arbitrarily many arbitrarily negative lives.

I think it would be an interesting thing to add if you ever do a v2.

This is cool! Did you consider testing for the very repugnant conclusion? Maybe I'm misremembering but I don't think any of your choices tested if we can just outweigh adding suffering people by adding more happy people.

The Very Repugnant Conclusion (taken from here)

A population of arbitrarily many lives with arbitrarily high welfare is worse than a population of arbitrarily many arbitrarily negative lives plus sufficiently many “ε-lives”^[1] that each have an arbitrarily small quantity of positive welfare (Figure 4.1).