Edit 20th October:

References to the infohazard I describe below have now essentially been removed from the post. Thanks very much to Ariel for engaging in productive dialogue and considering my suggestions!

Original text:

I think one of the ideas that is heavily referenced in this post is a serious infohazard, even despite being presented carefully and alongside objections. I would recommend instead that all references to this idea be removed completely.

[Description of the infohazard now redacted.]

As an AI safety researcher, I still think this idea is very implausible, mostly because I think it implies one of several claims that are all very extraordinary from my perspective:

[Description of counter-arguments redacted.]

But even with all the hedging and counter-arguments, I can imagine this idea being intuitively compelling to some people, and leading to morally repugnant behaviour (including potentially in important decisions).

Edit: Why?

1. A lot of people here have a bias towards taking unusual ideas seriously / scout mindset.
2. As @Yarrow Bouchard 🔸 said elsewhere, strong objections aren't always forthcoming, even to shaky arguments. It took me a while to form a view on this argument myself.

Unfortunately I don't have time to write a thorough response but I just wanted to flag that I think this post fails to seriously engage with any of the strong counter-arguments to the central point (including the objections that were mentioned).

For starters, acting under moral uncertainty is a lot more complicated than briefly considering some abstract notion of virtue ethics and assuming that it endorses the same decisions. I think virtue ethics would more likely have a problem with stopping some of the "low return on sacrifice" actions because this would reflect poor moral character, e.g. poor integrity. Similarly, deontological views would likely caution against being willing to cause harm (e.g. human or animal rights violations) in order to do more good in other ways.

Even from a purely consequentialist perspective, I think the indirect impacts of "low RoS" actions are being unduly shrugged off here, such as signalling and value drift. E.g. the suggestion of "tracking impact" as a solution to the latter seems to involve a misunderstanding. Value drift is not about losing sight of whether your actions are effective, but losing the motivation to care about whether your actions are effective. If you get rid of all the most visible / obvious ways you're doing good in your everyday life, you might start seeing yourself as less of a morally-motivated person and eventually stop tracking your less-visible impact too.

My understanding is that the self-effacing utilitarian is not strictly an 'ex-utilitarian', in that they are still using the same types of rightness criteria as a utilitarian (at least with respect to world-states). Although they may try to deceive themselves into actually believing another theory, since this would better achieve their rightness criterion, that is not the same as abandoning utilitarianism on the basis that it was somehow refuted by certain events. In other words, as you say, they're switching theories "on consequentialist grounds". Hence they're still a consequentialist in the sense that is philosophically important here.

Brilliant post. Thanks for writing it. I just want to add to what you said about ethics. It seems that evaluating whether an action / event is good or bad itself presupposes an ethical theory.^[1] Hence I think a lot of the claims that are being made can be described as either (a) this event shows vividly how strongly utilitarianism can conflict with 'common-sense morality' (or our intuitions)^[2] or (b) trying to follow^[3] utilitarianism tends to lead to outcomes which are bad by the lights of utilitarianism (or perhaps some other theory). The first of these seems not particularly interesting to me, as suggested in your post, and the second is a separate point entirely - but is nonetheless often being presented as a criticism of utilitarianism.

1. ^{^}

   Someone else made this point before me in another post but I can’t find their comment.

2. ^{^}

   But note that this applies mostly to naive act utilitarianism.

3. ^{^}

   By which I mean 'act in accordance' with, but it's worth noting that this is pretty underdetermined. For instance, doing EV calculations is not the only way to act in accordance with utilitarianism.

I believe the ‘walls of text’ that Adrian is referring to are mine. I'd just like to clarify that I was not trying to collapse the distinction between a decision procedure and the rightness criterion of utilitarianism. I was merely arguing that the concept of expected value can be used both to decide what action should be taken (at least in certain circumstances)^[1] and whether an action is / was morally right (arguably in all circumstances) - indeed, this is a popular formulation of utilitarianism. I was also trying to point out that whether an action is good, ex ante, is not necessarily identical to whether the consequences of that action are good, ex post. If anyone wants more detail you can view my comments here.

1. ^{^}

   Although usually other decision procedures, like following general rules, are more advisable, even if one maintains the same rightness criterion.

I'm not sure I agree with this. As far as I can tell the EA community has always been quite focused on being inclusive, kind and welcoming - see for instance this and this post from CEA, which are both years old. I'm very sorry to hear about the OP's experiences of course, and honestly surprised personally since my own experience has been a lot more positive. However, this doesn't automatically imply to me that we need a whole new community or something to that effect.

I would see this more as presenting an opportunity to improve our culture and amend any failures that our currently happening despite the efforts of a lot of community leaders. I don't think there's a 'fundamental flaw' in how the EA community is trying to operate in that respect. Also it seems to me that distancing the EA brand in this way you're suggesting would potentially incentivize it to become even less human and amiable - because then it would be distinguished by being the 'weird, rationalist / philosophical community'. (Not to mention that it would seemingly decrease opportunities for collaboration with the 'other community' and create confusion for those looking to get involved in EA.)

Edit: Just to be clear, I'm not making any general claims here about how successful the EA community has been in implementing the ideals I mentioned above. Obviously this post points to updating against that.

No worries. It is interesting though that you think my comment is a great example when it was meant to be a rebuttal. What I'm trying to say is, I wouldn't really identify as a 'utilitarian' myself, so I don't think I really have a vested interest in this debate. Nonetheless, I don't think utilitarianism 'breaks down' in this scenario, as you seem to be suggesting. I think very poorly-formulated versions do, but those are not commonly defended, and with some adjustments utilitarianism can accomodate most of our intuitions very well (including the ones that are relevant here). I'm also not sure what the basis is of the suggestion that utilitarianism works worse when a situation is more unique and there is more context to factor in.

To reiterate, I think the right move is (progressive) adjustments to a theory, and moral uncertainty (where relevant), which both seem significantly more rational than particularism. It's very unclear to me how we can know that it's 'impossible or unworkable' to find a system that would guide our thinking in all situations. Indeed some versions of moral uncertainty already seem to do this pretty well. I also would object to classifying moral uncertainty as an 'ad-hoc patch'. It wasn't initially developed to better accommodate our intuitions, but simply because as a matter of fact we find ourselves in the position of uncertainty with respect to what moral theory is correct (or 'preferable'), just like with empirical uncertainty.

I can't speak for others, but this isn't the reason I'm defending utilitarianism. I'd be more than happy to fall back on other types of consequentialism, or moral uncertainty, if necessary (in fact I lean much more towards these than utilitarianism in general). I'm defending it simply because I don't think that the criticisms being raised are valid for most forms of utilitarianism. See my comments below for more detail on that.

That being said, I do think it's perfectly reasonable to want a coherent ethical theory that can be used universally. Indeed the alternative is generally considered irrational and can lead to various reductios.

Hmm perhaps. I did try to address your points quite directly in my last comment though (e.g. by arguing that EV can be both a decision procedure and a rightness criterion). Could you please explain how I'm talking past you?