This is what a lot of people who worry about AI would call "technofeudalism", where society is controlled by the owners of AI, and the rest of humanity becomes a permanent underclass that only exists at their whims. It -is- a scenario that could, in theory, happen. It would mean though that governments/revolutionaries did not step in to limit the power of the AI owners, provide basic income, and/or implement socialism. It would also mean that the AI alignment project was successful, but in a way that aligns with the parochial interests of the owners, rather than with human values/morality in general. It would also likely mean values lock-in, and is a possible dystopian outcome.

There are a few alternative scenarios, like oft-imagined utopian outcome of "luxury space communism" stemming from a post-scarcity economy that is fairly redistributed, or perhaps more realistically (but rarely mentioned), the AI owners living in a cloistered society of abundance apart from everyone else, who boycott AI and run a parallel "organic" economy that looks similar to what currently exists.

There are also, of course, the scenarios where the AIs disempower humanity, including their owners. Note, that this can happen even with "successful" alignment (i.e., gradual disempowerment), so it's not just extinction risk. We could, again, in theory, end up as defacto pets to some benevolent superintelligences who make all the actual decisions.

The reality is that no one knows what will actually happen, and all these speculations depend on the key assumption that AI can function as capital replacing labour.

My impression of MAGA is from a mixture of talking with my mom at length about what she believes, Facebook convos with friends who are MAGA-sympathetic, reading things like Project 2025 to try to understand what they actually want and believe, reading randos who claim to be MAGA on Twitter as they dogpile and generally act hostile to anything resembling reasonable discourse, and yes mainstream news coverage of things like how Trump is once again threatening to annex my country (Canada).

I will admit it is a mixed bag of sources. If you could provide other sources that show what MAGA supporters actually value and care about to dispell my apparent misunderstandings, sure, I'll look at them too, for the sake of steelmanning and fairness.

Well, my mom is a devout fundamentalist Christian who supports Trump, so she could technically be considered MAGA-adjacent (even though we are Canadian). We stopped talking about politics a long time ago because we disagree so completely.

I'm talking about the extremes. There are many more people who have supported Trump in the election than who are full-throated MAGA. I'm not talking about the former. I'm talking about the latter, the people who marched on January 6th and are loud and obnoxious on Twitter and who drown out any reasonable voices that might exist in the movement.

To be fair, MAGA is basically the antithesis of all that Effective Altruism stands for, so it would not be surprising that we would be natural enemies and this would eventually happen.

MAGA is about contracting the moral circle of concern to "real Americans". This is literally the opposite direction as EAs wanting to expand the moral circle to animals. MAGA is flagrantly anti-intellectual. A stereotypical example of a MAGA is a gun-loving hunter who shoots animals for sport and brags about being a red-meat eating tough guy.

Like, the socialists disagree with us about the methods (we advocate individual action like charity and career change, while they advocate systemic change) but ultimately we share the ends of equality and wanting a better future. We compete in terms of recruiting for the idealistic altruists in the world. We both tend towards internationalism and some form of the common good for all.

MAGA doesn't share any of that. They harken to past national glory, see the people we donate bednets to as subhuman, see the elites we listen to as inherently untrustworthy, denigrate empathy and glorify egoism, etc. I just don't see why we should expect anything else from them than pure disdain.

In theory, AI x-risk is more important than all of that, and we do have a single common cause in not wanting to be dead, but I see little else we can even attempt to agree on.

This is easier said than done. EA has these problems in part because of our associations with the whole "TESCREAL" lumping together of various Silicon Valley centric ideologies. We're closely tied to the Rationalists of Less Wrong for various reasons (I once met Yudkowsky at an EA Global conference), and many of our so-called thought leaders like Bostrom and MacAskill are Oxford philosophers with a particular nexus of views that tend to be somewhat controversial right now.

We also, as a very open-minded, truth-seeking community, tend to, in my humble opinion, overly humour ideas that are contrarian and distinctly unpopular. The merits of Eugenics for instance, were debated here several times, though mostly disapproved of.

EA in general tends to make assumptions that are a subset of the umbrella of liberal philosophy, not necessarily progressive or socialist philosophy. We've historically tended to downplay systemic change (i.e. anti-capitalism, government intervention, etc.), in favour of individual actions like donating to charities and career change. This puts us in the crosshairs of criticism from the more left-wing progressives and socialists. The Guardian is, known to have become more progressive lately, and Naomi Klein is from a family with connections to the social democratic NDP in Canada.

This is not to say there aren't EAs who are sympathetic to more left-wing considerations. Many of the rank and file are, as seen in the EA Surveys. But many of our thought leaders are known to be more centrist, and even occasionally right-wing. Peter Thiel despises us now, but did give a talk at an EA conference, as did Elon Musk, way back in the day (circa 2013-2015).

EA has a complicated history, and unfortunately, a lot of it is controversial stuff that gets dredged up everytime someone wants to come criticize us, fairly or not.

Within EA there is definitely a diversity of views, and you're probably more familiar with the Global Health and Animal Welfare "faction" that does obviously good uncontroversial stuff, but we also have the AI Safety and Longtermism "faction" that gets strongly associated with Silicon Valley. I say faction, but there is obviously overlap and not clear boundaries, so much as different emphases.

So, it's hard. I've been around since the early days, and I really, really, dislike how the media mostly focuses on the controversy and ignores the good that we do. But I don't really see a way to solve this. Independent western journalism is hypercriticial of pretty much everything remotely political, and efforts at PR and optics by CEA tend to grate against the pure truth-seeking mindset of many EAs.

Your solution seems to be to "move EA into the centre of the Overton Window", and get better exemplars to represent EA. I'm not sure this is really possible. For the first, it'll strike many EAs as less than truth-seeking, to try to hue to arbitrary public opinion, and risks removing the differences between EA and generic progressivism or centrism or whatever you think fits the Overton Window (in which case, why bother becoming an EA?). The Overton Window may not even be a useful concept anymore in our highly polarized discourse.

As for exemplars, Peter Singer is actually somewhat controversial for his old position on infanticide, and the only famous celebrity I know of who is EA sympathetic is Joseph Gordan-Levitt, who recently had a bit of a scandal over being an invitee to Peter Thiel's Dialog secret society thing. Hank Green has also sometimes engaged with EA-like ideas, but has been critical of the kind of hyper-calculating, efficiency focused mindset we often use. There are a few vegan celebrities like Billie Eilish, but I don't know that they are on-board with EA generally.

So, about optics generally, it's not like people haven't tried. A few years back there were positive, sympathetic articles about EA, including this one about MacAskill. Vox Media also had its Future Perfect coverage of EA, that was pretty sympathetic.

Also, might as well mention that if you really want to dig into EA stuff, there's the simple fact that the by far the largest donor to EA is billionaire Dustin Moskovitz, through Coefficient Giving (formerly Open Philanthropy). Admittedly, Moskovitz is surprisingly progressive for a billionaire, and was one of the largest donors to the Democrats in the last election.

Anyways, hope that helps to show how complicated the situation is.

That is sorta the idea yes. Agents would choose this decision criteria mostly because it vastly increases their odds of survival, which allows them to further whatever goals they have. I would hope that this result is obvious enough that many agents will be able to converge on it, increasing the proportion using it, and thus increasing the overall survival rate.

The other takeaway is that, given that humans will be weaker than AGI/ASI, any game theoretic reason for such entities to still cooperate with us can potentially help reduce the existential risk.

I agree that the model requires further scrutiny to determine if it is realistic enough to matter.

I have this game theory thing I've been working on that involves modifying the Iterated Prisoner's Dilemma to include death, asymmetric power, and aggressor reputation. Agents' points are their "power" that dynamically impacts their payoff matrix. 

The basic takeaway is that this simple simulation seems to make a case for cooperating with weaker agents, by showing how the cooperative strategies outcompete the aggressive ones in the long run. I think, before I can make a proper post about it, I'll need to run some analysis to graph out how, for instance, having a higher percentage of cooperative agents increases the odds of survival, which implies a kind of Veil of Ignorance logic towards being cooperative.

Note that I mean cooperative in the sense that you don't defect first except against aggressors that have defected first against non-aggressors.

With default settings, the most common result of any given run is that a significant number of the cooperative strategies survive and almost all of the aggressive ones die out. Very occasionally, particularly if you adjust the settings are certain way, a single "Opportunist" strategy, that Tit-For-Tats against stronger agents and Defects against weaker ones, will be the only survivor. This seems to imply, at least, to me, that being a cooperative strategy significantly increases your odds of survival, as the alternative is to hope to win a "Highlander" scenario.

I think this is relevant to AI alignment as a variation on Anthropic Capture, the "Hail Mary" approach that Bostrom mentions in Superintelligence. It could work as part of a defence-in-depth, a kind of "infoblessing" that could persuade some AGI to spare us as a kind of Superrational Signalling. While you might assume this only works if aliens are probable, it also functions in a multi-agent scenario where there are several AGI at near peer levels of power. It also potentially could be a way to align a previously unaligned AGI even after it is deployed. If enough AGIs are aligned in this way, their alliance could defeat the unaligned AGIs.

You can run the simulation yourself here: https://paxscientia.com/power/

I have the code and initial analysis here: https://github.com/josephius/power

I realize that a very obvious critique of this work is that the simulation is probably too simple. I intentionally tried to keep it an MVP in its first iteration. I also should, as mentioned earlier, complete a more thorough and rigorous analysis of the apparent results. I'm also keenly aware that it seems like this is a "neglected" path towards alignment, and I'm uncertain whether this is because the idea is a bad one that's already been discarded by others who are more competent. I know that there are related ideas around Decision Theory, Acausal Trade, and Superrationality, but I've never seen this particular kind of effort, which confuses me, because it seems obvious and trivial to try.

My main question to ask is simply, does this seem like something worth pursuing and expanding further, or am I wasting my time on a foolish endeavour?

Yeah, getting something to be both meaningful and fun at the same time is hard. I took a quick look at your prototype. It could have some potential, but at the same time, it's not the only game out there with a similar idea. I recently came across The Choice Before Us, which is in a similar vein.

Given how fast things are moving in terms of AI developments, I'm not sure it's realistic to try to make a game that's polished enough to go viral before things change enough that the game is essentially obsolete.

Also, games are hard. Creative work seems very lottery-like in terms of success. Maybe you can argue from a risk neutral EV perspective that it's worth it, but that doesn't pay the bills.