Thanks! I look forward to reading about the space governance module :)

I expect we'll be targeting and collaborating with space law and policy people on a lot of the research,^[1] as well as talking to decision-makers in space companies, governments, and space-related NGOs. This means it would be better to have a separate research stream because:

• Our audience will be new to AI but familiar with space (which is the opposite for Forethought), so the research outputs have to explain different concepts.
• The assumptions that the space policy community have are different to Forethought's audience (e.g. we would have to do more work to justify why we care about future people in space and why we think AI is super important). We'd need to argue different points.
• We would want to build our own reputation and relationships with other space governance organisations, space companies, and governments. We may also start a fellowship program.
• We will want more freedom over our strategic direction as we engage more with the space domain on ideas related to transformative AI and the long-term future. 

I think we'd also like to collaborate with AI governance orgs where it makes sense (e.g. on AI governance related to orbital data centres), and we'd still be talking to AI people (the biggest space company is also an AI company). Acting as a bridge from the AI community to the space community could be great (it's not clear to me that Forethought could or should be that).

I'd personally like to continue collaborating with Forethought on topics where there's overlap, and I don't think the founding of this org affects Forethought's overall strategic direction.^[2] They're supporting us in building capacity in a community that they otherwise wouldn't be able to reach easily.

1. ^{^}

   We would publish things that would not be of great interest to Forethought's typical audience, which I think is mostly EAs and people involved in frontier AI, e.g.

   • Forecasting of the impacts of advanced AI on space activities.
   • Policy proposals to be implemented by the space governance community.
   • Reflections on current events in the space community.
2. ^{^}

   We've recently come to the end of a 6-month research sprint organised by Forethought that was focused on space governance. So I expect them to do less space governance research as the org spins out. 

Chase Hamilton and I are spinning a longtermist space governance organization out of Forethought, and we're looking for people to join us.

We'll be doing field-building to influence and mature the existing field of space policy to more effectively address longtermist issues, and technical and policy research on orbital data centres, military activity beyond Earth orbit, AI integration of space surveillance data, and other work at the intersection of transformative AI, the industrial explosion, extreme power concentration, viatopia, and the long-term future. Happy to share internal docs on this stuff, please DM me. 

If you might be a good fit to help set up an org, want to do research on space governance, or just have strong opinions we should hear, we'd love to chat: [book a meeting with us here]. Or feel free to email me at [email protected] with any questions or introductions.

I think that challenges from misrepresentation and lying might be understated - the truthfulness of the AIs is a structural issue for adopting AI delegates in the early stages. 

There's a potential asymmetry where adopting the defense-favoured coordination tech might actually disadvantage you. With AI delegates, they would presumably be verifiable and would be programmed to tell the truth and keep to deals, but humans could still lie (even if they do so by changing their mind after the interaction with the AI delegate). So if one person adopts the AI delegate and another doesn't, then the human can overexaggerate their preferences, withhold information, and even defect on the deal (without blatantly lying), but a verifiable AI delegate presumably wouldn't be able to do that? So, humans without AI delegates might be advantaged. 

Also, I don't think that many humans do seek a fair deal - they seek a deal that benefits themself more than the other person. I think this, and the issue with AI delegates being truthful, either leads to a slow adoption of AI delegates, or maybe motivations to manipulate the AI delegates to act in deceptively or manipulatively.

The equilibriums are like: 
1. Everyone adopts AI delegates 
2. No-one adopts AI delegates 
3. AI delegates become corrupted to act in ways that might not be defined as defense-favoured 

I don't know how society gets through the transitionary period where AI delegates start getting adopted.

So helpful and clear! 

Well put. I like to think of the digital world and outer space as the big multipliers of what's possible for the extent of sentient experience. Having control over what can scale in these two worlds seems essential to achieving the best futures and avoiding the worst. 

Thanks Tom, yeah the threat model for stage two is quite similar to your post, where I'm expecting one actor to potentially outgrow the rest of the world by grabbing space resources. However, I do think there might be dynamics in space that feed into a first mover advantage, like Fin's recent post about shutting off space access to other actors, or some way to get to resources first and defend them (not sure about this yet), or just initiating an industrial explosion in space before anyone else (which maybe pays off in the long-term because Earth eventually reaches a limit or slows down in growth compared to Dyson swarm construction). 

As for the threat model of stage 1, I don't have strong opinions on whether a decisive strategic advantage on Earth is more likely to be achieved with superexponential growth or conflict, though your post is very compelling in favour of the former.

My current guess is that there are so many orders of magnitude for growth on Earth that super-exp growth would lead to a decisive strategic advantage without even going to space. If that's right (which it might not be), then it's unclear that stage two adds that much. 

I'm thinking about this sort of thing at the moment in terms of ~what percentage of worlds a decisive strategic advantage is achieved on Earth vs in space, which informs how important space governance work is. I find the 3 stages of competition model to be useful to figure that out. It's not clear to me that Earth obviously dominates and I am open to stage 2 actually not mattering very much, but I want to map out strategies here. 

I do already think that stage 3 doesn't matter very much, but I include it as a stage because I may be in a minority view in believing this, e.g. Will and Fin imply that races to other star systems are important in "Preparing for an Intelligence Explosion", which I think is an opinion based on works by Anders Sandberg and Toby Ord. 

Super sceptical probably very highly intractable thought that I haven't done any research on: There seem to be a lot of reasons to think we might be living in a simulation besides just Nick Bostrom's simulation argument, like:

• All the fundamental constants and properties of the universe are perfectly suited to the emergence of sentient life. This could be explained by the Anthropic principle, or it could be explained by us living in a simulation that has been designed for us.
• The Fermi Paradox: there don't seem to be any other civilizations in the observable universe. There are many explanations for the Fermi Paradox, but one additional explanation might be that whoever is simulating the universe created it for us, or they don't care about other civilizations, so haven't simulated them.
• We seem to be really early on in human history. Only about 60 billion people have ever lived IIRC but we expect many trillions to live in the future. This can be explained by the Doomsday argument - that in fact we are in the time in human history where most people will live because we will soon go extinct. However, this phenomenon can also be explained by us living in a simulation - see next point.
• Not only are we really early, but we seem to be living at a pivotal moment in human history that is super interesting. We are about to create intelligence greater than ourselves, expand into space, or probably all die. Like if any time in history were to be simulated, I think there's a high likelihood it would be now. 

If I was pushed into a corner, I might say the probability we are living in a simulation is like 60%, where most evidence seems to point towards us being in a simulation. However, the doubt comes from the high probability that I'm just thinking about this all wrong - like, of course I can come up with a motivation for a simulation to explain any feature of the universe... it would be hard to find something that doesn't line up with an explanation that the simulators just being interested in that particular thing. But in any case, that's still a really high probability of everyone I love potentially not being sentient or even real (fingers crossed we're all in the simulation together). Also, being in a simulation would change our fundamental assumptions about the universe and life, and it be really weird if that had no impact on moral decision-making. 

But everyone I talk to seems to have a relaxed approach to it, like it's impossible to make any progress on this and that it couldn't possibly be decision-relevant. But really, how many people have worked on figuring it out with a longtermist or EA-mindset? Some reasons it might be decision-relevant:

• We may be able to infer from the nature of the universe and the natural problems ahead of us what the simulators are looking to understand or gain from the simulation (or at least we might attach percentage likelihoods to different goals). Maybe there are good arguments to aim to please the simulators, or not. Maybe we end the simulation if there are end-conditions?
• Being in a simulation gives some weight to the probability that aliens exist (they probably have a lower probability of existing if we are in a simulation), which helps with long-term grand planning. Like, we wouldn't need to worry about integrating defenses against alien attacks or engaging in acausal trade with aliens.
• We can disregard arguments like The Doomsday Argument, lowering our p(doom)

Some questions I'd ask is: 

• How much effort have we put into figuring out if there is something decision-relevant to do about this from a moral impact perspective? How much effort should we put into this?
• How much effort has gone into figuring out if we are, in fact, in a simulation, using empiricism? What might we expect to see in a simulated universe vs a real world? How we can we search for and detect that? 

Overall, this does sounds nuts to me and it probably shouldn't go further than this quick take, but I do feel like there could be something here, and it's probably worth a bit more attention than I think it has gotten (like 1 person doing a proper research project on it at least). Lots of other stuff sounded crazy but now has significant work and (arguably) great progress, like trying to help people billions of years in the future, working on problems associated with digital sentience, and addressing wild animal welfare. There could be something here and I'd be interested in hearing thoughts (especially a good counterargument to working on this so I don't have to think about it anymore) or learning about past efforts. 

Yeah, agreed on that point. Folks at Forethought aren't necessarily thinking about what a near-optimal future should look like, they're thinking about how to get civilisation to a point where we can make the best possible decisions about what to do with the long-term future. 

Yeah, lists exist for all the people working on space governance from a longtermist perspective, and they tend to list about 10-15 people. I'm like 90% sure I know of everyone working on longtermist space governance, and I'd estimate that there are the equivalent of ~3 people working full time on this. There's not as much undercover work required for space governance, but I don't like to share lists of names publicly without permission.

At the moment, the main hub for space governance is Forethought and most people contact Fin Moorhouse to learn more about space governance as he's the author of the 80K problem profile on space governance and has been publishing work with Forethought on or related to space governance. From there, people tend to get a lay of the land, introductions are made, and newcomers will get a good idea of what people are working on and where they might be able to contribute.