What To Watch At The End

What to Watch at the End

I've been happy to run in to a couple pieces of media back-to-back over the last week or so- plenty of down time, since I have that bug that's going around. They make pretty interesting companion pieces to one another, actually. With the end of the world so close now, we're starting to get a bit more genuinely thoughtful art about the subject, stuff you really can't say until you have this kind of vantage point.

They are The Power Fantasy (written by Kieron Gillen), an early-days ongoing comic of the 'deconstructing superheroes' type, and Pantheon (created by Craig Silverstein), one of those direct-to-streaming shows that get no marketing and inevitably fade away quickly; this one's an adult cartoon with two eight-episode seasons, adapted from some Ken Liu short stories, with a complete and satisfying ending. I'll put in a cut from here; targeted spoilers won't occur, but I'll be talking about theme and subject matter as well as a few specific plot beats, so you won't be entirely fresh if you read on.

Pantheon is a solid, if wobbly, stab at singularity fiction, with more of a focus on uploaded intelligence than purely synthetic (though both come in to play). It's about two-thirds YA to start, declining to about one-fifth by the end. The Power Fantasy, by contrast, is an examination of superpowers through a geopolitical lens that compares them to nuclear states; I'm not as good a judge of comics over all (particularly unfinished comics), but this one seems very high quality to me.

The intersection of the Venn Diagram of these two shows is the problem of power, and in particular the challenges of a human race handing off the baton to the entities that supersede it. They're both willing to radically change the world in response to the emergence of new forces; none of them even try to 'add up to normal' or preserve the global status quo. Both reckon with megadeath events.

I'm a... fairly specific mix of values and ethical stances, so I'm well used to seeing (and enjoying!) art and media that advance moral conclusions I don't agree with on a deep level. I used to joke that Big Hero Six was the only big-budget movie of its decade that actually captured some of my real values without compromise. (I don't think it's quite that bad, actually, I was being dramatic, but it's pretty close.)

Pantheon was a really interesting watch before I figured out what it was doing, because it felt like it was constantly dancing on the edge of either being one of those rare stories, or of utterly countermanding it with annoying pablum. It wasn't really until the second or third episode that I figured out why- it's a Socratic dialogue, a narrative producing a kind of dialectical Singularity.

The show maintains a complex array of philosophies and points of view, and makes sure that all of them get about as fair a shake as it can. This means, if you're me, then certain characters are going to confidently assert some really annoying pro-death claims and even conspire to kill uploaded loved ones for transparently bad reasons. If you're not me, you'll find someone just as annoying from another direction, I'm sure of it. Everybody has an ally in this show, and everybody has an enemy, and every point of view both causes and solves critical problems for the world.

For example, the thing simply does not decide whether an uploaded person is 'the same as' the original or a copy without the original essence; when one man is uploaded, his daughter continues thinking of him as her dad, and his wife declares herself widowed, and both choices are given gravitas and dignity. He, himself, isn't sure.

This isn't something you see in fiction hardly at all- the last time I can think of was Terra Ignota, though this show lacks that story's gem-cut perfection. It's that beautiful kind of art where almost nobody is evil, and almost everything is broken. And something a little bit magical happens when you do this, even imperfectly, because the resulting narrative doesn't live in any single one of their moral universes; it emerges from all of them, complexly and much weirder than a single simplistic point of view would have it. And they have to commit to the bit, because the importance of dialogue is the core, actual theme and moral center of this show.

The part of rationalism I've always been least comfortable with has been its monomania, the desire to sculpt one perfect system and then subject all of reality to it. This becomes doomerism very quickly; in short order, rationalists notice 'out of the crooked timber of humanity, no straight thing was ever made', and then conclude that we're all very definitely going to die, once the singleton infinite-power system takes over, because it too will be flawed. (e.g. this joking-not-joking post by Big Yud.)

And don't get me wrong, I do take that concern seriously. I don't think I can conclusively, definitely convince myself that rationalism is wrong on this point, not to a degree of confidence that lets me ignore that risk. I don't at all begrudge the people devoting their entire professional lives to avoiding that outcome, even though I don't take it as given or even as particularly likely myself.

But it is precisely that monomania that is the central villain of this show, if it even has one. Breakdowns in dialogue, the assertion of unilateral control, conquering the world for its own good. The future, this show says, is multipolar, and we get there together or not at all.

That's a tremendously beautiful message, and a tremendously important one. I do wish it was more convincing.

The Power Fantasy works, quite hard, to build believably compassionate personalities into the fabric of its narrative. It doesn't take easy ways out, it doesn't give destroy-the-world levels of power to madmen or fools. Much like Pantheon, it gives voice to multiple, considered, and profoundly beautiful philosophies of life. Its protagonists have (sometimes quite serious) flaws, but only in the sense that some of the best among us have flaws; one of them is, more or less literally, an angel.

And that's why the slow, grinding story of slow, grinding doom is so effective and so powerful.

In a way that Pantheon does not, TPF reckons with the actual, specific analysis of escalation towards total destruction. Instead of elevating dialogue to the level of the sacred, it explores the actual limits and tendencies of that dialogue. It shows, again and again, how those good-faith negotiations are simply and tragically not quite good enough, with every new development dragging the world just an inch closer to the brink, making peace just a little bit more impossible. Those compassionate, wise superpowers are trapped in a nightmare that's slowly constricting around them, and they're compassionate and wise enough to know exactly what that means while remaining entirely unable to stop it.

It's most directly and obviously telling a story about the cold war, of course, not about artificial intelligence per se. The 'atomics' of TPF are just X-Men with the serial numbers filed off, and are therefore not constructed artifacts the way that uploaded and synthetic minds are; there's some nod to an 'superpowers arms race' in the AI sense of the term, but it's not a core theme. But these are still 'more than human' in important ways, with several of the characters qualifying directly as superintelligences in one way or another.

The story isn't complete (just getting started, really), so I don't want to speak too authoritatively about its theme or conclusions. But it's safe to say that the moral universe it lives in isn't a comfortable one. Echoing rationalists, the comic opens with an arresting line of dialogue: "The ethical thing to do, of course, would be to conquer the world."

In his excellent book Superintelligence, Nick Bostrom discusses multipolarity somewhat, and takes a rather dim view of it. He sees no hope for good outcomes that way, and argues that it will likely be extremely unstable. In other words, it has the ability to cloud the math, for a little while but it's ultimately just a transitional phase before we reach some kind of universal subordination to a single system.

The Power Fantasy describes such a situation, where six well-intentioned individuals are trying to share the world with one another, and shows beat-by-beat how they fail.

Pantheon cheats outrageously to make its optimism work- close relationships between just the right people, shackles on the superintelligences in just the right degree, lucky breaks at just the right time. It also has, I think, a rather more vague understanding of the principles at play (though it's delightfully faithful to the nerd culture in other ways; there's constant nods to Lain and Ghost in the Shell, including some genuinely funny sight gags, and I'm pretty sure one of the hacker characters is literally using the same brand of mouse as me).

TPF doesn't always show its work, lots of the story is told in fragments through flashbacks and nonlinear fragments. But what it shows, it shows precisely and without compromise or vagueness. It does what it can to stake you to the wall with iron spikes, no wiggle room, no flexibility.

But all the same, there's an odd problem, right? We survived the Cold War.

TPF would argue (I suspect) that we survived because the system collapsed to a singleton- the United States emerged as the sole superpower, with the Pax Americana reigning over the world undisputed for much of the last forty years. There were only two rivals, not six, and when one went, the game functionally ended.

In other words, to have a future, we need a Sovereign.

So let me go further back- the conspicuous tendency of biospheres to involve complex ecosystems with no 'dominant' organism. Sure, certain adaptations radiate quickly outward; sometimes killing and displacing much of what came before. But nature simply gives us no prior record of successful singletons emerging from competitive and dynamic environments, ever. Not even humans, not even if you count our collective species as one individual; we're making progress, but Malaria and other such diseases still prey on us, outside our control for now.

TPF would argue, I suspect, that there's a degree of power at which this stops being true- the power to annihilate the world outright, which has not yet been achieved but will be soon.

But that, I think, has not yet been shown to my satisfaction.

Obligate singleton outcomes are a far, far more novel claim than their proponents traditionally accept, and I think the burden of proof must be much higher than simply having a good argument for why it ought to be true. A model isn't enough; models are useful, not true. I'm hungry for evidence, and fictional evidence doesn't count.

It's an interesting problem, even with the consequences looming so profoundly across our collective horizon right now. TPF feels correct-as-in-precise, the way that economists and game theorists are precise. But economics and game theory are not inductive sciences; they are models, theories, arguments, deductions. They're not observations, and not to be trusted as empirical observations are trusted. Pantheon asserts again and again the power of dialogue and communication, trusts the multipolar world. And that's where my moral and analytical instincts lie too, at least to some degree. I concern myself with deep time, and deep time is endlessly, beautifully plural. But Pantheon doesn't have the rigor to back that up- this is hope, not deduction, and quite reckless in its way. Trying to implement dialectical approaches in anything like a formal system has led to colossal tragedy, again and again.

One narrative is ruthlessly rigorous and logically potent, but persistently unable to account for the real world as I've seen it. The other is vague, imprecise, overconfident, and utterly beautiful, and feels in a deep way like a continuation of the reality that I find all around me- but only feels. Both are challenging, in their way.

It's a bit scary, to be this uncertain about something this consequential. This is a question around which so much pivots- the answer to the Drake paradox, the nature of the world-to-come, the permanence of death. But I simply don't know.