Hello! Inspired by the neolab, I decided to read “The Metamorphosis of Prime Intellect” by Roger WIlliams. Two things to start: there are SPOILERS AHEAD, and I would also like to repeat the author’s disclaimer from the book:

This online novel contains strong language and extreme depictions of acts of sex and violence. Readers who are sensitive to such things should exercise discretion.

This post will be family-friendly, but if you decide to read the original source, you should keep that in mind. In addition, I found some good suggestions on what parts to skip here (spoilers there), though I did not find this until after reading because I wanted to go in clean, and I’m not traumatized.

Anyways! To the meat:

Spoilers inside of here for ‘The Metamorphosis of Prime Intellect’

One spoiler (which shouldn’t be that surprising) is that there exists an artificial, extremely powerful, superintelligent entity in the book which I will refer to as John. I think overall it provided an interesting perspective on one potential failure mode of a system like John, which in the book is lack of meaning. I know this is a pretty common concern, but I think the book does a great job of making it very visceral. In particular, it paints a very good picture of how Asimov’s Laws of robotics can go wrong:

Asimov’s Laws of robotics, from “Runaround”

  1. Protect human safety,
  2. Follow human orders, unless the conflict with the first law,
  3. Protect the self, unless it conflicts with the first and second law. Because there’s been like three spoiler warnings, the specific failure mode it gives is that after coming into power, many humans begin taking on forms that are not usually associated with humans, such as becoming animals or zombies or even just becoming incorporeal. This has a bunch of consequences in the book, but I think this is a really good point: if a human order can make a human not human, thereby exempting the human from the first law, that would be putting it in danger of dying from the robot, therefore a human can never become not a human. Is this actually true? Maybe not.
    Another interesting specific detail from the book was that many humans just isolated themselves completely from other humans, because John provided everything for them and could even create “fake” humans for them to interact with.
    This is where I’m going to stop because it’s late and I want to go to sleep. One piece of future reading I want to embark on is one in which the AI is a dictator (as opposed to MOPI where it’s sort of a benevolent, all-powerful fairy), and according to Claude the best in this genre is Colossus by D. F. Jones.

This has been stated in the spoiler box, but it doesn’t necessarily have spoilers: I will be reading Colossus by D. F. Jones soon.