The Intelligence Curse w/ Evan Miyazono

I spoke to Evan Miyazono, founder of Atlas Computing, about the neglected risks of advanced AI and what it would actually take to govern it.

We dig into the threats he thinks aren't getting enough attention, from asymmetrically offensive cyber capabilities and economic disruption to what he calls the "intelligence curse," a dynamic where governments lose any incentive to invest in their populations once labor becomes synthetic. We also get into formal verification as a framework for AI governance, why consensus and agreement become scarcer as intelligence gets cheaper, and where blockchains might actually fit into that picture in ways that have nothing to do with libertarian fantasy.

This episode is sponsored by NYM, the world's most private VPN. Unlike traditional VPNs, Nym uses a decentralized mixnet to scramble your internet data — hiding who you're talking to, when, and how often. You can switch between full mixnet mode for maximum anonymity, or a faster VPN mode for everyday use.

Use the code blockchainsocialist when signing up and get an extra month!

If you liked the podcast be sure to give it a review on your preferred podcast platform. If you find content like this important consider donating to my Patreon starting at just $3 per month. It takes quite a lot of my time and resources so any amount helps. Follow me on Twitter (@TBSocialist) or Mastodon (@theblockchainsocialist@social.coop) and join the r/CryptoLeftists subreddit. 

Transcript

SPEAKER_01 0:00

It is incredibly valuable to have a publicly accessible, decentralized, append-only ledger that has like strong economic security properties. The only problem is it's far more useful for speculation and fraud than a lot of other things. I think that just being open to like, do you think that humanity might not make it? And if the answer is, yeah, I'm willing to entertain that humanity might not make it, and then you look at the probabilities, the types of scenarios, civilization starts looking really fragile. Yeah, I'm sitting on a bunch of zero days, some of the Linux curl, because I haven't vetted them. And like that seems asymmetrically offensive if that capability shows up in the wild. I do think that unemployment is going to spike at some point, because I think that in the next few years it feels pretty likely that we'll get something that's like a drop-in knowledge worker. I think that as intelligence gets cheap, that agreement or consensus gets very expensive.

SPEAKER_00 0:49

This episode is sponsored by NIM, the world's most private VPN that protects your internet traffic and metadata. Unlike traditional VPNs, NIM uses a decentralized mixnet to scramble your internet data, hiding who you're talking to, when and how often. You can switch between full Mixnet mode for maximum anonymity or a faster VPN mode for everyday use. Pay in crypto or fiat, and even your payment stays anonymous thanks to ZK-powered anonymous credentials. Take back control of your online life at NIM.com. Sign up today using the code BlockchainSocialist and get an extra month for free. Hi everyone, you're listening to the Blockchain Socialist podcast, where I am here in my apartment getting lasered by the sun. So you may see some of the light bouncing off of my face right now. But today I'm here with Evan Miazzono. He is the founder of Atlas Computing, which is a super interesting organization that I came across recently trying to solve some of the like really big problems around AI that I feel like a lot of people haven't yet even like touched or began to think about, I think, which is what has been really cool reading into some of the work that they're doing, the things that they're proposing, and like how they're going about these problems, I think is really interesting. So I asked Evan to come on and share some of the stuff that they're doing at Atlas Computing. So yeah, you can just dive right into it. Evan, if you want to give maybe just a quick introduction to yourself and what is Atlas Computing and the problem that you guys are trying to solve.

SPEAKER_01 2:23

Yeah, absolutely. Thanks for having me, Josh. Into a bit of who I am, where I came from, any blockchain connection, because Atlas doesn't really have much of a blockchain connection. I shortly after finishing experimental physics PhD, I went looking for hardware startups to join, and a college buddy of mine convinced me to join his peer-to-peer file sharing startup in 2017 for an unconventional fundraise, and that was the Filecoin token free sale. I imagine that people here might be familiar with Filecoin, IPFS, libpedp. They may or may not know that those are all from the same company, Protocol Labs. I've had I've had the delight of surprising people that yes, those are all in fact one company. Protocol Labs did a lot of various things, and I basically built out the research and meta science infrastructure. So this was everything from like um I think I interviewed basically every researcher who joined the company for a period of six years. I like set up many of those interview flows, set up the grants program, the research grants, set up our advisorships, sponsorships, figured out how we like leveled and promoted researchers, also ran a meta science team, also ran a special projects team for a good long while, also ran a, for the last two years there, a venture studio focused on mechanisms for improving public goods funding. The idea was that Protocol Labs was going to spin out all of its various teams and products into different companies. Some of those would be creating public goods. How do you properly incentivize the creation of valuable open source software? There's a lot of really interesting questions around governance and like social choice theory related to that, because what is the value of this public good is something that is entirely subjective. You would quantify it. In fact, most things that are quantified are in fact subjective. But a lot of interesting questions there. I like helped out on the file coin white paper and the economics of Filecoin white paper. A lot of like I like to joke that I did some experimental macroeconomics for a bit. About three years ago, I left to start uh Atlas computing, mostly because one of the researchers on my independent research team, David Dalmrymple or David, was working on a mechanism to or a approach to quantifiable safety properties for AI systems. And this seemed very promising. It seemed like there was a lot of work to be done. We decided he should go to ARIA and run this as a program there, and I should go start a nonprofit to do all the things that would be useful for that c for that approach that couldn't be done effectively as a government. And over the course of the last three-ish years, I've landed on Atlas being kind of something that fits between the think tank and the incubators, where the think tank will have a really good proposal for how like understanding of this problem and a proposal for how to solve it. And we'll have a lot of relationships with people who can speak to the problem, people who would you be users or stakeholders on a solution. And the incubator assumes that a founder has shown up with the right solution. And so I'm taking basically all of these gaps that experts have said, like, yeah, it's this is this could be a problem, and no one is working on it. And I have been doing some amount of work to it make it easier for someone to take that problem. I like to quote that between the Venture Studio Protocol Labs and Atlas Computing, I think there are about seven founders, many of whom were not uh repeat founders, where I introduce them to the problem and the potential solution and their first funder. And I think that really lowers the barrier for what it means to be a founder type. I think that people who would disagree with me on that would say that all of those people are founder types because they did end up founding a thing and that are that's going well. But I've I think that maybe those people would agree that there are a lot more founder types out there than are currently empowered or clearly see the affordances to go and do the thing. So I'll I will sometimes jocularly refer to myself as either a research fixer or a professional career derailleur, where I help people find the thing they want to do. I approach people when it seems like there is a much better way to achieve goals that a person might have with their skills. And I say, you have the right skills. Would you like to be doing what you want to do, but more effectively in a new organization or a different organization or something like that?

SPEAKER_00 7:00

Right, right. Yeah. One of the things that I think really caught my eye about Atlas Computing was honestly the the job descriptions of what you guys are looking for. I mean, when I read it, it reads like a dream job type of thing for you know, if you're the right person for the thing that you want to do, it definitely sounds like a dream job. And I mean, I definitely recommend people to check out check out Atlas Computing and check out the just the job descriptions to me was really fascinating. I was like never read that type of job description before, which is like really, really interesting.

SPEAKER_01 7:29

But I can't if I can't get excited about a job by reading the job listing, that's like in many instances, the most excited someone will ever be about a job. And so like I I think job descriptions must be held to a very high bar. This is actually something I inherited from protocol labs, where like I think consistently I would write job descriptions and be like, cool. Do I want do I want this more than I want my job? And if the answer isn't a little bit, then I like would go back and rewrite it.

SPEAKER_00 7:59

Nice. But so then you are you're you're one of the one of the I mean, I've been, yeah, I've interviewed at least maybe one or at least one off the top of my head, people who have they have they have switched from from crypto to AI. So you've gone from, you know, doing a lot of the things that I think I mean me and a lot of other people I think who listen were interested in going into the crypto space, which was like trying to figure out different forms of economic relations and economic mechanisms for public goods and and related things, to now yeah, taking on some of the which which is a really big problem, but then also taking on other really big problems around AI. And yeah, I I just wanted to I just wanted to note that I thought it was really, really it was funny, interesting, but I think I think in this interview we'll be able to connect the two later down the line.

SPEAKER_01 8:46

Yeah, there are a lot of people who went from crypto to AI, and I like I think the people that listen to this podcast probably understand why I might shudder to be associated with the majority of them, even though a lot of your listeners have probably done the same thing as I have for the same reasons, which is technology just moves faster than a lot of the other aspects of society. And if you want to try to fix important problems in society, it is important to kind of like steer the fastest moving thing. And it was delightful, like I think protocol labs very well known in the like the subset of crypto who were very ideologically motivated. And I still, amongst AI people, will say it is incredibly valuable to have a publicly accessible, decentralized, append-only ledger that has like strong economic security properties. The only problem is it's far more useful for speculation and fraud than a lot of other things. And so it's really hard to actually get adoption. There was at least one like really impressive thing that like deal that Protocol Labs made, and part one of the riders in the clause was you can't use this in your advertising because we don't want to get you accused of misusing taxpayer money. Like that is not a good sign for your economic structure.

SPEAKER_00 10:08

Yeah. So I before we get into like the AI risks that you guys are tackling, I think it would be interesting to talk about why why calling it Atlas specifically.

SPEAKER_01 10:20

It will be related to Atlas Shrugged. That did come up when I was like floating it to people, and I was like, ah, I don't think I think this is just a function of who I'm talking to. I don't think this will come up that much, and it hasn't come up that much. Mostly I couldn't come up with a better term for a set of maps or something that was related to making a lot of maps. The plan for Atlas was try to do uh do a lot of the strategy work to make it easier to chart courses to better futures. I feel like technological non-determinism probably has a meaning, but that I don't mean to invoke. But this idea that like if you move, I guess differential development is differential technological development is like maybe the more appropriate term where like if you develop diff technologies in different orders and you could do this somewhat intentionally, you can lead to different outcomes. And I think that this is an important thing. It's fairly technocratic and uh like fairly rationalist coded as ideas go. It's interesting how much I feel like having spent so much time at Protocol Labs, there was like a very consistent vibe and ethos, and a lot of interesting like philosophical questions that were like I find it very helpful to reach for like the decentralized solution to this thing first, because it turns out that in the event that there could be a like singleton super intelligence, you want something that is much more robust to like power concentration or things like that. And I think having focused on finding decentralized, democratized, accessible solutions is good, but so is having the security mindset of like, okay, but it's really easy to think of something that sounds secure, but like, did you kick the tires? Did you really try to see if someone can use this as an attack?

SPEAKER_00 12:07

Right, right. I really like the metaphor or the the the association with maps because in in my book, actually, one of the quotes that I use is you know, the map is not the territory. It was one of like my one of my favorite quotes of just like, I mean, to me, I find it very, very deep, or just like a good explanation for very compact description of like a fairly complex idea of like trying to how you you know map the world in a way in that like that map itself doesn't necessarily it cannot a hundred percent reflect reality, but that's something that we do a lot with I mean, especially in the case with with digital systems. And I use the term techno probabilism as like an alternative to techno-determinism. So I thought that was uh that was funny.

SPEAKER_01 12:47

The physics analog of the map is not the territory, is all models are wrong, some models are useful, which I think.

SPEAKER_00 12:52

Well that was my set that was my second quote in my book that I used the most. I'll send you a copy afterwards.

SPEAKER_01 12:58

I I look forward to it. Thank you.

SPEAKER_00 13:01

So I think this type of like concept will come up in the conversation later as well. But what are the types of AI risks that you guys are looking at and that you think really need some attention right now that you guys are trying to bring to that maybe people just haven't really considered when it comes to AI, besides the kind of like, I don't know, images of apocalypse iRobot or these types of things?

SPEAKER_01 13:26

Yeah, I think that it's interesting how many of those view, like those those downside scenarios, get caricatured and sometimes dismissed. It is really interesting to see that, like I think that there's a much higher correlation between does someone acknowledge that these catastrophic risks are possible and did they seriously at some point consider whether or not they might be possible, rather than like how much evidence have they been presented with or something like that. I think that just being open to like, do you think that humanity might not make it? And if the answer is, yeah, I'm willing to entertain that humanity might not make it, and then you look at the probabilities, the types of scenarios, civilization starts to looking really fragile. I'm specifically looking at neglected problems where like because AI capabilities are growing, you could imagine new capabilities leading to asymmetrically offensive or asymmetrically oppressive technologies or technological equilibria. And the question becomes what else could you put into that world to make it asymmetrically defensive? That like now we have this, and as a result, it is net safer. And so an easy example would be AI-based red teaming seems like it is getting scary, good, very fast, slash already. There was a great talk from Nicholas Carlini at Unprompted like last week, I think. Um, where he talked says, like, yeah, I'm sitting on a bunch of zero days, some in the Linux kernel, because I haven't vetted them. And like that seems asymmetrically offensive if that capability shows up in the wild. That said, there are a bunch of things that you could imagine having. Like, first off, I would be very excited to have a bunch of funders decide to get a bunch of high profile cybersecurity firms on retainer to just take these and vet them themselves, because this shouldn't get blocked on Nicolas Carlini and a bunch of other people at Frontier Labs who are finding these in order to vet them, because they're doing lots of things and their job description isn't doing this. And paying someone to do a thing is a really good way to make someone pay attention to a thing. Another one would be if we had public key infrastructure so that if I am trying to red team my server, my server, like the AI that I'm using for that red teaming, knows that this server is mine. I can sign, I can I can yeah, I can generate a signature that validates to the same public key as what the server is sending out. And so you don't have to worry about if is Evan red teaming or like actually attacking a different team rather or server rather than red teaming this one. You could imagine a lot of progress in hot patch updating where you don't need to reset the system to update the code. I feel like that there are lots of like as AI gets more capable, the window between like was this discovered and was this patched needs to get much shorter because the ability to attack in that window might be much greater. And so being able to update the system trivially as soon as you know without having to reboot anything or having to interrupt anything seems like something that would be very useful. So these are the kinds of interventions that would lead to something that's more asymmetrically defensive. I think that I, when I think about risks, I ask, like, would this prevent like I want a world where AI listens to people, like follows instructions, and where like people like where there isn't something that looks like catastrophic harm. If it's not catastrophic, I'm not really that worried about it. Not because I don't care, but because I think that there are enough catastrophic harms to keep me busy and you kind of got a scope. If it's bad already and AI makes it much, much worse, I care. If it's bad already and AI makes it a little bit worse, I consider that a little bit out of scope for me. I wouldn't necessarily like rule it out if a funder came to me and said, like, hey, I really want field strategists on this, and they brought a field strategist candidate or something like that, that would be fine. At this point, I'm like trying to scale as quickly as reasonable. I think in terms of like following instructions, this is like includes like uh safety, alignment, trustworthiness, control, a lot of like model robustness, lots of terms for like very niche things in that in that category. But I also care about like economic impacts or epistemic impacts. Do you trust the information that you're getting and your ability to make decisions based on that information? Do you have like is are is the cyber physical infrastructure robust? If AI is totally aligned, but someone manages to convince it to that like the right the moral thing for it to do or the best thing for it to do at this moment is to completely destroy the electric grid. That seems like a bad thing, but there are two points at least where you can intervene here. One would be to secure the grid, one would be to make sure the model doesn't go and do things like that, lots of other interventions. I strongly believe in defense in depth, which is this notion that you should try to defend at multiple instances. The Swiss cheese effect is another thing that gets flagged here that everyone got uncomfortably familiar with during COVID. But in terms of like economic risks, I think that there are probably a lot of things that people haven't started really thinking about. I do think that unemployment is going to spike at some point because I think that in the next few years it feels pretty likely that we'll get something that's like a drop-in knowledge worker. I mean, the prices of the like AI subscriptions has been going up roughly an order of magnitude a year. And like, what would you pay like $2,000 a year for or $20,000 a year for, or sorry, per month for? $2,000 a month is like something that someone might pay an intern. Right, right. And like you should expect it to have at least that level of capability if they're going to charge for it, because I think that market is at least pretty efficient. I like to flag that another possible category of concern is like I think that if AI automates legal work, then lawyers benefit. But if it automates driving, the truck drivers don't benefit. It's the probably the consumer that benefits on that one, or um like the firm that owns the shipping companies. And so there are a lot of questions like this that I have around like what as AI gets better, what breaks? And if something breaks, then like how can you either prevent it from breaking preemptively or like shift the equilibrium into something better? Like if if unemployment spikes by like five percent in a month, then I assume that in the US there would be a UBI bill that gets written and probably passed. And I don't think it would be a very well thought-out bill because it would be passed that month, like written that month. So I'm like very interested in trying to get people to start drafting versions of that bill and talking about it, it would not solve all the problems. One of the biggest problems that I think of is like one of the people often say like dignity of labor and things like that. I worry less about that than something called the intelligence curse, which was a phrase coined by a friend of mine and his co-author. And like the the common notion is like there's a concept in economics, the resource curse. And this is that if you have if your country has a lot of natural resources, you don't invest a lot in the population because that's not where you get your tax revenue from. You invest a lot in the like extraction of these natural resources, and the you don't really have a financial incentive. The like power structures of the country itself focus on like you need to secure the supply chains. And so you get a lot of like dictators from mineral-rich countries. You might see the same thing with intelligence. If intelligence, like if you have a knowledge economy or a service economy, and you replace all of that with AI, then like you probably, as a government, don't have a lot of incentive to support the people because the people have no connection to your tax basis or things like that. I also worry about like possible, like I worry a little bit about monetary risks from inflation from UBI, but I honestly think that if we end up in that scenario, that the growth of the like service outputs probably are so big that like you can really start insanely pumping the monetary supply without losing buying power. I'm a little bit I I'm MMT curious. So I like I don't worry as much about that. It may come out at some point also that I have strong Georgist inclinations, which has impacts on other things that I think about these things. For those who haven't seen it, I would definitely recommend the book Blood in the Machine. Have you come across this one?

SPEAKER_00 22:18

Yeah, yeah. I actually bought it very recently at CCC, actually. I bought the book.

SPEAKER_01 22:25

I I enjoyed it. I think that some of the conclusions drawn at the end about limitations on AI systems are incorrect. And I think that it is probably scarier and probably would have even made a better call to action if the author had shared more of my concerns about what AI will be capable of. But in terms of the like, I was I was shocked by how everything I thought I knew about the Luddites was actually the like post-rebellion propaganda that everyone was getting. Yeah, yeah. And also the fact that like a lot of the things that people are talking about now, like attacks on automation or like responsible deployment of the technology, was like absolutely discussed back then by these like specialist guild members who were prohibited from competing in many ways.

SPEAKER_00 23:14

Right. Yeah. I think I mean a lot of different risks. And I think you've you focused on a lot of like the economic ones, I think, are interesting because I think that is something that at least for the average person, that's probably the thing that will touch them the most, or like will like get them maybe.

SPEAKER_01 23:28

If the electrical grid goes down, that'll be a problem for them. If there's another wildfire pandemic, that would probably all like I think still economic in many ways.

SPEAKER_00 23:37

It all is, I guess.

SPEAKER_01 23:38

Yeah, yeah. I I think that it was like a very interesting awakening when I realized that like law is to human actions, what software engineering is to information, um, it's just the best that we've come up with, and economics is actually the science of decision making, not the science of money. Very like, oh shit moments in my in my improvement of the my modeling the world.

SPEAKER_00 24:01

Right. I mean, so a lot of the things so I I just so happened to be watching the show Pantheon, like during this time I'm like I'm like near the end of the second season. So just it reminded me, like as I was watching that, I was like, oh wow, and then I'm like interviewing Evan about all these like AI risk. It was a great preparation for that. But yeah, like in in I think it was yeah, super interesting that that show it doesn't talk about AI, it talks about uploaded intelligence where you like you upload people's brains and stuff, which is also I think I don't I think that's like probably a little bit more sci-fi from now, and like yeah, completely maybe a different moral thing. I don't know. I'm a little bit I'm a little skeptical. I don't know. I studied neuroscience, so maybe that's why I'm like I feel skeptical about it because it didn't show up in my neuroscience courses.

SPEAKER_01 24:44

We all harbor latent suspicions around claims of progress from the fields we used to live in.

SPEAKER_00 24:49

Oh, I mean, I should probably ask you about the the recent uh Google papers.

SPEAKER_01 24:54

I don't follow it. I'm too skeptical for for not for bad reasons. My one-line take on quantum computing is that people who are very, very excited about quantum computers and aren't deeply familiar with complexity theory should probably look a little bit at like what is in BQP, what is in BPP, what might be in like other complexity classes that are like interesting and possibly relevant that feels unintuitive to me, is that like there are problems that AI systems are really good at now, like protein folding. Hamiltonian minimization is the name of the general problem that solves, and that is an NP complete problem, which means that if you have an Oracle that can do that, you can map basically any problem that is in NP to that. And you have you now have an Oracle for that other NP problem. Um, that I'm like 90% sure that everything I've said is correct, but Claude would do a better job. But yeah, like there are reasons why that why you can't use Alpha Fold as like a universal NP oracle, but like that doesn't mean that we couldn't build oracles for a lot of other things. Another thing to look at is the fact that SAT solvers, and this gets into a little bit of formal verification stuff, which we can talk about later. Um, on the formal verification side, it's uh like proving sat is also NP, like set this there's a satisfiability problem, which is like I have a bunch of ands and ors and variables together. Is there a set an assignment of like true and false to these that leads to the overarching expression being true? Um and solving this is also NP complete. And yet we have solvers that solve most of these in polynomial time. Not all of them, obviously, but like it it gets weird fast where you like the problems that we care about are often easier. The halting problem is another one of these where it's like it does this program stop. Every programmer listening to this has solved the halting problem for like a finite number, like a finite non-zero number of problems. So it is solvable in in the cases that we care about, just not in the like arbitrary general case in polynomial time. Or no, in finite, in finite time, yeah.

unknown 27:21

Okay.

SPEAKER_00 27:22

Yeah, I think yeah, I think uh for me what I find fascinating is that when you go deep enough into math, eventually it just becomes philosophy, and vice versa. I think I guess kind of.

SPEAKER_01 27:32

I also wanted to mention if you like Pantheon and haven't read QNTM's uh Lina or MM Acevedo, um, this is a very short story that it's like the other possible end of a spectrum of I love science fiction, Pantheon is amazing. I would also recommend the Jean Le Flambert ser trilogy by Hanurainimi for like similar, like imagine there are uploads. I've heard great things about the age of M. I have not read it yet. I would also strongly recommend oh man, there was another thing. Um, We Are Bob, We Are Legion is another one of these fun, like upload flavored ones. I think that a lot of a lot of fiction seems to unsurprisingly handle the notion of super intelligence really badly, and even like the TV show upload like doesn't acknowledge that like yes, if we can slow people down, we could also speed them up. Um, or at least I I haven't watched much past the like first season or so, but the like I think that it is I think there are some like concerning back of the calculations of like, yes, if you did in fact upload a person, then like someone tried to convince me that you could run, like, depending on your model of the brain, run a human in a pretty good MacBook Pro. And like the weird thing is like if that is not true now, it might be true in four years if you're originally off by like a factor of 10. So like the the the brains aren't getting more complicated. At some point, a lot of these really weird things become answerable, and a lot of questions of like, what is identity? Who is me, like the teleporter paradox gets gets weird in like sorry, gets real in weird ways. And but this goes back to what you're saying about how like a lot of the philosophy questions seem to be getting more applied, less theoretical.

SPEAKER_00 29:20

Yeah, yeah. So yeah, policy stuff. I mean, because one of the things that I found really interesting about this kind of like intersection of a AI safety, I know you prefer like a different term. I think you've said like AI control or AI security. I think the AI safety, I mean, I these are all weirded out. I mean, I am weirded out by a lot of the AI safety and rationalist people, but uh, you know, open to open to conversation, of course. But um of the risks that you talk about, you know, I think there's on one camp, maybe what's kind of like being what I've seen a small push from people like Bernie Sanders who have proposed moratoriums on building data centers, in part, I think, in some respect to maybe some people who would like really just want to turn off AI or just ban AI, just like not just like close Pandora's box if we can. Yeah, I'm curious, like what what are your thoughts on some of these policy proposals and solutions? Like, is there anything really in the policy realm that that you think would be a good idea? And what are some like what are some of the bad ideas maybe that some people are because I think like like banning AI, for example, is just something for me. Not because I can I can accept it in the premise of like, yeah, maybe we should do that, but also in the premise of just like as far as like I don't know how it gets how it happens. Like, how do you actually ban that? I mean, maybe there are ways, I just don't know.

SPEAKER_01 30:44

I love the adage that, well, I don't know where this came from. This might have come from me, which might be why I I'm particularly fond of it, but it is easier to get it probably came from Davidon. I feel like most my most clever things kind of came from Davidon. It is easier to uh get a hold of a GPU than it is your anymore, but it is harder to build TSMC than it is a centrifuge. And so like there is a there is a natural choke point if like if the US and China decided, like, hey, we think that a pause would be really useful, you could actually like you can't do this with a hundred percent guarantee, but um, I would point people to uh flexheg.com, I think is the URL. This is a proposal for a robust tamper responsive modification you could make to GPUs, where you could have something like you embed some cryptographic keys, you formally verify the entire tech stack to prove isolation and the fact that you no one from the outside can in software modify these properties of like there will always be a there must always be a monitor that is signed by some authority that gets run, and then you have a blockchain out in the world that the various world like national governments are like federated consensus participants in, and the like every time there is an update to these classifiers, it gets signed by the multi-sig and broadcast to everyone, and these devices have to be able to connect to that blockchain every certain amount of time, otherwise they shut off. They have to be able to like run the classifier on every output from the AI, otherwise they shut off. They have to be able to like see and report and like reliably share metadata about any training run they are part of, otherwise they shut off. And if you try and open the thing, the chips get shattered. Like there are things like this that you could design and build and implement. It would take a lot of international cooperation. I'm not sure we're gonna have that. It would take like upstream of that, you need agreement that like this is a catastrophic risk. We are on the verge of losing control. Maybe there's what is what they call in the biz a warning shot. This would be a like recoverable catastrophe that leads to people saying, like, oh, we should do something about this. I think that it is, I think that I vary on a lot of the like what is the most likely outcome from all of this. I think that Doomer has become an interesting pejorative to just dismiss concerns generally, but I think that like it's reasonable to have a term for people who are like, there is nothing we can do, we're all doomed. And so I I like push Doomer over there and I do not ascribe that label to myself. I think there's a lot of things we can do. I would point to Dean Ball and Zvi Mofwitz as like probably the two clearest voices in the space who are like very vocal and public about a lot of these things. They are not for the most part, I think neither of them is like is dumb in any in any takes that I have seen of theirs, which is a very high bar.

SPEAKER_00 34:00

Um these are politicians.

SPEAKER_01 34:02

No, Dean, well, Dean is a like, I guess he's a researcher, but he's a policy guy. Okay, but also very, very good with with the models. He wrote, he was the main author on the AI action plan. And Z is a very well-known EA rationalist writer who does a lot of regular updates about the course of AI progress. And like I would they are obviously not in agreement about a lot of things, but they are very reasonable. And like it, I feel like if we were in the version of America where I think it was like 20% of the population when it came out, read common sense is the typical estimate by Thomas Paine. Like I think that you know, if we if America were like that, then like Dean and Z V would be like the main two voices that any anyone is listening to, and there wouldn't be a lot of anxiety about the water usage from data centers or things like that, because the the stakes are much, much higher than water.

SPEAKER_00 35:09

Right. Right. I think one of the things that I find really interesting sometimes about kind of like on one side, I want to say that I think AI is actually a moment for progressive politics to kind of step in more in in the stage in in American politics at least. But at the same time, then it's this kind of like a lot of misplaced fears, I guess, or like being being worried about the wrong thing of what can actually like what is actually the problem is not is not one thing. It's not like like the water stuff. I'm not super knowledgeable, but I've definitely read some things about like kind of putting a bit of skepticism on on some of the claims. But there are like much bigger problems that probably should be tackled first. Yeah, or thought about first.

SPEAKER_01 35:55

I think that there's one of the biggest risks I see is like this mimetic attractor of you don't care about my problem, therefore you are wrong, or you are not you aren't putting my problem first, therefore you're wrong. And I think that like I worry about water usage and like access to clean water generally. I'm not saying that that is zero concern, it's just not the highest concern, and I think that people who say it is, if they had the context that I have, would probably agree. And right like I think that the particularly I think the political left, because there isn't a notion of returning to some idyllic past where all the problems were solved, and like we don't need to like we don't need to go forward, we just need to go back to something that never was existed. Would strongly recommend Ada Palmer's inventing the Renaissance for those who haven't come across it in one of my favorite sci-fi authors, even though that's a history piece, which is actually her specialty. Like, there's the conservative, like, we're gonna go back to the golden age and like let's reinvent the golden age for that. But the like if you want to improve the world, there's a question of in what way do you want to improve it? And it's really hard for people on the left to build a coalition because there are a lot of different visions for what the future could or should look like. One of I I think that there are a lot of it's another Ada Palmer quote is that science fiction helps us or enables us to fight the moral skirmishes that we will face as a society before we actually have to face them.

SPEAKER_00 37:27

Right.

SPEAKER_01 37:28

And I think that it is very good to engage in worlds where super intelligence to in the idea that a super intelligence could exist. I think that there are some good science fiction efforts around this. The ones I mentioned, there's also things like the culture series gets invoked a lot. Greg Egan's diaspora is another one where I think that this is actually like a version of things going quite well, actually. Maybe maybe that one's the best, like one of my one of my favorite like things going well versions. It's not his most popular. People usually reference Permutation City. I think that they're wrong. Diaspora's better. But also, like if you if you read that and you don't like it, there's a glossary in the back. And also, like, I may have liked it for mathy physics-y reasons that other people might not like. Anyway, the I think there's a lot of ways this could go, a lot of th concerns that people could have. I don't know what it looks like for humans to have influence over super intelligences. Maybe this looks like AI systems generate fiction for like fictional universes that it could build for us to. And like, if you really enjoy Harry Potter magic, or if you really enjoy like exploring space, then the fraction of people who engage in those universes l changes the allocation of resources. But like there will only be a finite amount of compute for all time, finite amount of energy being captured and allocated towards it. How does that get decided? Is it a small number of people? Is it the intelligence itself? Is it a set of superintelligences? Does the do those last two matter in terms of the human experience? Why would why would the AI systems listen to people? And even if they did, how would you know? Why would you trust them?

unknown 39:15

Right.

SPEAKER_01 39:16

Like a superintelligence should be better at persuading you that it's listening to you just like it's better at solving all the other problems.

SPEAKER_00 39:23

Yeah, I guess the the the line that maybe I would identify for people who are maybe more concerned about the water issue than they are maybe the control of superintelligence, are probably people who are skeptical of superintelligence or like the idea that like a system could be so so powerful or so good at at the thing, and that's why the the issue is like they're selling us another bullshit thing and it's taken all our water. I think that's kind of like maybe the line, whether or not you believe in that.

SPEAKER_01 39:54

I understand that, and I think that one of the greatest disservices that the media has done the world at this point is convincing people that intelligence looks like characters from the Big Bang Theory and the ability to like stand up at a black blackboard and recite facts.

SPEAKER_00 40:10

Yes, I hate the Big Bang Theory. I hate this show so much just because it's like a dumb person's idea.

SPEAKER_01 40:18

Yeah, it's a dumb person's idea of a smart person. Exactly. I think that one of the more controversial takes I have is that like I think that there probably are really influential genes for intelligence. It just seems weird that there wouldn't be. Like your eye is sensitive to the polarization of light, and it's not because it's useful evolutionary, it's because it's really, really hard to make a polarization insensitive photon detector. And like, I think it would be really, really, really hard for evolution to make all humans exactly equally capable of grasping abstraction, which is kind of how I think of intelligence. But also, like, this is not something that should be, I think that people shouldn't be branded as like being on one end of the political spectrum for holding this belief, because the like there are polars that there are there are two different polarized takes you could have about this. One would be like, therefore, some people are better than others, and therefore they should have more, or something like that, because they'll distribute the resources more effectively, or something. You could tell how I probably infer how I feel about that take. Or, like, this is very important to know, therefore, we can help people who are disadvantaged, and we're actively disadvantaging them more by pretending like they are just as capable as everyone else. I also don't believe that like it's a hundred percent deterministic. I I think it only attribute accounts for part of it, but like this shouldn't like it could be an info hazard. It could be that civilization right now isn't well equipped to know which of these genes it is. I worry about like monoculturing uh like brain phenotypes as a result of this in the short term. I worry about a lot of weird things. Um but like I think that it's like I mean when I say monoculturing in this way, it's like I think that I'm clever enough that like if I grown up in a family of lawyers, I probably could have been a decent lawyer. If I'd grown up in a family of doctors, I probably could have been a decent doctor. Like that's transferable, but like I don't think you can take Mozart and put him into like Einstein's house as a baby and get a get Einstein again or get Mozart again. I think that that just doesn't work. And so like there you need some amount of diversity in that to get the level of flourishing that we have now. And I don't want people to suppress that because they're like intelligence maxing or something. But like there's yeah, there are lots of concerns in this, lots of yeah, directions.

SPEAKER_00 42:42

I I I I definitely like I definitely hear what you're saying. I think uh there is this kind of like murky question. Yeah.

SPEAKER_01 42:50

I remember the the point I was trying to make it too is I think that a lot of people think like going back to the they think intelligence is the big bang theory, they think that intelligence does not actually help solve problems, at least the real problems that they they deal with day to day. And I think that this is a mischaracterization. I think the people who are like, oh, like I've I've done so much better than so many intelligent people. I can like I'm not worried at all about a super intelligence. I think that the people who have that are actually just smarter than they think they are, and they have that raw intelligence and they don't realize it, and they think that they are not intelligent, and therefore intelligence is not this useful. This thing's been useful to them. And actually, it is a core thing that has made them much more effective. And the fact that society has convinced them that this is separate has led them to fail to extrapolate what that looks like much, much better. Like Robert Moses was like a a general, like an he was a natural general intelligence with a slightly higher clock speed and a slightly longer context window than everyone else. And he just wrecked all of New York governance controls in ways that like people. Should be aware of. And like I would point politicians to that as opposed to like 2001 or The Matrix or Terminator when I say, like, this is what superintelligence can do. It's like, imagine if Robert, if you had 10 Robert Moseses who could all coordinate with each other and were all deciding to back one particular person. What if it's a hundred? What if it's a thousand? Like that should hopefully feel more visceral.

SPEAKER_00 44:27

Yeah, there's to to maybe like expand a little bit on on also why I think big bang theory is kind of dumb, was or just like the what it what it kind of like conveys, I think, is like the idea of intelligence as being like you either have more of it or you have less of it. Like there is intelligence and it's like oh, it's like more or less intelligence. Which is just like ridiculous. It's like a ridiculous conception of intelligence. It's like people are better at better, at better at different things, but then there is a risk, I think, of like, you know, if we really take the logical conclusion of, you know, designer babies with AI, you know, scanning genetics and then being able to choose which genes get are inside your baby's brain, and everyone being like, well, I want my kid to be like really good at math, of course, because that's like the most coveted skill or whatever in in the workplace right now. So that's you know, and then that being like this kind of yeah, monoculturing then of of the human brain where everyone's like really good at math, but no one has knows how to talk about their feelings or whatever.

SPEAKER_01 45:28

And thinks that their babies are going to be better at math than computers anytime. Also, I mean that was maybe that was a bad example. But yeah, no, I I think the point stands. There's there like this notion of non non-scalar intelligence also is very relevant to AI. There's a great um take from Helen Toner about jaggedness of AI capabilities. And like, yeah, AIs are really, really good at some things. In fact, like most of the things that I think about, most of the capabilities I think about are I think things that basically no one uses, like cyber red teaming or like uh like assisted biodesign and things like this, where like it takes some amount of skill with AI to be able to evoke these and knowledge of the field to be able to use them well. Like, my understanding is that at the Frontier Labs, humans don't write any code anymore. And like this is something that seems like it should be having economic implications soon, but I think it'll take a procurement cycle or two. I yeah, I like people don't even if the procurement people don't understand that this isn't just a tool, this is actually like an employment, like an employee replacement, then like they're going to slow this down. So I think that there are bottlenecks, but like there are also definitely things that AI systems can't do. And some of those things are things that people are much more likely to ask AI systems to do, and so they can be much like justifiably skeptical. But I think that the like the radio chart of like where is it better than humans, where is it much, much, much better than humans, and where is it not as good as humans, like it is good to understand intuitively more than like trying to say, like, oh yes, it we're not at an AGI yet, or we are at an AGI now, like that those terms just stop holding meaning.

SPEAKER_00 47:16

Yeah, yeah. Super, super meaningless terms. But I think so one of the things that I've that I've gathered from like your writings and what I whatever you say before is that like essentially what we're kind of like needing to gear towards probably is some sort of governable AI. Like how how like or like to really the question is like how do we govern this AI? And if the question is, do we want to govern it, which I think for a lot of people is yes, then what does that look like? I mean, there are people, of course, on the I don't know if they exist anymore, really affect any as much, but effective uh accelerationists saying we shouldn't, we shouldn't govern it, that we should just let it freely do whatever it wants, and we just like cave to its whim. But yeah, I think probably the two of us here don't don't really agree with that that take.

SPEAKER_01 47:59

Yeah, I there are effective accelerationists still out there. Mark Andreessen still posts some like pretty unhinged things about AI. And there's a lot of money behind this, like, don't regulate AI and in ways that like seem deeply irrational to me. I do think that there's a connection to like, well, I ended up thinking about formal verification because I think that it is a very useful construction for a lot of different purposes. Formal verification is this notion that you have code and you have a specification that is a mathematical description of how the code should, or like mathematical or logical description of how the code should behave. And then the there's a mathematical proof that takes the definition of the programming language as the like starting axioms and then proves that the code has these properties. This is what's used for the most like critical of software infrastructure for a lot of things. Like you can prove parts of the stability of aircraft systems, and that's why you never hear of like, oh yeah, there was a bug in the code. You hear like, oh, there was a problem with the sensor. Because, like, obviously, if the sensor's broken, then the code is going to misfunction malfunction. But like you formally verify a lot of the critical systems so you know that like if like two trains in France are not going to crash, this is another like prime example of where it gets deployed. I think that you could hypothetically generate a system where if you have like widely deployed flex hegs, which I mentioned earlier, you could imagine having something that looks like a bill of rights encoded into AI systems so that before they take any action, they have to prove that their action is, to the best of their understanding, consistent with that bill of rights. This seems like a delightfully like uh like Western liberal direction for how one might govern AI. It requires an a separable auditable world model in which the AI can test, like, does this in fact do this thing? But like it was I had I came to this with some delightful conversations with Mark Miller of Agoric about who like takes some very strong libertarian views on basically everything. And we landed on like, yes, this would actually be really cool because you could embed, you could imagine embedding in AI systems a something that is as fundamental as the laws of physics are for us, but because they all exist in compute, if you can put this into all compute, you could put this into all of the compute that the AI systems generate in a like reflections on trusting, trust-ish kind of way for those who get that reference. Like you could just embed this and it would would propagate in a useful way. And you could have constraints on like you should value like honesty. You should like do not take actions where the transmission of withheld information would lead your interlocutor to um like have certain responses, including like changing their mind about collaborating with you, kind of thing, in in ways that you could start defining this. This starts getting very far from formal verification in that like specifying those things objectively in properties that you can describe in the simulation sounds completely intractable today. But you could start thinking about what does it look like to like what do things in the interim start looking like? And between here and there, we could have mathematical, like logical descriptions of like the building code, tax policies, legal contracts. A lot of these things are like the tax policy, as a great example, is software. Like the canonical implementation of the tax code is running on a server somewhere. For the US, I understand that is not running on a particularly well-updated operating system or language, because it still works, but like that's actually what the tax code is, plus all of the infrastructure and like the institution around it that lead to modifications thereof. But it's like it's not actually the law, and it would be really useful to be able to formalize all of those things into something that looks like software or at least property, like pre and post conditions of like if the input satisfies these properties, the output will satisfy these properties. That would be an example of a spec. You could have specs for all of these things, and you could have like one-click check. Does this meet the building code? Does like, is this the correct tax? And we could have all of these. I'm actually working on fundraising for a focus research organization at Convergent Research that will build tools to help people validate is this the correct formalization of this math theorem or this software property, or it could apply to like this hardware property or this piece of the tax code or this piece of the contract. Like, help me understand the difference between this and a variant I might generate. Why would it matter? When would it matter? And I think that we could imagine a world where we have AI systems that we govern not by like saying this is the correct set of virtues, embody those virtues, but rather here's a set of laws that are we consider fundamental as like ground rules. Do not break these, whatever else is fine. And I think that there are versions of this that would feel much, much better to most people who have grown up in like Western liberal democracies.

SPEAKER_00 53:20

So more than just like a clot skill, but having a kind of like mathematical instantiation representation of like what you expect the AI to act or how you expect it to act in certain maybe conditions, and that formal verification is a a particular field, I guess, within mathematics that is meant to be like comprehensive, I guess, in in ways.

SPEAKER_01 53:46

Within your science, actually. But like you end up with like some formal logicians from the math side. And yeah, it's it's like trying to trying to make things very explicit. It gets used in a lot of places. I think it should be used in more places. It's been mostly limited by the fact that you need people who have deep expertise in formal logic to learn these languages and then to generate these proofs, and all these things are very hard. They require time and expertise, and thankfully, time and expertise are two things that get much cheaper as AI gets more capable. So I think that we're seeing a lot of really cool things happening. There's a a different focus research organization. The and like FRO, focus research organizations or FROs are these like 10 to 20, sorry, 10 to 50 million dollar three to five year research efforts to create public goods that are too big for a university lab, too small for like a big government center, not profitable enough for a private industry, very startup flavored, but it's like the the goal for each one is to build a product that seeds an ecosystem that drives a scalable revolution. And conversion research is like I I wear a hat there as a research fellow, and I do so delight delightedly. And they've created 10 of these FROs, I think, and are maybe maybe more at this point. And there's a lot more interest in those. And each of them, like you can check the conversion research website, each of them seem poised to revolutionize like single-cell proteomics or connectome map, like brain connectome mapping, or like intergalactic astronomy and things like this. And like they're very cool. I am trying to help them build things that are useful for AI resilience, AI security, etc. And there's another one that it built lean, the proof checker, which has been getting a lot of attention in a lot of like software security a little bit, right? But definitely the math world a lot. And I'm like very excited about some of the progress they're making where they had Claude generate an instance of the compression library in lean entirely and prove that unzip of zip of file equals file for all file, which would be an example of a simple spec that you would want for your compression library. So anyway, there lots of cool things. Specifying things is very hard. And like you never know you specified the right thing. And if your model of the world is wrong, you've also not like there will be gaps. But I think it's it's much closer to how we it's much closer to the like legal form of how we deal with society. We don't point to like that's the most moral person, be like them, which is what alignment is. It's more like follow the ground rules.

SPEAKER_00 56:28

This is definitely, I feel like it it's like the other direction of Lessig's, you know. I interviewed him before, but you know, code is law, except in this time in this way it's uh you know formally verified law on AIs, which is like, you know, and if you're in crypto, you're used to that being thrown around as a way to like justify hacks or whatever. In this case, it's a quite different yeah.

SPEAKER_01 56:51

I I loved that like in the original context, that was not that like software as physical law, but rather uh it was intended as like software as like legal legal code, like software code as legal code. And like I think that that one makes a lot more sense and like it makes sense to try and optimize the checking of these things. If you can put more effort into saying what you want, then you don't have to check to work as hard to know if what you got was what you wanted.

SPEAKER_00 57:21

Right. So if we can maybe before we end, try to make the the connection with blockchains because I feel like because there there is, I think there is there is a connection, we talked about it a little bit before, you know, which I think is kind of funny because Peter Thiel had that. I mentioned this before, he had that ridiculous quotes, but also I kind of see where he's coming from, but also I think it's dumb of like AI is communism and crypto is libertarianism, which I think is kind of related to to maybe what what you'll mention as like where blockchain kind of comes in with AI. To me, like it's potentially a tool for helping out the governance of AIs to some degree, and in certain places where where where it makes sense, which is like not the maybe libertarian angle so much that that people would have thought.

SPEAKER_01 58:08

I think that this is an interesting question. I apologize that I have to rush it a little bit. I think that as intelligence gets cheap, that agreement or consensus gets very expensive. And I think as software becomes cheap, that like the moat around a lot of software infrastructure and information processing generally disappears. And I think that having consensus, having agreement, making that agreement quantifiable and objective and verifiable is very, very useful. A lot of the like I look at lots of risks and I try to map out what could be done. And like blockchains do show up a maybe surprising amount to a lot of the people as like possibly the best way to have this because you can start reasoning robustly about security models. What at least one of the ideas, like one of them I mentioned, another one I am hesitant to say because I'm not sure it falls nicely on the line. I I think it becomes very good in some scenarios and very bad in other scenarios. And so I like don't want to put it too much into the water if I don't know which scenario we're in. I think that there like a lot of the questions around labor and capital become very, very interesting. And I I expect that people who have thought very deeply about implications as we move into making more labor synthetic will be very interesting. Um lot of interesting discussions around like a permanent uh economic underclass uh being a possible outcome. And like it's very unclear what it's very unclear what good even looks like, but also very very unclear how we would go and get there separately.

SPEAKER_00 59:54

Right. Fair. We'll we will continue the discussion another time, because I know you gotta go. Maybe if you just want to give the last last plugs on Latless Computing, I highly recommend people check it out, check out their website and check out the writing.

SPEAKER_01 1:00:07

Yeah, very much appreciate it. I would look forward to another conversation very much. I would say that our efforts are currently to try to scale the field strategy work that I've been doing of like making it very, very easy for founders to pursue new efforts that looks like a combination of like taking going from a problem to a plan that has been somewhat de-risked with buy-in from experts and the potential like users or stakeholders. And I'm like the rough skill set is like can you go into a meeting with like a Fortune 500 director and convince them that this new solution is viable and that they should agree to spend time possibly engaging in collaboration to address some market failure that would lead to a better outcome for society or civilization more generally? People who can point at a thing and say, this would be very useful. No one has done this. And I can tell you with confidence because if they had one of these seven people would know, and I've talked to all of them, and they could all they all agree that no one is doing this thing. I it's a like a weird skill set that shows up in like startup product market fit, or like being an entrepreneur, or like being a like head of strategy at a big company, or like sometimes in consulting, often in research. I assume it shows up in like the intelligence community, a bunch of weird places for different fields. If you, as a listener, are very, very interested in doing that and worried about how AI will break society in the very near future, particularly for cybersecurity risks or biosecurity risks or maybe epistemic risks. I would love to hear. You can find more information at atlascomputing.org. And then also we're somewhat funder funding limited. So if people are sitting on a like large capacity to donate to a US 501c3, would love to be would love outreach in that for that too.

SPEAKER_00 1:02:06

Sure. Awesome. Well, thanks so much for coming on. Really appreciate it you sharing your perspective. Yeah, thanks for having me, Josh. Thanks, time. If you like what I'm doing here, consider supporting the show on Patreon. Your contributions help me keep doing this work and dive deeper into the politics of decentralized technologies. I promise you absolutely zero financial returns, no airdrops, and your investment may go to zero. But you will get good content. Check out patreon.com slash the blockchain socialist to support the show.