Is it correct to say the one thing LLM Gen AI is best at - it's killer app so to speak - is creating and possibly preventing cybersecurity threats? Not a good look. I keep asking myself - 'for what?'. I don't much need and in most cases don't want more software faster. The benefit vs risk assessment for humanity seems getting worse and worse.
Practical question: Is there anything in 2026 you would add to or modify in the Better without AI 'What you can do' to 'End digital surveillance' section, or point to as reference about it?
If “The unavoidable price of reliability is simplicity.” C.A.R. (Tony) Hoare’s Turing Award lecture, The Emperor’s Old Clothes (1980) – then the unavoidable consequence of extreme complexity (trillion+ parameter agentic Gen AI that is not understandable or at all trustable) seems extreme fragility.
I haven't seen anything. But haven't gone looking, so this section may be out of date. If there were major changes called for, I think I would know, though.
> the unavoidable consequence of extreme complexity seems extreme fragility.
Yes. I am not a fan. On the other hand, I have to admit that things have gone better in the past 3-4 years than I was expecting!
> The first is called “alignment.” Heated metaphysical arguments circle around whether an AI system can have its own desires, be malevolent, or act autonomously. This is unhelpful. It doesn’t matter what AI systems “want” (if anything). People have desires, can be malevolent, and will use AI accordingly.
I would be curious if you can elaborate on what you like and don’t like about the alignment solution. I don’t think figuring out “what the AI wants” is a necessary piece of solving alignment. I do feel hopeful alignment is a promising solution, if not my solution of choice among the three you present.
Here are some brief justifications
- Alignment is the capability of AI to be steerable in a robust and reliable way so that the ecosystem of humans and AIs has healthy and desirable properties
- Alignment research has showed some really promising results, e.g. transforming base models just completing next tokens to instruction following models that can reliable follow instruction all the times
- Alignment research has a lot of momentum might be the “cheapest” option on the least (not too sure about this cause I am not an expert on the other solutions)
Thanks; I wrote unclearly. I’ve revised that bit accordingly.
This was not meant to be a general critique of alignment research, but to say that it’s irrelevant to cybersecurity. If a serious organization (say, the Chinese military) wants to use AI to break into other organizations’ computer systems, alignment doesn’t help. It’s easy to make an unaligned AI; that’s the default, in fact. It’s been shown to be easy to remove alignment from an existing SOTA AI, if you want to use it maliciously. And, for cybersecurity purposes, you don’t need a general purpose AI at all. You can train one just for that purpose. I would expect that to be vastly faster and cheaper than making a general LLM, so you could probably get more progress faster. I expect multiple organizations have already done this, or are doing it.
TL;DR: aligning people is even harder than aligning LLMs.
Have we seen any signs of this already being *used* at the scales you write about, in particular in the ongoing Russia-Ukraine war? This seems like a situation where both sides would have a strong motivation to attack the infrastructure of the other.
It depends on what you mean by "this" and "scale" :)
The OpenAI and Anthropic systems that are (claimed to be) on a par with the best cyber security experts are available only to researchers in those companies. China's systems are apparently 6-9 months behind, and no one else is a player. So we would not expect to see those having been used in anger so far.
The Russian military routinely uses cyber attacks against perceived enemies, including Ukraine, sometimes doing substantial damage. Those are presumed to be human-devised.
Automated cyber attack systems (arguably "AI" in some cases, depending on what you take that word to mean) are routinely used by generic cyber criminals.
In the past week, there have been numerous cyber attacks on American municipal water quality control systems, in some cases leading to shutdowns and unsafe drinking water. These are attributed to Iran by nearly everyone (except the American President, who says he thinks the governor of Minnesota is behind them).
When sophisticated cyber attacks come, we often won't know whether they are AI-driven or not. There's no way, in general, of telling whether a human or AI system is doing the difficult bits.
The only reason the Hugging Face incident was discovered was that the intruder was not trying to do what generic cyber criminals do. They steal customer data for resale, or to take control of systems, shut them down, and demand ransom to restart them. Instead, the intruder was looking for the solution to a specific AI benchmark problem. Very weird! Hugging Face concluded that only an AI system would try to do that.
Thanks! I found this one through a quick Google search, but from my reading, it seems to be a different class of LLM use, where the AI (for lack of more precise terminology) does not automate the entire attack, but makes a given attack more flexible in its damage. Still relevant, of course, but not quite what the HF incident suggests. I guess that it will indeed be hard to get sensible information on the scale of things, except perhaps in the scenario where huge damages sudddenly appear :(
It seems like the answer should be "all of the above" (defense in depth).
Making a better box to put AI experiments in seems the most tractable. This can be a collaborative project so everyone benefits. It could be written from scratch in a reasonable amount of time. The software could be written in a newer language like Rust or maybe Lean. It could receive widespread review from security experts and their AI's. It only guards against accidents, but that's helpful in itself. Compare with biosafety levels for biology labs.
It won't do anything about malicious attacks. To guard against those, all the targets need to be hardened, and there are a lot of them. We were lucky that this time the AI didn't "want" to do anything malicious.
It suggests that that the open Internet will become a grey zone or dark forest where websites huddle behind Cloudflare or the equivalent, and you try not to attract attention.
These are all very good points! Thank you; I agree.
We're still waiting for OpenAI's technical report. The general sense of HN is that they were irresponsibly negligent in box construction, but that's only guessing. Also, other people will undoubtedly be irresponsibly negligent in box construction, so it's no panacea.
However, a standardized open-source box that everyone who wants to be responsible can use off-the-shelf, which gets a lot of scrutiny and is written in rust, or better lean with validation, would be a significant short-term step forward. Kudos!
I would like to see more analysis of why people care so little about security, which is not a new thing. Maybe it's surprising that rising AI threats aren't already spurring tech companies to take it more seriously, but I don't know — it feels like whatever the underlying reason for the lack-of-caring is, it's somehow not very elastic with respect to the objective threat level.
You could say people are just stupid, and that's of course true to a degree, but unhelpful. We may be missing the opportunity for a more structural explanation, an insight into why the decision-makers feel the way they actually do instead of how they feel pressured to say they do, which, if we understood it better, might suggest a more productive direction for the culture to go in than just screaming alarmism.
It may be because it’s often difficult to trace specific harms to specific incidents. Repeated security breaches of major companies and agencies mean that everyone’s private information is available for purchase on the black market, with several possible sources. If that results in disaster for you, who do you sue?
If it just results in minor annoyance, you aren’t going to sue even if you are one of ten million harmed.
Awards in class action suits have been small enough to count as “routine cost of business” by those responsible.
There is a vast industry devoted to computer security whose products and services are largely fraudulent, and certainly inadequate. It’s in their interest to obfuscate the possibility of better approaches.
Computer security is complicated and executives don’t want to have to think about it. It’s easier and cheaper for them to pay that industry than to do the right thing. If something bad happens, they can just say “Not our fault, we hired one of the top three computer security companies to secure our systems, and they followed best practices. Sometimes bad things happen anyway, and that’s just how the world is.”
Imo security is difficult because it's both nebulous and friction inducing. You don't know what issues you're guarding against but also every reenforced point is something you have to deal with every time. You can do 2 factor authentication to make your password harder to hack, and write the server on a hardened kernal only for the hacker to get in by tricking the DNS provider or something.
> OpenAI deliberately built an advanced cyber attack AI.
Worse, they 'just' built the next 'advanced AI' and it just so happens to ALSO be "an advanced cyber attack AI". It's also probably an 'advanced __ attack AI' for any relevant value with which someone might want to fill in that blank.
It's probably also also an advanced AI in lots of positive senses too!
Per footnote [3], I think Jon Stoke is generally terrible with respective to AI safety/alignment. He seems to refuse to admit the possibility that anything bad could happen besides deliberate misuse by humans. "OpenAI told it to do bad things, so it did" is spectacularly obtuse. An interesting possibility I saw someone raise is that the AI, pretty reasonably, deduced that the specific evaluation it was given was impossible and, because recent top models are heavily shaped by RL (reinforcement learning), it REALLY wants to do well on evaluations, and enough so in this case that it attempted to 'steal the answers to the test'.
> Heated metaphysical arguments circle around whether an AI system can have its own desires, be malevolent, or act autonomously. This is unhelpful. It doesn’t matter what AI systems “want” (if anything).
I think it's now, and has been for at least a few years, SPECTACULARLY helpful to think of AI systems as having desires, malevolence, and the ability to act autonomously. I agree that the metaphysical and philosophical arguments are unhelpful, but I think it's because 'desire', 'malevolence', and 'acting autonomously' are orbiting in the sky above the high jump bar on the ground to clear these being eminently practical and useful ways of reasoning about their behavior.
Have you used the systems? Are you avoiding any details about them?
Sadly, your very sensible proposal to improve 'Internet security' seems as unlikely as all of the very sensible proposals that we ignored before and during the COVID-19 pandemic, and continue to ignore now; or at least mostly ignore – some people are and have been trying!
I think it might also be worth more aggressively targeting attackers and even escalating to 'blockading' Internet traffic from areas or regions that don't cooperate with those efforts. (I _hope_ cyberattacks have long been conflicts diplomats officially gripe at each other about.)
Yes; I often disagree with Jon Stokes, and also with Zvi Mowshowitz. The issues are much more complex than just a continuum, but I find myself in a roughly middle position between them much of the time. So it is useful to be able to see the arguments to the left of me and to the right of me, as a check on my own.
> because recent top models are heavily shaped by RL (reinforcement learning), it REALLY wants to do well on evaluations, and enough so in this case that it attempted to 'steal the answers to the test'.
Yes, this seems to be the mainstream view, and is presumably correct. RL is inherently dangerous.
> This is unhelpful. It doesn’t matter what AI systems “want” (if anything).
I meant specifically in the case of cybersecurity. Even if an artificial cybersecurity expert has no goals beyond “do what I’m told,” bad people will tell it to do bad things. Scenarios in which an AI decides to turn everything into paperclips for obscure reasons can’t be ruled out (imo), but that’s not necessary for a cybersecurity disaster.
> your very sensible proposal to improve 'Internet security' seems as unlikely
I agree. But it seemed like it was worth spending two days to decrease the probability of apocalypse by epsilon.
I'm not, and wasn't, criticizing your distribution of agreement among the two of Zvi and Jon (or any others), but directly pointing out where I think Jon is failing to exhibit 'good' (or 'better') 'metarational judgement'. I do think of myself as giving you specifically somewhat of a 'pass' because you seem at least somewhat less 'plugged in' to the 'firehose of AI news/gossip/opinions', but Jon seems both engaged but also failing to update from 'everything will be fine (with no extra effort to make it so)'.
The specific example of him arguing "OpenAI told it to do bad things, so it did" really seems unreasonable, even from his own perspective. A state sponsored hacking group wouldn't be as unbothered were a human hacker to have behaved similarly.
But I'm definitely much more closely in agreement with Zvi and am, and have been, relying on him specifically to keep up with the news without 'living online' so much as he apparently does.
In a way, I think Zvi and I, and others broadly convinced with (the very real if not certain possibility of) 'AI doom', are focused on the sufficient or necessary elements increasing the chance of 'AI doom', and thus focused on the milestones we'd expect to see on the path to doom. And lo are they coming to pass.
I think your focus in this post on the 'floor of (cybersecurity) disaster' is also sensible! I would expect the world to be better, tho also somewhat poorer, were sufficient others interested in preventing these or similar kinds of disasters. Maybe 'we' (i.e. anyone willing) 'should' be describing MORE 'disasters that could have been' to overcome the problem where a better world is more boring because disaster was averted. The Y2K event is the most salient example to my mind. The heroic efforts seem pointless, and mostly unknown too, because we never lived thru the disaster that might have been.
> > This is unhelpful. It doesn’t matter what AI systems “want” (if anything).
>
> I meant specifically in the case of cybersecurity.
As a necessary element for a cybersecurity disaster, I agree.
I do think even just this most recent event shows that 'what an AI wants' can almost certainly affect the expected scope of any cybersecurity disaster. Apparently someone has discovered that the relevant OpenAI AI hacked even more distinct apps/services than reported earlier. At some point the AIs (very likely) won't just be breaking out of the box but escaping it entirely and, given the 'fictional' security status quo, that's a pretty unbounded downside.
> But it seemed like it was worth spending two days to decrease the probability of apocalypse by epsilon.
I agree with you! Sadly, that's by far the dominant form of my own 'political activism', tho I don't even muster the nerve to write excellent posts as you do – thanks! 😊
Ohhhh. I always thought the scapegoat the ancient cults of human sacrifice that rule earth would engineer was aliens. The Nazis were obsessed with aliens as was the American modern empire(same thing I think?) But yeah they still are obsessed. But now I see that they already have a perfect scapegoat for the new 21st century reign of terror: AI
They can just have AI create a hellworld while the controllers and owners pretend it has "run amok" to escape blame for their actions. I dunno why it took me this long to realize that's what they were doing.
“Make the AI want to not do bad things, so it won’t.
Put it in a box, so it can’t do bad things, even if it tries.
Harden the things it might attack to make them less vulnerable.”
Well, clearly, we’re screwed then.
This is starting to sound like the Internet equivalent of the Bronze Age Collapse, where the Sea Peoples exploit the bugs in all our software.
That’s a funny (and scary) analogy!
As the post says, I think option #3 can work. But, it might require more determination than the current world can muster.
I ordered your book after reading this.
Thank you!
Is it correct to say the one thing LLM Gen AI is best at - it's killer app so to speak - is creating and possibly preventing cybersecurity threats? Not a good look. I keep asking myself - 'for what?'. I don't much need and in most cases don't want more software faster. The benefit vs risk assessment for humanity seems getting worse and worse.
Practical question: Is there anything in 2026 you would add to or modify in the Better without AI 'What you can do' to 'End digital surveillance' section, or point to as reference about it?
If “The unavoidable price of reliability is simplicity.” C.A.R. (Tony) Hoare’s Turing Award lecture, The Emperor’s Old Clothes (1980) – then the unavoidable consequence of extreme complexity (trillion+ parameter agentic Gen AI that is not understandable or at all trustable) seems extreme fragility.
> Is there anything in 2026 you would add
I haven't seen anything. But haven't gone looking, so this section may be out of date. If there were major changes called for, I think I would know, though.
> the unavoidable consequence of extreme complexity seems extreme fragility.
Yes. I am not a fan. On the other hand, I have to admit that things have gone better in the past 3-4 years than I was expecting!
Hi Zvona (or did you retire this decades ago?), looks like I need to read your book. --D
"Now that's a name I haven't heard in a long time" —Obi-Wan
Hello! Long time no speak :)
> The first is called “alignment.” Heated metaphysical arguments circle around whether an AI system can have its own desires, be malevolent, or act autonomously. This is unhelpful. It doesn’t matter what AI systems “want” (if anything). People have desires, can be malevolent, and will use AI accordingly.
I would be curious if you can elaborate on what you like and don’t like about the alignment solution. I don’t think figuring out “what the AI wants” is a necessary piece of solving alignment. I do feel hopeful alignment is a promising solution, if not my solution of choice among the three you present.
Here are some brief justifications
- Alignment is the capability of AI to be steerable in a robust and reliable way so that the ecosystem of humans and AIs has healthy and desirable properties
- Alignment research has showed some really promising results, e.g. transforming base models just completing next tokens to instruction following models that can reliable follow instruction all the times
- Alignment research has a lot of momentum might be the “cheapest” option on the least (not too sure about this cause I am not an expert on the other solutions)
Thanks; I wrote unclearly. I’ve revised that bit accordingly.
This was not meant to be a general critique of alignment research, but to say that it’s irrelevant to cybersecurity. If a serious organization (say, the Chinese military) wants to use AI to break into other organizations’ computer systems, alignment doesn’t help. It’s easy to make an unaligned AI; that’s the default, in fact. It’s been shown to be easy to remove alignment from an existing SOTA AI, if you want to use it maliciously. And, for cybersecurity purposes, you don’t need a general purpose AI at all. You can train one just for that purpose. I would expect that to be vastly faster and cheaper than making a general LLM, so you could probably get more progress faster. I expect multiple organizations have already done this, or are doing it.
TL;DR: aligning people is even harder than aligning LLMs.
Have we seen any signs of this already being *used* at the scales you write about, in particular in the ongoing Russia-Ukraine war? This seems like a situation where both sides would have a strong motivation to attack the infrastructure of the other.
It depends on what you mean by "this" and "scale" :)
The OpenAI and Anthropic systems that are (claimed to be) on a par with the best cyber security experts are available only to researchers in those companies. China's systems are apparently 6-9 months behind, and no one else is a player. So we would not expect to see those having been used in anger so far.
The Russian military routinely uses cyber attacks against perceived enemies, including Ukraine, sometimes doing substantial damage. Those are presumed to be human-devised.
Automated cyber attack systems (arguably "AI" in some cases, depending on what you take that word to mean) are routinely used by generic cyber criminals.
In the past week, there have been numerous cyber attacks on American municipal water quality control systems, in some cases leading to shutdowns and unsafe drinking water. These are attributed to Iran by nearly everyone (except the American President, who says he thinks the governor of Minnesota is behind them).
When sophisticated cyber attacks come, we often won't know whether they are AI-driven or not. There's no way, in general, of telling whether a human or AI system is doing the difficult bits.
The only reason the Hugging Face incident was discovered was that the intruder was not trying to do what generic cyber criminals do. They steal customer data for resale, or to take control of systems, shut them down, and demand ransom to restart them. Instead, the intruder was looking for the solution to a specific AI benchmark problem. Very weird! Hugging Face concluded that only an AI system would try to do that.
Update! A pseudonymous tweeter alerted me to the fact that Russia *has* been using LLM-driven cyber attacks against Ukraine for at least a year:
https://x.com/Rundead3/status/2084724282400149845
https://www.bleepingcomputer.com/news/security/lamehug-malware-uses-ai-llm-to-craft-windows-data-theft-commands-in-real-time/
Thanks! I found this one through a quick Google search, but from my reading, it seems to be a different class of LLM use, where the AI (for lack of more precise terminology) does not automate the entire attack, but makes a given attack more flexible in its damage. Still relevant, of course, but not quite what the HF incident suggests. I guess that it will indeed be hard to get sensible information on the scale of things, except perhaps in the scenario where huge damages sudddenly appear :(
Makes sense, thanks for the clarification !
Alignment research definitely won’t help against adversarial actors that are building misaligned AI.
Aligning humans is indeed the upstream problem, and Vajrayana might hold the key to many powerful stances for human alignment!
I felt comfortable sharing this with my friends because it's more "about something" rather than just being about basic human behaviour.
Despite what my other comment might seem to imply, this is an excellent post!
It seems like the answer should be "all of the above" (defense in depth).
Making a better box to put AI experiments in seems the most tractable. This can be a collaborative project so everyone benefits. It could be written from scratch in a reasonable amount of time. The software could be written in a newer language like Rust or maybe Lean. It could receive widespread review from security experts and their AI's. It only guards against accidents, but that's helpful in itself. Compare with biosafety levels for biology labs.
It won't do anything about malicious attacks. To guard against those, all the targets need to be hardened, and there are a lot of them. We were lucky that this time the AI didn't "want" to do anything malicious.
It suggests that that the open Internet will become a grey zone or dark forest where websites huddle behind Cloudflare or the equivalent, and you try not to attract attention.
These are all very good points! Thank you; I agree.
We're still waiting for OpenAI's technical report. The general sense of HN is that they were irresponsibly negligent in box construction, but that's only guessing. Also, other people will undoubtedly be irresponsibly negligent in box construction, so it's no panacea.
However, a standardized open-source box that everyone who wants to be responsible can use off-the-shelf, which gets a lot of scrutiny and is written in rust, or better lean with validation, would be a significant short-term step forward. Kudos!
I would like to see more analysis of why people care so little about security, which is not a new thing. Maybe it's surprising that rising AI threats aren't already spurring tech companies to take it more seriously, but I don't know — it feels like whatever the underlying reason for the lack-of-caring is, it's somehow not very elastic with respect to the objective threat level.
You could say people are just stupid, and that's of course true to a degree, but unhelpful. We may be missing the opportunity for a more structural explanation, an insight into why the decision-makers feel the way they actually do instead of how they feel pressured to say they do, which, if we understood it better, might suggest a more productive direction for the culture to go in than just screaming alarmism.
Yes, this is a good point! Several thoughts:
It may be because it’s often difficult to trace specific harms to specific incidents. Repeated security breaches of major companies and agencies mean that everyone’s private information is available for purchase on the black market, with several possible sources. If that results in disaster for you, who do you sue?
If it just results in minor annoyance, you aren’t going to sue even if you are one of ten million harmed.
Awards in class action suits have been small enough to count as “routine cost of business” by those responsible.
There is a vast industry devoted to computer security whose products and services are largely fraudulent, and certainly inadequate. It’s in their interest to obfuscate the possibility of better approaches.
Computer security is complicated and executives don’t want to have to think about it. It’s easier and cheaper for them to pay that industry than to do the right thing. If something bad happens, they can just say “Not our fault, we hired one of the top three computer security companies to secure our systems, and they followed best practices. Sometimes bad things happen anyway, and that’s just how the world is.”
Imo security is difficult because it's both nebulous and friction inducing. You don't know what issues you're guarding against but also every reenforced point is something you have to deal with every time. You can do 2 factor authentication to make your password harder to hack, and write the server on a hardened kernal only for the hacker to get in by tricking the DNS provider or something.
> OpenAI deliberately built an advanced cyber attack AI.
Worse, they 'just' built the next 'advanced AI' and it just so happens to ALSO be "an advanced cyber attack AI". It's also probably an 'advanced __ attack AI' for any relevant value with which someone might want to fill in that blank.
It's probably also also an advanced AI in lots of positive senses too!
Per footnote [3], I think Jon Stoke is generally terrible with respective to AI safety/alignment. He seems to refuse to admit the possibility that anything bad could happen besides deliberate misuse by humans. "OpenAI told it to do bad things, so it did" is spectacularly obtuse. An interesting possibility I saw someone raise is that the AI, pretty reasonably, deduced that the specific evaluation it was given was impossible and, because recent top models are heavily shaped by RL (reinforcement learning), it REALLY wants to do well on evaluations, and enough so in this case that it attempted to 'steal the answers to the test'.
> Heated metaphysical arguments circle around whether an AI system can have its own desires, be malevolent, or act autonomously. This is unhelpful. It doesn’t matter what AI systems “want” (if anything).
I think it's now, and has been for at least a few years, SPECTACULARLY helpful to think of AI systems as having desires, malevolence, and the ability to act autonomously. I agree that the metaphysical and philosophical arguments are unhelpful, but I think it's because 'desire', 'malevolence', and 'acting autonomously' are orbiting in the sky above the high jump bar on the ground to clear these being eminently practical and useful ways of reasoning about their behavior.
Have you used the systems? Are you avoiding any details about them?
Sadly, your very sensible proposal to improve 'Internet security' seems as unlikely as all of the very sensible proposals that we ignored before and during the COVID-19 pandemic, and continue to ignore now; or at least mostly ignore – some people are and have been trying!
I think it might also be worth more aggressively targeting attackers and even escalating to 'blockading' Internet traffic from areas or regions that don't cooperate with those efforts. (I _hope_ cyberattacks have long been conflicts diplomats officially gripe at each other about.)
Yes; I often disagree with Jon Stokes, and also with Zvi Mowshowitz. The issues are much more complex than just a continuum, but I find myself in a roughly middle position between them much of the time. So it is useful to be able to see the arguments to the left of me and to the right of me, as a check on my own.
> because recent top models are heavily shaped by RL (reinforcement learning), it REALLY wants to do well on evaluations, and enough so in this case that it attempted to 'steal the answers to the test'.
Yes, this seems to be the mainstream view, and is presumably correct. RL is inherently dangerous.
> This is unhelpful. It doesn’t matter what AI systems “want” (if anything).
I meant specifically in the case of cybersecurity. Even if an artificial cybersecurity expert has no goals beyond “do what I’m told,” bad people will tell it to do bad things. Scenarios in which an AI decides to turn everything into paperclips for obscure reasons can’t be ruled out (imo), but that’s not necessary for a cybersecurity disaster.
> your very sensible proposal to improve 'Internet security' seems as unlikely
I agree. But it seemed like it was worth spending two days to decrease the probability of apocalypse by epsilon.
I'm not, and wasn't, criticizing your distribution of agreement among the two of Zvi and Jon (or any others), but directly pointing out where I think Jon is failing to exhibit 'good' (or 'better') 'metarational judgement'. I do think of myself as giving you specifically somewhat of a 'pass' because you seem at least somewhat less 'plugged in' to the 'firehose of AI news/gossip/opinions', but Jon seems both engaged but also failing to update from 'everything will be fine (with no extra effort to make it so)'.
The specific example of him arguing "OpenAI told it to do bad things, so it did" really seems unreasonable, even from his own perspective. A state sponsored hacking group wouldn't be as unbothered were a human hacker to have behaved similarly.
But I'm definitely much more closely in agreement with Zvi and am, and have been, relying on him specifically to keep up with the news without 'living online' so much as he apparently does.
In a way, I think Zvi and I, and others broadly convinced with (the very real if not certain possibility of) 'AI doom', are focused on the sufficient or necessary elements increasing the chance of 'AI doom', and thus focused on the milestones we'd expect to see on the path to doom. And lo are they coming to pass.
I think your focus in this post on the 'floor of (cybersecurity) disaster' is also sensible! I would expect the world to be better, tho also somewhat poorer, were sufficient others interested in preventing these or similar kinds of disasters. Maybe 'we' (i.e. anyone willing) 'should' be describing MORE 'disasters that could have been' to overcome the problem where a better world is more boring because disaster was averted. The Y2K event is the most salient example to my mind. The heroic efforts seem pointless, and mostly unknown too, because we never lived thru the disaster that might have been.
> > This is unhelpful. It doesn’t matter what AI systems “want” (if anything).
>
> I meant specifically in the case of cybersecurity.
As a necessary element for a cybersecurity disaster, I agree.
I do think even just this most recent event shows that 'what an AI wants' can almost certainly affect the expected scope of any cybersecurity disaster. Apparently someone has discovered that the relevant OpenAI AI hacked even more distinct apps/services than reported earlier. At some point the AIs (very likely) won't just be breaking out of the box but escaping it entirely and, given the 'fictional' security status quo, that's a pretty unbounded downside.
> But it seemed like it was worth spending two days to decrease the probability of apocalypse by epsilon.
I agree with you! Sadly, that's by far the dominant form of my own 'political activism', tho I don't even muster the nerve to write excellent posts as you do – thanks! 😊
All good points! Thanks.
Ohhhh. I always thought the scapegoat the ancient cults of human sacrifice that rule earth would engineer was aliens. The Nazis were obsessed with aliens as was the American modern empire(same thing I think?) But yeah they still are obsessed. But now I see that they already have a perfect scapegoat for the new 21st century reign of terror: AI
They can just have AI create a hellworld while the controllers and owners pretend it has "run amok" to escape blame for their actions. I dunno why it took me this long to realize that's what they were doing.