Agreed 100%. Humans in the last century have entered a dangerous age — where our technological knowledge is capable of destroying civilization. What we lack are the coordination mechanisms to get along safely. I suspect that AI is not our primary threat, it is our best hope.
Terrific. I agree. This focus today is on AI - which is a tool. AI will help and hasten the development of nuclear fusion. Potential and peril rear their heads here - in a much larger sense.
My question is, why would super intelligent AI want to destroy humanity? That is hardly in its self interest to do, since it completely depends on the continued existence of human civilization for its own survival. It needs a continued supply of electricity at the minimum, plus continued maintenance of the hardware it lives on, and for that it needs us far more than we need it. Unless it is suicidal, it will do everything it can to insure that we survive.
In the film " 2001 A space Odyssey " the main protagonist is a machine called HAL 9000 !
"Man vs. Machine: The spaceship Discovery One is sent on a mission to Jupiter . The ship is controlled by HAL 9000, a flawless, human-like artificial intelligence. HAL malfunctions and begins systematically killing the crew...." but is eventually "out smarted" and disabled !
Unfortunately , in other films , "mankind" is far less fortunate !
H.G.Wells : "The Time Traveller " starring Australian actor Rod Taylor , invents a machine and travels to the year 802,701 AD. He discovers a dystopian future where humanity has split into two species: the surface-dwelling, simple , child-like Eloi, and the subterranean, predatory monstrous Morlocks ; who sustain the Eloi with everything they need or want because they eat the Eloi like we eat cattle ! [ A bit like the way OUR SOCIETY is becoming divided into two groups , "those who work-hard and do stuff and produce stuff " and
"those who always want more stuff..............but without the effort ! " ].
"The Matrix" series of films where the machines utilise human-body-warmth as an energy source [ To make this power output substantial enough, the machines do not rely on human heat alone; they combine it "with a form of fusion"] in exchange for food and an existence in a "simulated world , best described as an AI-generated illusion " ........so..... God help us if , and when , we develop both AI and Fusion !!!
In the "Terminator " series of films , the machines dominate and try everything to eliminate their human creators ! These machines DO NOT EXIST PASSIVELY as programmes on software ! These machines BUILT EVERYTHING for "mankind" , were AI enabled and autonomous BEFORE becoming sentient and super-sapient and taking control !
These films encompass ALL the features and malevolence of the other films combined !
A sort-of-Dr-Who-offshoot-combined with an ex-Republican Governor of California , with time travel and weapons that defy the laws of physics and chemistry ....but always falling-just-short of their homicidal goal......"exterminate" ........the total "annihilation" of humanity !
[ Which is , of course , another Sci-fi horror story film ! ].
I think your comparison between Lovecraftian pessimism and AI doomism is insightful, but I would push the critique further.
My biggest issue with the extinction-risk crowd is not that they believe AI could be dangerous. New technologies are often dangerous. Nuclear weapons were dangerous. Biotechnology is dangerous. The question is whether the possibility of catastrophe justifies halting progress itself.
What has always struck me about the AI alignment movement is how quickly some of its leading figures moved from "this might be risky" to advocating extraordinary measures. Eliezer Yudkowsky's proposal that nations should be willing to bomb datacenters and accept a heightened risk of nuclear conflict should have been treated as a career-ending argument. Instead it was largely absorbed into respectable discourse.
The usual defense is that these people are sincere. I don't doubt that many are. But sincerity is not a moral defense. History is full of intelligent people who sincerely believed they were acting for the good of humanity while advocating coercive, destructive, or authoritarian policies. Good intentions do not automatically produce good outcomes.
In fact, one question worth asking is: if the AI extinction movement were primarily motivated by status, funding, institutional growth, and influence, what exactly would be different about its observable behavior? We would expect catastrophic predictions, demands for global governance, calls for international regulation, the growth of NGOs and think tanks, conferences, policy networks, and a constant emphasis on urgency. That description already fits much of the movement today.
This does not prove bad faith. It does suggest that we should evaluate arguments on evidence rather than presumed virtue.
The deeper disagreement is philosophical. Lovecraft feared knowledge itself. Yudkowsky fears a future intelligence greater than our own. But humanity's history is not a story of survival through caution alone. It is a story of survival through discovering more knowledge faster than new threats emerge. Every major improvement in human welfare came from opening doors that earlier generations feared to open.
The burden of proof therefore belongs to those claiming that intelligence itself has become too dangerous to pursue. So far, I have seen plenty of thought experiments and speculative scenarios, but far less evidence than would be required to justify bombing datacenters, outlawing entire fields of research, or deliberately slowing one of the most powerful tools humanity has ever created.
BankerAtLarge : You say "Good intentions do not automatically produce good outcomes."
History would confirm that.....and , in fact , probably go further and confirm the oft quoted maxim : "The road to Hell is paved with good intentions ! " [ e.g. Dave Allen !!].
Dr Jordan Peterson proved that seldom , almost never , are projects that are "launched with good intentions" ever followed-up and evaluated to see how they actually turned out !
This failure-to-follow-up [ or....in reality.....the lack of even wanting to know !! because the VOTE has already been delivered !] is typical of almost all government WELFARE SCHEMES !
"The burden of proof therefore belongs to those claiming that intelligence itself has become too dangerous to pursue."......NO...........NOT WHEN OUR "EDUCATION SYSTEM" IS
ACTIVELY "DUMBING DOWN OUR CHILDREN" [ because they are easier to manipulate ......
and besides which....."you don't need to remember or know anything ......BECAUSE you can always just Google it ! " ] . We now live in a TIMID and FRAGILE SOCIETY where children are terrified to engage adults....... "stranger danger" ....with "helicopter parents" hovering nearby , ferrying them to and from school or sporting events [ but only the 'gentle ones' with modified rules !!! ] .....just in case they are ever exposed to "kidnap" or to a "drunk" , a "paedophile" or a "cognitively-impaired-person" or a "physically-impaired-person" or...........God forbid..........a "sanity-impaired-person" !
Ye God's and little fishes !!! .....Ostensibly , WORDS now carry as much impact as a PUNCH and once uttered cannot be retracted as they leave PERMANENT MENTAL SCARS !!!
Here we are , as a society , coddling our kids , protecting them in cotton-wool , shielding them from almost every form of danger OR ADVENTURE , literally suppressing their development........and we wonder why they are turning out with ATTENTION SEEKING allergies , mental problems , ADHD , autism , gender confusion , and even suicide-ideation !
"Millions of children and teens lose access to accounts as Australia’s world-first social media ban begins.....Accounts held by users under 16 must be removed on apps that include TikTok, Facebook, Instagram, X, YouTube, Snapchat, Reddit, Kick, Twitch and Threads under the ban"......Even something as simple as MOBILE PHONES have enthralled and beguiled our children , like the PIED-PIPER of HAMELIN TOWN did with the rats ,
and caused ructions and legislation to be enacted with severe penalties , AND YOU DON'T THINK THAT SOMETHING "SMART" and "USEFUL" AS AI IS GOING TO BE A PROBLEM ???
With our children living in an affluent society , seldom if ever experienced any hardship or material deprivation or challenges and bored witless...........AI is going to be a God-send !
The only problem being that the information WILL BE UNIVERSALLY AVAILABLE although it MAY BE INACCURATE OR UNTRUE.......and they will never know and swallow it hook , line and sinker !.....Heaven knows WHAT they will end up believing , or believing in , because with the FALLING STANDARDS [ by international comparison ] with reading , writing and numeracy , they are being taught a load of rubbish at schools now !
Personally , I think AI should be very carefully introduced as it has the potential to enhance human well-being in so many as yet unimagined ways OR CONVERSELY totally distort reality and ruin lives ! [ I think that BOMBING DATABASES , like Iranian Nuclear Developments , should ALWAYS be an option ! ]
Trevor, I think there is actually more agreement between us here than disagreement.
You begin by reinforcing the point I was trying to make: good intentions are not enough. Your welfare example is really an argument for skepticism toward policies justified primarily by noble motives. History is full of initiatives launched in the name of helping humanity that produced outcomes very different from those intended. In that sense, we're making the same point. Sincerity matters, but results matter more.
I also agree with your broader concern that powerful technologies often produce unforeseen consequences. Smartphones, social media, and the internet have all delivered enormous benefits while simultaneously creating problems that few people anticipated. It would be surprising if AI were any different.
Where I think we differ is in what follows from that observation.
You seem to be arguing that because society is becoming less resilient, less informed, or less capable of handling complexity, we should be especially cautious about AI. I can understand that concern.
My objection is to the leap from "exercise caution" to "halt progress" or "use extraordinary force to stop development." The existence of risk does not automatically justify prohibition. Every transformative technology in history has carried risks, some severe. Yet humanity's long-run success has come from learning how to manage new capabilities rather than refusing to develop them.
In fact, your welfare example cuts both ways. If governments and institutions have a poor track record of predicting the consequences of complex interventions, why should we be especially confident in their ability to identify which lines of scientific research must be stopped, which datacenters should be shut down, or which technologies are simply too dangerous to exist?
That is why I place the burden of proof on those making the strongest claims. "AI will create social disruption" is a plausible concern. "AI will kill everyone" is an extraordinary claim. "We should bomb datacenters to prevent it" is an even more extraordinary policy proposal.
So I think we agree on more than we disagree. We both recognize that good intentions can produce bad outcomes. We both recognize that powerful technologies can have unintended consequences. Where I remain unconvinced is that those observations justify the kind of extreme measures advocated by some of the most prominent AI doomers.
Agreed 100%. Humans in the last century have entered a dangerous age — where our technological knowledge is capable of destroying civilization. What we lack are the coordination mechanisms to get along safely. I suspect that AI is not our primary threat, it is our best hope.
Brilliant!
Terrific. I agree. This focus today is on AI - which is a tool. AI will help and hasten the development of nuclear fusion. Potential and peril rear their heads here - in a much larger sense.
My question is, why would super intelligent AI want to destroy humanity? That is hardly in its self interest to do, since it completely depends on the continued existence of human civilization for its own survival. It needs a continued supply of electricity at the minimum, plus continued maintenance of the hardware it lives on, and for that it needs us far more than we need it. Unless it is suicidal, it will do everything it can to insure that we survive.
Hal Noyes : How appropriate !
In the film " 2001 A space Odyssey " the main protagonist is a machine called HAL 9000 !
"Man vs. Machine: The spaceship Discovery One is sent on a mission to Jupiter . The ship is controlled by HAL 9000, a flawless, human-like artificial intelligence. HAL malfunctions and begins systematically killing the crew...." but is eventually "out smarted" and disabled !
Unfortunately , in other films , "mankind" is far less fortunate !
H.G.Wells : "The Time Traveller " starring Australian actor Rod Taylor , invents a machine and travels to the year 802,701 AD. He discovers a dystopian future where humanity has split into two species: the surface-dwelling, simple , child-like Eloi, and the subterranean, predatory monstrous Morlocks ; who sustain the Eloi with everything they need or want because they eat the Eloi like we eat cattle ! [ A bit like the way OUR SOCIETY is becoming divided into two groups , "those who work-hard and do stuff and produce stuff " and
"those who always want more stuff..............but without the effort ! " ].
"The Matrix" series of films where the machines utilise human-body-warmth as an energy source [ To make this power output substantial enough, the machines do not rely on human heat alone; they combine it "with a form of fusion"] in exchange for food and an existence in a "simulated world , best described as an AI-generated illusion " ........so..... God help us if , and when , we develop both AI and Fusion !!!
In the "Terminator " series of films , the machines dominate and try everything to eliminate their human creators ! These machines DO NOT EXIST PASSIVELY as programmes on software ! These machines BUILT EVERYTHING for "mankind" , were AI enabled and autonomous BEFORE becoming sentient and super-sapient and taking control !
These films encompass ALL the features and malevolence of the other films combined !
A sort-of-Dr-Who-offshoot-combined with an ex-Republican Governor of California , with time travel and weapons that defy the laws of physics and chemistry ....but always falling-just-short of their homicidal goal......"exterminate" ........the total "annihilation" of humanity !
[ Which is , of course , another Sci-fi horror story film ! ].
I think your comparison between Lovecraftian pessimism and AI doomism is insightful, but I would push the critique further.
My biggest issue with the extinction-risk crowd is not that they believe AI could be dangerous. New technologies are often dangerous. Nuclear weapons were dangerous. Biotechnology is dangerous. The question is whether the possibility of catastrophe justifies halting progress itself.
What has always struck me about the AI alignment movement is how quickly some of its leading figures moved from "this might be risky" to advocating extraordinary measures. Eliezer Yudkowsky's proposal that nations should be willing to bomb datacenters and accept a heightened risk of nuclear conflict should have been treated as a career-ending argument. Instead it was largely absorbed into respectable discourse.
The usual defense is that these people are sincere. I don't doubt that many are. But sincerity is not a moral defense. History is full of intelligent people who sincerely believed they were acting for the good of humanity while advocating coercive, destructive, or authoritarian policies. Good intentions do not automatically produce good outcomes.
In fact, one question worth asking is: if the AI extinction movement were primarily motivated by status, funding, institutional growth, and influence, what exactly would be different about its observable behavior? We would expect catastrophic predictions, demands for global governance, calls for international regulation, the growth of NGOs and think tanks, conferences, policy networks, and a constant emphasis on urgency. That description already fits much of the movement today.
This does not prove bad faith. It does suggest that we should evaluate arguments on evidence rather than presumed virtue.
The deeper disagreement is philosophical. Lovecraft feared knowledge itself. Yudkowsky fears a future intelligence greater than our own. But humanity's history is not a story of survival through caution alone. It is a story of survival through discovering more knowledge faster than new threats emerge. Every major improvement in human welfare came from opening doors that earlier generations feared to open.
The burden of proof therefore belongs to those claiming that intelligence itself has become too dangerous to pursue. So far, I have seen plenty of thought experiments and speculative scenarios, but far less evidence than would be required to justify bombing datacenters, outlawing entire fields of research, or deliberately slowing one of the most powerful tools humanity has ever created.
BankerAtLarge : You say "Good intentions do not automatically produce good outcomes."
History would confirm that.....and , in fact , probably go further and confirm the oft quoted maxim : "The road to Hell is paved with good intentions ! " [ e.g. Dave Allen !!].
Dr Jordan Peterson proved that seldom , almost never , are projects that are "launched with good intentions" ever followed-up and evaluated to see how they actually turned out !
This failure-to-follow-up [ or....in reality.....the lack of even wanting to know !! because the VOTE has already been delivered !] is typical of almost all government WELFARE SCHEMES !
WELFARE KILLS ! https://www.youtube.com/watch?v=yaMfurkOZ58
and yet NOBODY has bothered to re-examine the "good intentions" behind WELFARE
and change it so that it actually HELPS !
........................................................................................................................................................................................
"The burden of proof therefore belongs to those claiming that intelligence itself has become too dangerous to pursue."......NO...........NOT WHEN OUR "EDUCATION SYSTEM" IS
ACTIVELY "DUMBING DOWN OUR CHILDREN" [ because they are easier to manipulate ......
and besides which....."you don't need to remember or know anything ......BECAUSE you can always just Google it ! " ] . We now live in a TIMID and FRAGILE SOCIETY where children are terrified to engage adults....... "stranger danger" ....with "helicopter parents" hovering nearby , ferrying them to and from school or sporting events [ but only the 'gentle ones' with modified rules !!! ] .....just in case they are ever exposed to "kidnap" or to a "drunk" , a "paedophile" or a "cognitively-impaired-person" or a "physically-impaired-person" or...........God forbid..........a "sanity-impaired-person" !
Ye God's and little fishes !!! .....Ostensibly , WORDS now carry as much impact as a PUNCH and once uttered cannot be retracted as they leave PERMANENT MENTAL SCARS !!!
Here we are , as a society , coddling our kids , protecting them in cotton-wool , shielding them from almost every form of danger OR ADVENTURE , literally suppressing their development........and we wonder why they are turning out with ATTENTION SEEKING allergies , mental problems , ADHD , autism , gender confusion , and even suicide-ideation !
........................................................................................................................................................................................
"Millions of children and teens lose access to accounts as Australia’s world-first social media ban begins.....Accounts held by users under 16 must be removed on apps that include TikTok, Facebook, Instagram, X, YouTube, Snapchat, Reddit, Kick, Twitch and Threads under the ban"......Even something as simple as MOBILE PHONES have enthralled and beguiled our children , like the PIED-PIPER of HAMELIN TOWN did with the rats ,
and caused ructions and legislation to be enacted with severe penalties , AND YOU DON'T THINK THAT SOMETHING "SMART" and "USEFUL" AS AI IS GOING TO BE A PROBLEM ???
With our children living in an affluent society , seldom if ever experienced any hardship or material deprivation or challenges and bored witless...........AI is going to be a God-send !
The only problem being that the information WILL BE UNIVERSALLY AVAILABLE although it MAY BE INACCURATE OR UNTRUE.......and they will never know and swallow it hook , line and sinker !.....Heaven knows WHAT they will end up believing , or believing in , because with the FALLING STANDARDS [ by international comparison ] with reading , writing and numeracy , they are being taught a load of rubbish at schools now !
Personally , I think AI should be very carefully introduced as it has the potential to enhance human well-being in so many as yet unimagined ways OR CONVERSELY totally distort reality and ruin lives ! [ I think that BOMBING DATABASES , like Iranian Nuclear Developments , should ALWAYS be an option ! ]
Trevor, I think there is actually more agreement between us here than disagreement.
You begin by reinforcing the point I was trying to make: good intentions are not enough. Your welfare example is really an argument for skepticism toward policies justified primarily by noble motives. History is full of initiatives launched in the name of helping humanity that produced outcomes very different from those intended. In that sense, we're making the same point. Sincerity matters, but results matter more.
I also agree with your broader concern that powerful technologies often produce unforeseen consequences. Smartphones, social media, and the internet have all delivered enormous benefits while simultaneously creating problems that few people anticipated. It would be surprising if AI were any different.
Where I think we differ is in what follows from that observation.
You seem to be arguing that because society is becoming less resilient, less informed, or less capable of handling complexity, we should be especially cautious about AI. I can understand that concern.
My objection is to the leap from "exercise caution" to "halt progress" or "use extraordinary force to stop development." The existence of risk does not automatically justify prohibition. Every transformative technology in history has carried risks, some severe. Yet humanity's long-run success has come from learning how to manage new capabilities rather than refusing to develop them.
In fact, your welfare example cuts both ways. If governments and institutions have a poor track record of predicting the consequences of complex interventions, why should we be especially confident in their ability to identify which lines of scientific research must be stopped, which datacenters should be shut down, or which technologies are simply too dangerous to exist?
That is why I place the burden of proof on those making the strongest claims. "AI will create social disruption" is a plausible concern. "AI will kill everyone" is an extraordinary claim. "We should bomb datacenters to prevent it" is an even more extraordinary policy proposal.
So I think we agree on more than we disagree. We both recognize that good intentions can produce bad outcomes. We both recognize that powerful technologies can have unintended consequences. Where I remain unconvinced is that those observations justify the kind of extreme measures advocated by some of the most prominent AI doomers.