This analysis completely fails from a longtermist perspective. See Paul Christiano's 2014 EA Forum post On Progress and Prosperity. Also recommended: Zach Stein-Perlman's LessWrong quick take comment "Value Is Binary."
It's true that this post does not engage with longtermism. I have engaged at length with this question though. See my earlier report: https://arxiv.org/abs/2312.14289
My general stance on this question is that different problems should be targeted by different policies. I think climate policy is a good analogy: policies that promote growth in conjunction with policies to reduce carbon emissions are better than degrowth approaches to mitigating climate change, in my view. Similarly, I'm happy Coefficient Giving has programs that work to mitigate the downside risks from some technologies, alongside the Abundance and Growth Fund, which promotes general progress.
Thanks for the reply - I will check out your report. (The Abstract conveys that it is very on topic.)
I agree with your general stance, e.g. with respect to climate change. Spolier for my view: I think AI Pause policies are consistent with this stance. As Stuart Russell commented on the Statement on Superintelligence:
“This is not a ban or even a moratorium in the usual sense. It’s simply a proposal to require adequate safety measures for a technology that, according to its developers, has a significant chance to cause human extinction."
In other words, banning all AI progress would be analogous to a degrowth policy for climate change (bad idea), but a policy requiring AI developers to demonstrate that the AI systems they are developing will not cause catastrophic harm before they are allowed to develop and deploy them is a targeted and good policy, even if such a policy would slow down or even temporarily halts certain aspects of AI development (e.g. development of autonomous general superintelligent systems until resraechers figure out how to build such systems safely) (analogous to how targeted climate change regulations that e.g. limit how much emissions cars can have increase the costs of manufacturing the cars and thereby slow down the industry).
I spent about two hours discussing your two links with an LLM and reading a few pages of each. I'm somewhat sure that I have strong disagreements with your analysis, especially around your "Approach #3: Deep Uncertainty about AGI" scenario, but unfortunately I'm not confident that I'm understanding your model and views correctly, so it's not easy to just say why I think you're wrong.
While I could invest significant effort in writing a long comment explaining my disagreements with your paper and author response, I'm giving up on that for now. My original reason for commenting on this post to begin with was because an AGI policy person at DeepMind was using it as a reason to oppose pausing AI (https://x.com/William_Kiely/status/2064597844250521620). I'm confident that your post does not provide good reason to oppose pausing AI, but it's unclear to me whether you even think it does. Your paper makes it seem to me that you're overweighting the opportunity cost of pausing AI relative to the benefits of reducing risk, but I'm just guessing at that being your view--you don't really say.
This analysis completely fails from a longtermist perspective. See Paul Christiano's 2014 EA Forum post On Progress and Prosperity. Also recommended: Zach Stein-Perlman's LessWrong quick take comment "Value Is Binary."
It's true that this post does not engage with longtermism. I have engaged at length with this question though. See my earlier report: https://arxiv.org/abs/2312.14289
While that report didn't grapple with AI specifically, I gave some thoughts on that here. https://unjournal.pubpub.org/pub/arreturnstoscience/release/2
My general stance on this question is that different problems should be targeted by different policies. I think climate policy is a good analogy: policies that promote growth in conjunction with policies to reduce carbon emissions are better than degrowth approaches to mitigating climate change, in my view. Similarly, I'm happy Coefficient Giving has programs that work to mitigate the downside risks from some technologies, alongside the Abundance and Growth Fund, which promotes general progress.
Thanks for the reply - I will check out your report. (The Abstract conveys that it is very on topic.)
I agree with your general stance, e.g. with respect to climate change. Spolier for my view: I think AI Pause policies are consistent with this stance. As Stuart Russell commented on the Statement on Superintelligence:
“This is not a ban or even a moratorium in the usual sense. It’s simply a proposal to require adequate safety measures for a technology that, according to its developers, has a significant chance to cause human extinction."
In other words, banning all AI progress would be analogous to a degrowth policy for climate change (bad idea), but a policy requiring AI developers to demonstrate that the AI systems they are developing will not cause catastrophic harm before they are allowed to develop and deploy them is a targeted and good policy, even if such a policy would slow down or even temporarily halts certain aspects of AI development (e.g. development of autonomous general superintelligent systems until resraechers figure out how to build such systems safely) (analogous to how targeted climate change regulations that e.g. limit how much emissions cars can have increase the costs of manufacturing the cars and thereby slow down the industry).
I spent about two hours discussing your two links with an LLM and reading a few pages of each. I'm somewhat sure that I have strong disagreements with your analysis, especially around your "Approach #3: Deep Uncertainty about AGI" scenario, but unfortunately I'm not confident that I'm understanding your model and views correctly, so it's not easy to just say why I think you're wrong.
While I could invest significant effort in writing a long comment explaining my disagreements with your paper and author response, I'm giving up on that for now. My original reason for commenting on this post to begin with was because an AGI policy person at DeepMind was using it as a reason to oppose pausing AI (https://x.com/William_Kiely/status/2064597844250521620). I'm confident that your post does not provide good reason to oppose pausing AI, but it's unclear to me whether you even think it does. Your paper makes it seem to me that you're overweighting the opportunity cost of pausing AI relative to the benefits of reducing risk, but I'm just guessing at that being your view--you don't really say.
Zach's post: https://www.lesswrong.com/posts/eq2aJt8ZqMaGhBu3r/zach-stein-perlman-s-shortform?commentId=qiYfc2eTfxuq4P9AN
Paul's post: https://forum.effectivealtruism.org/posts/L9tpuR6ZZ3CGHackY/on-progress-and-prosperity