AI will not kill us all

it would not resist or fight us.

• Before we make the decision to turn it off, and before it does anything that would cause us to want to turn it off, an AGI could use the internet and copy itself onto many different servers. You could not give an AGI internet access, but that would substantially limit how useful an AGI is – for example, many creators who’d want to make money with an advanced AI system (e.g. through fast-reacting algorithms on the stock market) would have the incentive to give it access to the internet. 
  
• It could anticipate that we would want to turn it off under particular circumstances, and communicate in ways that cause researchers to give it more time, computing resources, and training data so it can better accomplish its goals. Remember, if an AI system decides that the best way to accomplish its goals is to kill someone, it is going to act in ways that prevent you from blocking that goal. 
  
• In general, as Kelsey Piper puts it, we’re also at the mercy of the least cautious actor. If any government or corporation that has access to an AI system doesn’t employ really strict safety standards, that AI system could then engage in harmful actions. Don’t underestimate the possibility that this is the intention of whoever has access to the system – an AGI could make lethal autonomous weapons, for example, far more destructive, so if a government or non-state actor wanted to engage in maximal destruction, an AGI would allow them to do it more effectively. 
  
• When an AGI is on, it could hack vulnerable systems elsewhere and upload copies of itself onto such systems. 

• AI systems are already becoming a lot more general. Consider GPT-3 and AlphaGo/AlphaZero, for example – the tasks they perform or the types of learning they engage in are significantly less narrow than we’d typically associate with narrow AI. 
  
• The main bottleneck to progress in AI so far has been computational power. The cost of computational power is reducing a lot, and systems are close to reaching the computing power of the human brain. Ajeya Cotra of Open Philanthropy used a biological anchor framework to determine how much computing power and algorithmic advances are required for an AI to be as general as human brains, and forecasts an ~80% chance of transformative artificial intelligence by 2100. 
  
• Most ML and computer science researchers think AGI is possible, and surveys find that they expect human-level artificial intelligence in the next 100 years with consistently >50% probability. 
  

• A hard constraint on it successfully accomplishing that task is whether it survives.
  
• It could interpret the tasks in ways that are catastrophic, because the best way to accomplish a lot of human problems will involve killing or harming lots of people if you don’t have a moral conscience. 

I don't think an AGI can hack. Hacking is very logical and knowledge-based; intelligence alone isn't enough. Without training, nobody can hack, not even an ASI.

Those are fair points. However, I just want to raise three counterpoints:

• AI systems are already becoming a lot more general. Consider GPT-3 and AlphaGo/AlphaZero, for example – the tasks they perform or the types of learning they engage in are significantly less narrow than we’d typically associate with narrow AI. 
  
• The main bottleneck to progress in AI so far has been computational power. The cost of computational power is reducing a lot, and systems are close to reaching the computing power of the human brain. Ajeya Cotra of Open Philanthropy used a biological anchor framework to determine how much computing power and algorithmic advances are required for an AI to be as general as human brains, and forecasts an ~80% chance of transformative artificial intelligence by 2100. 
  
• Most ML and computer science researchers think AGI is possible, and surveys find that they expect human-level artificial intelligence in the next 100 years with consistently >50% probability.