'I'm not sure AI would make the top 50' in betting market on human extinction: Okta CEO

The cybersecurity business is poised to hit a new gear in the wake of fresh worries about powerful AI agents running amok on the internet.

"I think it's a powerful new technology and there's things that could go wrong. And there's things that could go right. And like every new technology, we have to be smart about how we use it," Okta ( OKTA ) co-founder and CEO Todd McKinnon said on Yahoo Finance's Opening Bid (video above). "Every huge new part of our economy is going to be regulated. There's going to be legal challenges. There's going to be people making sure it's safe. That's just naturally how technology works. It's about how we get that balance right. And I think the industry will get there."

Shares of top cybersecurity names such as Okta, CrowdStrike ( CRWD ), and Palo Alto Networks ( PANW ) popped on Monday amid fresh warnings from the biggest names in AI over the past few days. Gains continued to pile on Tuesday.

The advances likely reflect these companies seeing an even faster pace of growth as companies double down on protecting their infrastructure.

Within the course of two days, Anthropic ( ANTH.PVT ) CEO Dario Amodei and OpenAI ( OPAI.PVT ) CEO Sam Altman have managed to scare the hell out of the human race with the powerful technology they created and pushed their teams of pioneering supercoders to supercharge.

"Over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up," Amodei stated in a 3,800-word essay on Saturday. "We must slow the pace at which we improve the capabilities of A.I. models. Progress will still seem fast, and we must make wise use of the time we gain."

Altman said right after the post that he agreed with his rival.

He then followed that up on Monday with a dark X post of his own.

"There are two ways AI progress could go very badly, and that we must avoid. First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities," Altman said in an X post around 12:57 a.m. ET.

"Second, we could end up in a world with too much concentration of power," he continued. "If an extraordinarily powerful AI is used by one person or company to impress their worldview onto everyone else, the results could be extremely dystopian. Avoiding these two threats requires walking a narrow middle path; for example, one country could gain too much power. Another example is one lab ending up with too much power."