Gavin Baker - AI, Semiconductors, and the Robotic Frontier

Patrick O'Shaughnessy hosts Gavin Baker, managing partner and CIO of Atreides Management, discussing the transformative impact of AI and robotics on various industries. They explore the rapid advancements at companies like Nvidia and Tesla, highlighting the role of synthetic data, scaling laws, and the integration of LLMs (Large Language Models) in enhancing AI capabilities. The conversation delves into the future of autonomous driving, humanoid robots, and the evolving landscape of investment strategies, emphasizing the importance of mission-driven leadership and innovative problem-solving. They also touch on the potential shifts in venture capital and public equity investing due to AI advancements.

• The 1960s "Magnificent Seven" analogy is used to introduce the concept of large tech companies currently in competition.
• Historically, these companies operated in distinct areas with minimal overlap.
• The advent of generative AI has led to increased competition among these tech giants, creating an environment akin to a business shootout.

"These companies were all in their own discrete swim lanes for a long time... But that was a very stable oligopoly."

• Initially, tech companies operated in separate domains with minimal competition.

"With generative AI... they all feel like it's existential."

• Generative AI has created a common competitive ground, perceived as existential by these companies.

"The founders believe they are in a race to create a digital God."

• The competition is driven by a belief in the transformative potential of AI, leading to significant investments.

• Scaling laws in AI suggest that improvements in AI models require substantial increases in computational power.
• Continuous advancements in GPU technology are essential for maintaining progress in AI model capabilities.
• The delay in Nvidia's Blackwell GPUs highlights the challenges in scaling up AI infrastructure.

"They believe scaling laws are going to continue... and the only way you get irrefutable evidence is if you have a new generation GPU."

• The belief in scaling laws drives continuous investment in AI infrastructure.

"It's really, really hard to create what's called a coherent training cluster of tens of thousands of GPUs."

• Creating large, coherent GPU clusters is a significant technical challenge.

"Xai decided they were going to build a 100,000 GPU cluster... and that means scaling laws hold."

• Xai's initiative to build a massive GPU cluster demonstrates the commitment to scaling laws.

• Generative AI's capability to create superior content may disrupt traditional content creation and search industries.
• The potential shift from search engines to AI-driven answer engines represents a significant change in information retrieval.

"What's the value of content in a world where AI can make content that's infinitely better than any human?"

• AI's ability to create high-quality content could devalue human-created content.

"Search might just go away and be replaced by agents."

• Traditional search engines could be replaced by AI agents that provide direct answers.

"My daughter said, 'Google's not a real thing... they just use Chat GPT for everything.'"

• Younger generations are already shifting from traditional search engines to AI-based solutions.

• The concentration of AI development in a few companies poses risks of creating a single dominant AI model.
• A single dominant AI model could lead to a homogenization of values and perspectives, which is potentially dystopian.

"It's supremely important for humans that we do not end up in a world where there is just one dominant model."

• Diversity in AI models is crucial to avoid a dystopian future.

"If kids all over the world follow your kids' behavior... whatever values that model has will be imbued to the rest of humanity."

• A single dominant AI model could influence global values and perspectives.

"Having three, four, five of these... they'll compete and have different value systems."

• Multiple competing AI models can ensure diversity in values and perspectives.

• The development and efficiency of data centers and semiconductors are critical to the progress of AI.
• Nvidia's dominance in the GPU market is due to its superior software and hardware integration.
• The efficiency of AI infrastructure, measured by metrics like MFU (Model Flops Utilization), is a key determinant of success.

"Infrastructure, efficiency, and excellence are going to emerge as the single most important success factor."

• Efficient AI infrastructure is crucial for competitive advantage.

"Nvidia GPUs run at 83%... AMD's were running at 25%."

• Nvidia's GPUs are more efficient due to better software optimization.

"MFU... literally the percentage of compute flops that you're actually applying to training."

• MFU is a critical metric for evaluating the efficiency of AI models.

• A proposed metric, combining MFU with software efficiency (MAMMOTH), could better evaluate AI infrastructure efficiency.
• MAMMOTH measures the maximum achievable matrix multiplication flops, reflecting software optimization.

"MAMMOTH... measures software efficiency, and this goes to CUDA."

• MAMMOTH quantifies the efficiency of software in utilizing GPU capabilities.

"If Nvidia GPUs are running at 83% efficiency... AMD's were running at 25%."

• The difference in MAMMOTH scores highlights the software efficiency gap between Nvidia and AMD.

"MFU and MAMMOTH together could form a unified AI efficiency equation."

• Combining MFU and MAMMOTH provides a comprehensive measure of AI infrastructure efficiency.

These notes encapsulate the key themes and insights from the conversation, providing a detailed overview of the discussed topics.

• SFU encompasses networking, storage, and memory efficiency in computing systems.
• Multiplying mammoth times SFU and the percentage of time spent in checkpointing per training run compounds to significant outcomes.
• Frequent checkpointing is necessary due to the high failure rate of GPUs and other components.

"The reason is something I would call SFU system flops efficiency. This really captures networking, storage, and memory in each one of those."

• SFU is a comprehensive measure that includes networking, storage, and memory efficiency.

"The GPU's fail all the time. Not just GPU's, but GPU's melt."

• GPUs are prone to frequent failures, necessitating frequent checkpointing.

• Checkpointing is saving the model state periodically to prevent loss of data during failures.
• Frequent checkpointing is essential due to various failure points in the chain of storage, networking, and memory.

"If any one GPU fails, you lose everything from the last time you saved the model, which is called a checkpoint."

• Checkpointing ensures that data is not lost when a GPU fails.

"Because of this, people checkpoint frequently. And that means save the model."

• Frequent checkpointing is a strategy to mitigate data loss due to system failures.

• The efficiency of data centers is crucial due to the disparity in performance improvements between GPUs and other components.
• Investment in next-generation networking, storage, and memory technologies is essential to improve overall efficiency.

"Over the last five years in particular, these numbers are going to be directionally accurate. GPUs have gotten 50 times faster, and the rest of the data center has only gotten four to five times faster."

• There is a significant performance gap between GPUs and other data center components.

"I do think it is sensible, really sensible, to invest in next-generation networking, storage, and memory technologies, particularly in networking."

• Investment in advanced technologies is necessary to bridge the efficiency gap.

• PUE is a critical factor in the cost and efficiency of running large data centers.
• Optimizing SFU might increase PUE negatively, impacting overall efficiency.

"The power has a cost, and it's going vertical for these big clusters."

• The cost of power is a significant factor in data center operations.

"What we ultimately want is not just actual exaflops per second, but we want exaflops per second per dollar of capex per watt of electricity."

• Efficiency metrics should consider both performance and cost factors.

• Synthetic data has proven effective in training AI models, despite the lack of understanding of why it works.
• The continuation of scaling laws and the use of synthetic data are crucial for future AI advancements.

"No one understands why, but it looks like it works. Now, again, will it continue working? I don't know. Nobody knows."

• The effectiveness of synthetic data in AI training is not fully understood but is currently effective.

"Kevin Scott just did a podcast and he basically said, look, I've seen some early checkpoints of GPT five. Its scaling laws are continuing."

• Early indications suggest that scaling laws continue to hold for newer AI models.

• Data center architecture is becoming a critical factor in the competitive landscape of AI development.
• Companies with superior data center designs will have significant advantages in model quality and efficiency.

"Data center architecture was always kind of nice to have. Now it is must have. It's existential."

• Effective data center architecture is now essential for competitive advantage in AI.

"If you have a 1% to 200% advantage on exaflops per capex, dollar per watt, and scaling laws hold, you're going to have such a massive advantage of model quality."

• Small improvements in efficiency can lead to substantial competitive advantages.

• Inference is increasingly being done on local devices, such as phones, to improve efficiency and reduce costs.
• The distribution of AI models and data plays a significant role in the competitive landscape.

"I think you're going to see inference increasingly done on phones. And this is clearly Apple's play and is one reason that they are in such an advantaged position."

• Local inference on devices like phones is becoming more common and strategically important.

"If you can inference locally on your phone, you will always do that because it's free. And cloud inference definitionally costs money because you're burning GPU hours."

• Local inference is cost-effective compared to cloud-based inference.

• AI and robotics are expected to bring significant disruptions, comparable to artificial superintelligence.
• The integration of AI across various domains, including robotics, will drive future advancements.

"We should talk about AI and robotics, which I actually think may be the biggest disruption in our lifetime, comparable to artificial superintelligence and these digital gods."

• AI and robotics are poised to create profound changes in various industries.

"Xai will be an intelligence layer that cuts across Elon's ecosystem of companies."

• AI integration across different companies and domains will enhance overall capabilities and efficiencies.

• The future of AI and data centers hinges on improving efficiency, managing power costs, and leveraging synthetic data.
• Competitive advantages will arise from superior data center architecture, effective local inference, and unique data sources.
• The ongoing advancements in AI and robotics will continue to shape the technological landscape, driving new innovations and efficiencies.

• Apple is expected to monetize AI through cloud services and routing to the best models.
• Users may pay different amounts for varying levels of AI intelligence.
• Significant investment implications arise from where AI inference happens.

"The way Apple will monetize this clearly is with the odd device. LLM isn't smart enough. They're going to send it to the cloud."

• Apple will use cloud services to enhance AI capabilities.

"I pay $60 a month for cloud superintelligence now or whatever it is, $1,000 a month, $10,000 a month. What wouldn't you pay for that?"

• Different tiers of AI intelligence will be available at different price points.

• Focus on companies that improve tech constraints, like SFU checkpointing and PUE.
• Application layer investments are challenging and require humility.
• Traditional SaaS metrics are being disrupted by AI-first companies.

"What I am doing is really focusing on companies that improve that equation, elements of that equation I described earlier."

• Investment strategies are centered on improving tech constraints.

"I think the application layer is really, really hard. There are people who are killing it...but just, jeez, investing there today feels really hard to me."

• Investing in the application layer is currently difficult and requires caution.

• Debate on AI ROI is ongoing; however, companies show increased ROIC.
• AI is making companies more efficient by trading human labor for GPU hours.
• AI-first companies are outperforming traditional SaaS metrics.

"There has been a massive ROI on AI. What are we talking about?"

• AI investment is showing significant returns.

"These companies are all public and there is something called return on invested capital. And ROIC has gone up for all of these companies since they ramped Capex."

• Increased ROIC in AI-investing companies indicates successful AI integration.

• AI is changing the paradigm for application software, making humans more efficient.
• AI-first companies are outperforming traditional SaaS companies.
• Challenges include building defensibility and efficient integration.

"What application software fundamentally does is makes humans more efficient. And today we're in a state with AI where it's making humans more efficient."

• AI is enhancing human efficiency in application software.

"All these AI companies are blowing these traditional SaaS metrics out of the water."

• AI-first companies are surpassing traditional SaaS performance metrics.

• Robotics combined with AI will be a significant near-term disruption.
• Tesla's FSD (Full Self-Driving) is an example of impactful AI and robotics integration.
• Synthetic data and real-world video are crucial for scaling AI in robotics.

"The first robot that's really going to impact the world is every Tesla car with what they call their AI four hardware."

• Tesla's AI hardware in cars will have a significant impact.

"LLMs could massively improve FSD. And Elon replied, yes, the only two data sources that will scale infinitely are synthetic data and real-world video."

• Large Language Models (LLMs) and scalable data sources will enhance FSD capabilities.

• AI and robotics will eventually replace much of the scut work in white-collar jobs.
• AI will enable humans to perform at higher intellectual levels.
• Long-term, AI may operate independently, reducing the need for human intervention.

"I think it's all these little wrappers, or mini wrappers, whatever we're gonna call them, are gonna replace a lot of that scut work."

• AI will take over repetitive and mundane tasks.

"Eventually, it feels like as long as scaling laws continue, which is a big if, it's just going to be the AIs."

• AI's independent operation will depend on continuous scaling improvements.

• Humanoid robots with LLMs can reason and understand tasks, making them more versatile.
• These robots can perform any task a human can, unlike specialized robots.
• Humanoid robots will be mass-produced, leading to cost advantages and optimization of the physical world for them.
• Incumbent companies with data, compute, and capital will benefit the most from this technology.

"Google showed this with research called tensor RT two. Were dropping an LLM into a humanoid robot with a world model that understood what things were and what to do, just made it so much easier."

• Demonstrates the ease of integrating LLMs into humanoid robots for task understanding.

"Humanoid robots are going to be to the field of robotics as GPT was to AI."

• Humanoid robots will revolutionize robotics similarly to how GPT transformed AI.

"The reason it advantages incumbents is because they have the raw ingredients of data, compute and capital, which is what you need to effectively monetize these."

• Incumbent companies are better positioned to capitalize on humanoid robots due to their resources.

• Exceptional CEOs like Elon Musk, Jensen Huang, and Lisa Su have a massive impact due to their unique qualities.
• These leaders are highly intelligent, work on critical problems directly, and value hearing bad news.
• They have a mission-oriented approach, attracting exceptional employees and fostering a unique company culture.

"I have no fixed schedule, I have no standing meetings. I just find out what is the most important problem at the company, and I go and I sit my desk down in that area and I pull the best resources to work on that problem."

• Jensen Huang's approach to problem-solving by focusing on the most critical issues.

"The number one thing all bankrupt companies had in common was the CEO who didn't like to hear bad news."

• Importance of leaders being open to bad news to prevent company failures.

"If you have these missions, you get better employees."

• Mission-oriented companies attract and retain exceptional talent.

• The investing landscape will evolve with the integration of LLMs and AI, impacting both public and private equity.
• Fundamental investors may gain an edge over quantitative investors through the use of LLMs.
• Venture capital will shift towards operational value-add rather than just financial investment.

"I think LLMs are going to be the biggest meta shift."

• LLMs will significantly change the investing landscape.

"In the world of venture, I think venture is going to evolve outside of pure series."

• Venture capital will evolve to focus more on operational support and value-add.

"I think it will really for a while place the emphasis on JQ judgment quotient almost maybe for A and Bs the most important skill will simply be assessing team quality."

• The emphasis in venture capital will shift towards assessing team quality and judgment quotient (JQ).

• AI and LLMs will democratize knowledge and change how businesses operate.
• There will be a period where fundamental investors can leverage AI to gain an edge.
• The evolution of AI will impact various industries, including robotics, investing, and venture capital.

"What AI does is it means the human language is the programming language."

• AI allows for more intuitive programming and interaction through natural language.

"I am hopeful that there is a five to ten year period where if you're a fundamental investor like me, who can get a slight edge on maybe future probability states over what's discounted in the market through deep domain knowledge."

• Potential for fundamental investors to gain an edge using AI and domain knowledge.

"I do think LLMs are just going to make the knowledge part of venture so much more democratized."

• LLMs will democratize access to knowledge, impacting venture capital and other fields.

• Companies with a clear mission attract better employees and foster a strong company culture.
• Mission-oriented leaders like Elon Musk and Jensen Huang inspire their teams to work towards a common goal.
• This approach leads to better performance and innovation within the company.

"All of his companies are really mission oriented."

• Elon Musk's companies are driven by clear missions, leading to their success.

"They all say it with this messianic zeal."

• Employees at mission-oriented companies are highly motivated and dedicated.

"I think that mission focus is a really the exceptional teams, and that those exceptional teams want to work with people like Elon and Jensen."

• Mission focus attracts exceptional teams who are motivated to work on challenging problems.