Chetan Puttagunta and Modest Proposal - Capital, Compute & AI Scaling

Patrick O'Shaughnessy hosts a discussion with Chetan Puttigunta of Benchmark and the anonymous investor Modest Proposal on the evolving landscape of AI, particularly focusing on the shift from pre-training to test-time compute in AI development. They explore how this transition could democratize AI, affecting both public and private markets. The conversation highlights the potential for smaller teams to compete with major labs due to advances in open-source models like Meta's Llama. They also discuss the implications for large tech companies like Google and AWS, the economic viability of AI investments, and the potential for artificial general intelligence (AGI) in the near future.

• AI labs have reached a plateau in scaling through pre-training due to limits in available human-generated text data.
• Transitioning to test time compute involves the AI model generating potential solutions and verifying them iteratively, which is a new scaling paradigm.
• Test time compute could democratize AI development, allowing smaller teams to compete with larger labs.

"All the labs have hit some kind of plateauing effect on how we perceive scaling for the last two years, which was specifically in the pre-training world."

• AI models have exhausted the potential of scaling through pre-training, necessitating a shift in approach.

"We're now shifting to a new paradigm called test time compute."

• Test time compute represents a shift from pre-training to reasoning and inference time, allowing AI to solve problems by exploring multiple solutions simultaneously.

• Test time compute may not scale linearly with increased compute, and tasks can be complex and ambiguous.
• The switch to test time compute aligns revenue generation with expenditures, benefiting financial scalability for tech companies.
• This shift could require rearchitecting network design, impacting power utilization and grid design.

"You quickly realize that algorithms used for test time compute might exhaust the useful search space for solutions quite quickly."

• The effectiveness of test time compute is limited by the algorithms' ability to explore solution spaces efficiently.

"In a test time compute scaling world, you are now aligning your expenditures with the underlying usage of the model."

• Financial efficiency is improved as expenditures are tied to actual model usage rather than upfront costs.

• AI's influence extends beyond tech companies, affecting industrials, utilities, and a significant portion of the market cap.
• Public market investing involves scenario analysis, and the shift to test time compute changes the distribution of outcomes for companies.

"If you look at the cumulative market cap that is a direct play on artificial intelligence right now, it's enormous."

• AI's market impact is substantial, with many sectors being directly influenced by AI advancements.

"If you do see this shift towards inference time, I think that you need to start to think about how do you re-architecture the network design."

• The shift to test time compute could necessitate changes in network architecture, affecting various investment narratives.

• Small teams can now compete in AI development due to open-source models and reduced compute requirements.
• Meta's open-source models, like LLAMA, have enabled smaller teams to reach the frontier of AI performance with less capital.

"We're seeing these small teams of quite literally two to five people jumping to the frontier with spend that is not one order, but multiple orders of magnitude less."

• Small teams can achieve significant AI advancements with minimal resources, challenging larger incumbents.

"The incredible proliferation of open-source models... has been an extraordinary force here."

• Open-source models have democratized AI development, allowing for innovation beyond large labs.

• Open-source models benefit cloud providers like AWS by offering tools for developers to build new products.
• The focus on test time compute aligns with providing generative AI as a tool rather than developing all-powerful models.

"AWS... view clearly has been that LLMs are just another tool that they can provide their enterprise customers."

• Cloud providers see AI as a tool for innovation, aligning with the shift to test time compute.

"If instead the focus of the industry is at test time, at inference time... I think that again re-engineers and re-architects the entire vision."

• The industry shift to test time compute could redefine cloud computing and AI tool deployment strategies.

• Discussion on the potential shift in investment focus towards model companies, not just application companies.
• Meta's development of Llama 4 is highlighted, with a significant increase in computational resources.
• The importance of standardizing on Llama's architecture for new model companies is emphasized.
• The capital-intensive nature of model training is outlined, with a focus on application development as a strategic investment.

"In Meta's last earnings call, Mark Zuckerberg talked about them starting Llama 4 development and he said that Llama 4 is being trained on a bigger cluster than anything he's ever seen out there."

• Meta is significantly investing in computational resources for Llama 4, highlighting the scale and ambition of their AI projects.

"Starting with Llama today is not only great because Llama is open source, it's also extraordinarily efficient because the entire ecosystem is standardizing on that architecture."

• Llama's open-source nature and efficiency make it an attractive foundation for new model companies, indicating a trend towards standardization in AI development.

• Analysis of OpenAI, Anthropic, and XAI's strategic positioning and challenges.
• OpenAI's focus on consumer mindshare and profitability, with concerns about competition from Google and Meta.
• Anthropic's technical talent and challenges in consumer and enterprise markets.
• XAI's potential challenges with scaling and differentiation.

"A lot of it oriented around the idea that they had escape velocity on the consumer side and that ChatGPT was now the cognitive reference."

• OpenAI's strategy heavily relies on consumer recognition and brand strength, positioning ChatGPT as a key reference in AI.

"If OpenAI elects to declare that AGI is achieved, I think then you have a very interesting dynamic between them and Microsoft."

• The potential declaration of AGI by OpenAI could significantly impact its relationship with Microsoft, affecting strategic dynamics.

• Discussion on the proximity of AGI and its potential impact on economically valuable work.
• Examples of end-to-end task completion as indicators of approaching AGI.
• The gradual acceptance of advanced AI capabilities, likened to the boiling frog metaphor.

"AGI as narrowly defined, or maybe expansively defined, depending on your viewpoint, is a highly autonomous system that surpasses human performance in economically valuable work."

• AGI is defined as a system surpassing human performance in economically valuable tasks, suggesting its imminent arrival.

"We sort of become the frog boiling in water, where we pass the Turing test pretty easily, and yet nobody sits here anymore and talks about, holy crap, we passed the Turing test."

• The gradual acceptance of AI advancements is compared to the boiling frog metaphor, indicating a shift in perception towards AI capabilities.

• Examination of Google's capabilities and potential in AI, considering their historical innovations.
• Discussion on AWS's strategic positioning in providing resilient cloud services.
• The importance of consumer touchpoints and enterprise reliability in AI development.

"Google is so fascinating because...it's clear they were caught off guard by the brute force scaling of the transformer."

• Despite Google's historical innovations, they were surprised by the rapid scaling of transformer models, highlighting the dynamic nature of AI development.

"AWS has the biggest cloud. It has really built for resilience in a way that's very very differentiated."

• AWS's extensive cloud infrastructure and focus on resilience provide a strategic advantage in supporting AI development and deployment.

• The need for significant disruptions to unlock new distribution channels and create value.
• The role of AI-powered applications in transforming enterprise sales and procurement processes.
• The rapid adoption and deployment of AI solutions compared to traditional SaaS models.

"You need big disruptions as a venture investor to unlock distribution."

• Significant disruptions are necessary for unlocking new distribution channels and creating substantial investment opportunities.

"It's about we can eliminate significant amounts of software spend and human capital spend and move this to this AI solution."

• AI solutions offer substantial cost savings and efficiency improvements, transforming traditional enterprise procurement and sales processes.

• The pace of investment in AI companies has increased significantly, comparable to past technology booms like the App Store in 2009 and the Internet in the mid-90s.
• A $500 million fund with five partners has already made 25 investments in AI, focusing heavily on application companies over infrastructure.
• The application layer is witnessing significant innovation and distribution unlocks, providing startups with an advantage over incumbent software vendors.

"We've made 25 investments in AI companies and for a $500 million fund with five partners, that's an extraordinary pace."

• The pace of investment highlights the rapid growth and interest in AI technologies, emphasizing the current technological boom.

"We've clearly bet and anticipated there's dramatic innovation and distribution unlock happening at the application layer."

• The focus on the application layer indicates a strategic investment approach, expecting significant advancements and opportunities.

• Incumbent software vendors face challenges due to the innovators' dilemma, struggling to adapt to rapid changes in AI.
• Enterprises may find it difficult to pause operations for a complete re-architecture, hindering their ability to respond to AI advancements.

"It's not that the incumbent software vendors are standing still, it's just that innovators dilemma in enterprise software is playing out much more aggressively."

• The innovators' dilemma describes the struggle of established companies to adapt quickly to disruptive technologies.

"Could a giant SaaS company just pause selling for two years and completely re-architect their application stack? Sure, but I just don't see that happening."

• The inability to pause and re-architect highlights the challenges incumbents face in keeping up with rapid technological advancements.

• AI software spending is experiencing exponential growth, with an 8x increase expected from 2023 to 2024.
• Many AI application companies are supply constrained rather than demand constrained, indicating strong market demand.

"It's 8x year over year growth between 2023 and 2024 on just pure spend."

• The significant growth in AI software spending underscores the increasing adoption and investment in AI technologies.

"More of these companies are supply constrained than demand constrained."

• The supply constraint highlights the strong demand and opportunity in the AI market, suggesting a favorable environment for growth.

• The stabilization of the model layer is crucial for application developers, allowing them to confidently invest in infrastructure and UI improvements.
• Developers are now more willing to invest in new technologies due to the reasoning paradigm, which helps optimize performance and quality.

"If reasoning is now the new paradigm...I'm building technology and tooling that model companies are very, very unlikely to build."

• The reasoning paradigm offers a strategic advantage for application developers, enabling them to create unique and defensible solutions.

"The kind of performance gains you're going to see out of our systems is going to be huge."

• The anticipated performance gains highlight the potential for significant improvements and innovations in application development.

• AI is being applied in diverse fields such as sales automation, legal work, accounting, game development, and circuit board design.
• AI solutions are proving to be more efficient and profitable, transforming traditional business models and operations.

"We've got a great company called 11X that's going after sales automation."

• Sales automation is one example of how AI is revolutionizing traditional business processes, improving efficiency and effectiveness.

"Actually, lawyers end up becoming way more profitable by using AI."

• AI's impact on the legal industry demonstrates its potential to enhance profitability and efficiency, even in traditionally billable-hour-based professions.

• The shift from spending on model pre-training to inferencing is improving ROI for AI investments.
• The economic reality of AI is becoming more favorable as inferencing costs decrease and utilization increases.

"If we are plateauing and we're spending less money on pre-training and moving that capital towards inferencing...this spend is warranted."

• The shift in spending priorities supports a more sustainable and profitable model for AI investments.

"The cost of inferencing is plummeting, the utilization is soaring."

• The reduction in inferencing costs and increased utilization highlight the economic benefits and potential for revenue growth in AI.

• The market is experiencing high valuations for AI companies, driven by optimism and the potential for rapid growth.
• The dramatic drop in compute costs is enabling more efficient and cost-effective AI application development.

"The cost of inference of these models is down a hundred X, 200 X."

• The steep decline in inference costs is a game-changer for AI development, making it more accessible and profitable.

"It's a great time to be in the application development business."

• The current market conditions and technological advancements create a favorable environment for application developers.

• The focus is shifting towards test-time compute, with implications for network design and infrastructure.
• The utilization of GPUs and the efficiency of hyperscalers are critical factors in the evolving AI landscape.

"I think inference overtakes training much faster than we thought and gets much bigger than we may have suspected."

• The anticipated growth of inferencing over training signals a shift in focus and investment priorities in AI.

"Hyperscalers are really terrific and you just don't need to hit them as hard as you did when you were doing training."

• The efficiency and capabilities of hyperscalers are essential for supporting the evolving needs of AI applications.

• The current AI infrastructure is heavily focused on training large models with massive chip clusters, which may not be efficient for inference tasks.
• Inference requires a different infrastructure design due to its bursty and peaky nature, unlike the consistent demands of training.
• There is a potential shift towards more edge computing with low latency and high efficiency, impacting optical networking and power grid requirements.

"You're trying to utilize them at the highest possible percent for a long period of time. So you're trying to put 50, 100,000 chips in a single location and utilize them at the highest rate possible for nine months."

• The current infrastructure is designed for intensive, prolonged training processes, not for the variable demands of inference.

"In a sunk cost world you say sure, of course if I'm forced to build a million chip supercluster in order to train a $50 billion mob, I might as well sweat the asset when I'm done."

• The existing infrastructure is a result of significant investment in training, which may not be ideal for future inference needs.

• The shift from training to inference in AI is expected to accelerate innovation in the semiconductor industry, particularly in chip design and networking.
• Historical trends show that after a period of rapid expansion, there is a phase of optimization, leading to deflationary effects in technology.
• The semiconductor industry is poised for significant changes as companies optimize for inference efficiency.

"I would imagine this would accelerate it even more because it was very difficult to foresee a world where you took on big green in training."

• The transition to inference is expected to drive further innovation in semiconductor technology.

"And that's the beauty of technology over the long term is it is deflationary because it's an optimization problem."

• As technology matures, optimization leads to reduced costs and increased efficiency.

• Startups have opportunities to differentiate themselves in the semiconductor and networking layers as the focus shifts to inference.
• Organizations are finding themselves with excess capacity due to pre-purchased GPUs, allowing for cost-effective on-premises AI applications.
• The current overcapacity in infrastructure presents a favorable environment for developing AI applications.

"Inference on Llama 3.1405 billion for cerebras is it can generate 900 plus tokens per second which is a dramatic order of magnitude increase."

• Innovations in inference technology are significantly improving performance compared to traditional GPUs.

"So not only are all these application things that you're talking about hugely exciting because they unlock our and all the stuff, but the minute you can run any of this stuff on prem on our stuff, that dramatically decreases the cost for us."

• The ability to run AI applications on-premises reduces costs and increases capacity utilization.

• There is ongoing debate about the future of AI model development, particularly regarding the balance between pre-training, post-training, and test-time compute.
• Despite doubts about the effectiveness of brute force pre-training, AI progress is expected to continue at full speed.
• The focus of AI advancement may shift to more rational and sustainable approaches rather than limitless scaling.

"The reporting was not that the models weren't getting better, it's that the models weren't getting better relative to expectation or the amount of compute applied to them."

• While models are improving, the progress is not meeting the expectations set by the resources invested.

"AI is full speed ahead. I think the question is just what the axis of advancement is."

• AI development is progressing rapidly, but the direction of future advancements is still under discussion.

• Entrepreneurs are achieving significant advancements with limited capital, marking a shift in how AI innovation is approached.
• There is a need for more analysis on the new paradigm of test-time compute and its implications for the industry.
• The focus is on capital-efficient innovation, allowing startups to reach the frontier with minimal investment.

"This is just a shift that's happened very very recently and you're seeing people just show up and having spent under a million dollars to match performance not broadly, but in specific use cases with the frontier models."

• Startups are achieving frontier-level performance with significantly lower capital investment.

"Pre training is a big test of capitalism. If we pursue down this path, I feel much better with a microeconomic background analyzing what's going to happen because you don't have to put in the NPV of God."

• The shift in AI development approaches aligns better with economic principles, reducing the need for speculative investments.

• Breakthroughs in synthetic data and pre-training could dramatically alter the AI landscape, potentially leading to the development of AGI.
• There is significant untapped potential in data sources like video and audio, which could lead to new capabilities in AI models.
• The future of AI is unpredictable, with potential for both groundbreaking advancements and unforeseen challenges.

"If somebody came out with results that pre training was back on and there was a huge breakthrough on synthetic data and all of a sudden it's go, go again and ten billion dollars and a hundred billion dollars cluster would be back on the table."

• A major breakthrough in pre-training could reinvigorate large-scale AI model development.

"It's pretty clear now that while we've exhausted data on text, we are not close to exhausting data on video and audio."

• The focus on new data modalities like video and audio presents untapped opportunities for AI advancements.

• The concept of ASI (Artificial Superintelligence) raises philosophical questions about the future capabilities and impacts of AI.
• Human expectations of AI capabilities are constantly evolving, with each new advancement redefining what is considered intelligent.
• The potential for recursive self-improvement in AI models is a key area of interest and debate.

"Humans are really good at changing the goalposts on expectations, and AI in the 1970s meant something different than what it meant in 80s and 90s, 2000s and in 2024."

• Human perceptions of AI capabilities evolve with technological advancements, continually raising the bar for what is considered intelligent.

"There will be recursive self improvement at some point in time. And I think that that would be a big path to unlock in whatever hypothetically ASI means."

• The potential for AI models to improve themselves autonomously is a significant factor in the discussion of ASI.

• Silicon Valley remains a central hub for AI innovation, attracting talent and investment from around the world.
• The concentration of AI research and development in Silicon Valley highlights the region's continued importance in the technology sector.
• There is a growing awareness of the need to protect and sustain the innovation ecosystem in Silicon Valley.

"It's really amazing. There was an investor friend of mine who's not based in Silicon Valley, and he was just saying, I can't believe it's happening in Silicon Valley again."

• Silicon Valley continues to be a focal point for AI innovation, drawing attention and investment from global stakeholders.

"The agglomeration effects are real. If the reporting is right, the way the transformer paper came to pass is that someone was rollerblading down the hall and heard two guys talking about something and went in and whiteboarded."

• The collaborative environment in Silicon Valley fosters innovation through spontaneous interactions and idea exchanges.