Will the AI start to deliver their promise?
The spending on AI data centers between 2024 and 2027 is expected to exceed $1.4 trillions. Yet few AI startups are turning a profit. As the cost of these start-ups are mounting, yet their deliveries are somewhat limited. AI companies are racing to make their AI more efficient, useful before investors losing their enthusiasm.
2 biggest constraints that these companies are facing is: Energy constraint and Data constraint.
Energy
Electricity used to train GPT-4 could have powered 5,000 American households a year, this figure for GPT-3 was 100. It is estimated that the next generation of models could cost $1 billion to train; and the larger they become, the more the cost of querying them (known as âinferenceâ) will mount.
AcOpenAI popularized the concept of the âscaling laws.â
Scaling laws applying to transformer said that âbigger models = better modelsâ
These âlawsâ provocatively make the case that bigger models = better models. The obvious implication is that we can just scale our way to AGI. (Page)
It has driven labs like OpenAI and Anthropic to build bigger and bigger clusters in order to train larger and larger models. (Page)
Which made AI companies to build bigger clusters to train larger models
The pretraining process â where models lazily train off the entire Internet â requires a massive amount of inputs, most notably, electricity. This poses a problem for the grid, which has grown accustomed to ~5% growth over the past decade. Our infrastructure is simply not ready for electricity demand to double over the coming decade. (Page)
The training from these clusters requires electricity, which the current supply does not meet expectation
Labs like Amazon, Google, and Microsoft â which signed climate pledges years ago that hold them to specific emissions targets â are racing to lock down nuclear power. But the US nuclear industry has been in perpetual decline since the Cold War. And most of these plants will take years, if not decades, to come back online. (Page)
Large corps opt for nuclear power, but it will take years before it can meet demand
Labs like xAI, which are not bound by the same environmental commitments, have resorted to burning natural gas in order to quench the thirst of their GPUs. (Page)
Labs like xAI use natural gas
The high-stakes game of locking down power highlights the challenge that lies ahead. If scale is really all you need, then we will need a lot more power soon. But where will it come from? Renewables are suboptimal because data centers require a steady stream of electricity that doesnât vary based on the wind blowing or sun shining. And nuclear, while promising, is in no state to scale up 10x any time soon. The obvious answer, at least for the US, seems to be natural gas. But itâs an open question whether the labs have the political stomach to start boiling the oceans in pursuit of silicon supremacy. (Page)
Currently there is not obvious energy source, except for natural gas
Some industries are further ahead in adopting ai than others: a fifth of information-technology firms, for instance, say they are using it. As the technology becomes more sophisticatedâsuch as with the arrival in 2025 of âagenticâ systems, capable of planning and executing more complex tasksâadoption may accelerate.. Although few firms tell statisticians they are using ai, one-third of employees in America say they are using it for work once a week. 78% of software engineers in America are using ai at least weekly, up from 40% in 2023, as are 75% of human-resources staff, up from 35%. And Openai says 75% of its revenue comes, tellingly, from consumers rather than from corporate subscriptions.
This suggest that individuals are adopting the AI technology more flexibly and freely than companies. To adopt AI â There are still concerns over privacy, and the questions of harm and good that this AI will bring. But this maybe the upward trend, as more people use AI â They may find more use cases for it â And then they might apply it to their companyâs official workflow.
Besides, this will be more growing need for private, individually train AI agents. in 2025, the most prominent ai breakthroughs come in other areas, such as drug development (the first ai-derived drugs may go into stage-three clinical trials) or defense (as intelligence is added to drones, which are emerging as key weapons systems of the future)
If 2023 was the year AI broke the Internet, then 2024 was the year AI broke financial markets. From Nvidia becoming the most valuable company on Earth, to OpenAI eclipsing a 6 billion for the one-year-old xAI. (Page)
The growth of AI over the years. First it changed the internet, then changed the market as it is, then brought out the leading companies in AI
We believe AI has two fundamental ingredients that distinguish it from previous boom and bust cycles:
capex generality (Page)
Analysis: 2 fundamental characteristics that set aside the AI
AI is unique from recent computing waves like mobile and social media as it requires a massive amount of physical infrastructure. To âmake AIâ you first need a lot of electricity, chips, and interconnects before you can even start. This requires building physical things in the real world. Itâs a vastly different paradigm than mobile, where devs built Flappy Bird apps and social media, where the only real âbuildingâ was follower counts. (Page)
Analysis/Infrastructure: The AI needs infrastructure for it to grow, instead of just apps and followers
AI will drive wealth creation across a much broader swath of society than in previous technology cycles. With social media, only the engineers in San Francisco and influencers in New York benefitted. But with AI, the local electrician benefits, the concrete guy benefits, and the mom-and-pop HVAC shop benefits. AIâs insatiable hunger for physical infra is driving a CAPEX boom across the country, and the spending looks set to accelerate from here. (Page)
Analysis/Infrastructure: It means the AI needs more services of more sectors, rather than just need devs â More people will win in this cycles than just dev
At a high level, there are two kinds of technologies. The first is an iterative update to the world â like moving from the iPhone 15 to iPhone 16. These âupdatesâ often introduce some cool new tech but rarely change anything in a meaningful way. (Page)
Analysis/ Technology: AI is the General Purpose Technologies.
The second set of technologies are the ones that deliver dramatic cost declines, impact many industries and geographies, and serve as a platform for future innovation. Economic historians refer to these as General Purpose Technologies (GPTs). (Page)
Electricity is considered a GPT since it provided a discontinuous reduction in the cost to generate, transmit, and deploy power. Adoption was widespread across sectors, with applications at both the business and the consumer levels that inspired many other innovations to be built on top of them. Similarly, artificial intelligence is accelerating computational capabilities far more rapidly than expected, with ramifications for every industry, including multi-trillion-dollar innovations beyond what we can imagine now. (Page)
âŠthat affect all aspects of our lives
AI will create orders of magnitude more wealth than past technology waves due to its ability to impact everything, everywhere, all at once. It will transform computing, medicine, manufacturing, agriculture, science, and the rest of the world with it. The economic ripple effects of such a profound change are hard to fathom and impossible to predict. (Page)
It will create more wealth in everywhere, everything
several open questions facing the AI industry will determine whether we continue climbing the exponential or pause and consolidate for a while. As we enter 2025, we have five big questions:
is pretraining hitting a wall? do the scaling laws hold for test time compute? how do we solve reasoning? when will the nation-state race begin? whatâs the next big breakthrough? (Page)
Transformer architecture has slowly eaten the AI space. Itâs now the most popular paradigm across modalities like text, code, and audio. (Page)
AI has been shifting to Transformer
The popularity of transformers boils down to their self-attention mechanism, which lets them process inputs in parallel rather than sequentially. This makes them far more scalable than other architectures. You can literally just throw compute at them, and they get better. (Page)
Aside from the question of âwhere will the power come from,â AI labs are also contending with other constraints imposed by the pretraining process. Data is the biggest. During pretraining, models consume vast amounts of data. They learn connections between words and model complex relationships by consuming trillions of tokens. This process requires a lot of data. In the early days, models would simply read the Internet. But weâve reached the point where even this massive corpus of text is insufficient to yield further performance gains. (Page)
Where does the data com from?
the leading labs are trying to solve this problem in similar, albeit distinct ways. OpenAI has been the most proactive about partnering with legacy media outlets to gain access to their data. This approach seems to be, at best, a temporary solution as companies like News Corp, TIME, and others only have so much data to share. (Page)
Google appears to have a clear advantage on the data front since it can tap YouTube, Gmail, and all its other honeypots. But itâs been rumored that OpenAI and others already trained off these sources, despite it being against the terms of service, so itâs unclear how much of an advantage this actually is (Page)
Other players like xAI can also access real-time data via platforms like Twitter. But again, most of this data isnât the high-quality kind they need to keep pushing the frontier. (Page)
Another strategy thatâs being pursued is synthetic data. This approach typically involves a larger model generating data thatâs used to train a smaller model. Weâve seen some indications this method is kinda working. Anthropic appears to have used Opus, their largest model, to generate training data for Claude Sonnet, their mid-sized model. And OpenAI is reportedly using o1 to create synthetic data for GPT-5/Orion. (Page)
Pretraining as we know it will end,â and synthetic data, in its current state, is still an unsolved problem. So, it remains to be seen how these labs will overcome the impending walls the pretraining paradigm presents. (Page)
these issues are constraints, not unsolvable problems. Scaling energy production requires human coordination, capital, and political will. While scaling data can probably only be solved through new techniques like synthetic data or more efficient algorithms. However, we tend to agree with Ilya. Pretraining does, indeed, appear to be plateauing, and the field can no longer solely rely on the âbigger is betterâ formula that got us this far. (Page)
As pretraining is becoming obsolete. Scaling data solution could be: Synthetic data, or more efficient algorithms.
Humans generalize on far less data than AI currently does. That means thereâs something our brains are doing algorithmically to do far more with far less data. (Page)
OpenAIâs o1 model uses a technique called test time compute, which lets models âponder.â This is a critical breakthrough that gives AI time to âthinkâ about harder questions like âWhatâs the meaning of life?â before answering. Aside from achieving SOTA-level reasoning on key benchmarks, this approach also appears to have unlocked a new scaling paradigm. (Page)
o1 is trained with RL to âthinkâ before responding via a private chain of thought. The longer it thinks, the better it does on reasoning tasks. This opens up a new dimension for scaling. Weâre no longer bottlenecked by pretraining. We can now scale inference compute too. (Page)
Instead of solely relying on larger clusters, o1 suggests we can eke out further performance gains by letting models think for longer. This approach is more economical and could help labs sidestep some of the scaling constraints we mentioned above. (Page)
We believe test time compute will be a key area to watch next year. OpenAI has a clear lead, but we expect Anthropic and other labs to release their own flavor of reasoning models within the next few months. If the scaling laws hold for inference, like they do for training, we believe the performance gains we have seen in recent years will continue. (Page)
If Iâm holding a glass of water and vertically rotate the glass 360 degrees, where is the water?
It doesnât take a genius to think about this scenario and realize the water would be on the ground. A child could reason about this. But the current models fail because they have no grounding and cannot reason coherently about the world. (Page)
o1 is the first model thatâs shown any ability to âreasonâ about the world. It does this through chain of thought reasoning (akin to a humanâs internal monlogue) and âthinking step by step.â It remains to be seen whether this is enough to solve reasoning or if additional breakthroughs are needed. But at the very least, we expect scaling test time compute, the core innovation behind o1, to yield impressive advances in reasoning ability. (Page)
one interesting thing is o1 can clearly do complicated geometry problems with no visuals. it approaches them completely symbolically, perhaps with some invisible world model in its activation states (Page)
t Nous Research calls âhunger.â They are essentially trying to teach models that things in the real world cost money, and if they cannot pay for stuff like inference, they will die. While this is not reasoning in the classical sense, we believe itâs a key unsolved problem with applications across the field. Nous is essentially trying to ground its models in the real world, which we see as an critical step toward general reasoning. After all, how can you reason about where the water is if you donât first understand your role in rotating the glass? (Page)
The US has also (surprisingly) been the most proactive on regulation. To be clear, thereâs much more work to be done on this front. But the recent White House memorandum was the most overt indication that the US government is taking the AI race seriously. We expect other flags to come to a similar realization next year. (Page)
It is now the official policy that the United States must lead the world in the ability to train new foundation models. All government agencies will work to promote these capabilities. (Page)
China has not yet adopted the same state-driven investment push we see in other strategic fields like manufacturing and semiconductors (Page)
This reticence is, admittedly, somewhat puzzling, but we expect China to eventually become AI-pilled and start building data centers like they built hospitals during COVID. We also predict that at least one major European country (our money is on France) will break ranks next year and implement AI-friendly policies to compete with the US and other world powers. (Page)
We believe we are already in a period of slight stagnation. And the media / X dot com poasters are just catching on to it now. But we also see reason for optimism. The biggest factor is o1. It proves that pretraining is no longer the only way to scale models. And its release has unlocked a new unexplored dimension for scaling. This could lead to a future where techniques are âstackedâ on top of each other, and together, they accomplish similar performance gains to the raw scaling of data centers. (Page)
To date, the leading edge of AI has been dominated by closed models from OpenAI, Anthropic, and Google. However, open source has somewhat closed the gap, primarily thanks to Zuckâs contrarian bet on opening sourcing Metaâs Llama models. (Page)
So far, the bet seems to be paying off handsomely for Meta. They bought themselves a bunch of goodwill and have also received a ton of free R&D from the open source community. But as the models grow bigger, costs rise, and capabilities increase, we wonder what Zuckâs tolerance will be for continued open sourcing. (Page)
if Zuck were to change his mind, perhaps due to safety concerns or mounting costs, we worry that open source efforts will struggle to compete against their well-funded closed competitors. (Page)
Every major lab is not-so-secretly working on agents, and has been for some time. So, we expect to see notable progress on this front next year. (Page)
the more a model reasons, the less predictable it becomes. The leading labs will likely be quite cautious with their releases, so this presents a unique opportunity for open source, crypto-native agents to leapfrog their closed source competitors. (Page)