• At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime.
• The outcome? Up to 40 percent time to first batch (TTFB) improvements, along with a 20 percent reduction in Jupyter kernel startup times.
• This advancement facilitates swifter experimentation capabilities and elevates the ML developer experience (DevX).

Time is of the essence in the realm of machine learning (ML) development. The milliseconds it takes for an ML model to transition from conceptualization to processing the initial training data can dramatically impact productivity and experimentation.

At Meta, we’ve been able to significantly improve our model training times, as well as our overall developer experience (DevX) by adopting Lazy Imports and the Python Cinder runtime. 

The time to first batch challenge

Batch processing has been a game changer in ML development. It handles large volumes of data in groups (or batches) and allows us to train models, optimize parameters, and perform inference more effectively and swiftly.

But ML training workloads are notorious for their sluggish starts. When we look to improve our batch processing speeds, time to first batch (TTFB) comes into focus. TTFB is the time elapsed from the moment you hit the “start” button on your ML model training to the point when the first batch of data enters the model for processing. It is a critical metric that determines the speed at which an ML model goes from idle to learning. TTFB can vary widely due to factors like infrastructure overhead and scheduling delays. But reducing TTFB means reducing the development waiting times that can often feel like an eternity to engineers – waiting periods that can quickly amass as expensive resource wastage.

In the pursuit of faster TTFB, Meta set its sights on reducing this overhead, and Lazy Imports with Cinder emerged as a promising solution.

The magic of Lazy Imports

Previously, ML developers explored alternatives like the standard LazyLoader in importlib or lazy-import`, to defer explicit imports until necessary. While promising, these approaches are limited by their much narrower scope, and the need to manually select which dependencies will be lazily imported (often with suboptimal results). Using these approaches demands meticulous codebase curation and a fair amount of code refactoring.

In contrast, Cinder’s Lazy Imports approach is a comprehensive and aggressive strategy that goes beyond the limitations of other libraries and delivers significant enhancements to the developer experience. Instead of painstakingly handpicking imports to become lazy, Cinder simplifies and accelerates the startup process by transparently deferring all imports as a default action, resulting in a much broader and more powerful deferral of imports until the exact moment they’re needed. Once in place, this method ensures that developers no longer have to navigate the maze of selective import choices. With it, developers can bid farewell to the need of typing-only imports and the use of TYPE_CHECKING. It allows a simple from __future__ import annotations declaration at the beginning of a file to delay type evaluation, while Lazy Imports defer the actual import statements until required. The combined effect of these optimizations reduced costly runtime imports and further streamlined the development workflow.

The Lazy Imports solution delivers. Meta’s initiative to enhance ML development has involved rolling out Cinder with Lazy Imports to several workloads, including our ML frameworks and Jupyter kernels, producing lightning-fast startup times, improved experimentation capabilities, reduced infrastructure overhead, and code that is a breeze to maintain. We’re pleased to share that Meta’s key AI workloads have experienced noteworthy improvements, with TTFB wins reaching up to 40 percent. Resulting time savings can vary from seconds to minutes per run.

These impressive results translate to a substantial boost in the efficiency of ML workflows, since they mean ML developers can get to the model training phase more swiftly.

The challenges of adopting Lazy Imports

While Lazy Imports’ approach significantly improved ML development, it was not all a bed of roses. We encountered several hurdles that tested our resolve and creativity.

Compatibility

One of the primary challenges we grappled with was the compatibility of existing libraries with Lazy Imports. Libraries such as PyTorch, Numba, NumPy, and SciPy, among others, did not seamlessly align with the deferred module loading approach. These libraries often rely on import side effects and other patterns that do not play well with Lazy Imports. The order in which Python imports could change or be postponed, often led to side effects failing to register classes, functions, and operations correctly. This required painstaking troubleshooting to identify and address import cycles and discrepancies.

Balancing performance versus dependability

We also had to strike the right balance between performance optimization and code dependability. While Lazy Imports significantly reduced TTFB and enhanced resource utilization, it also introduced a considerable semantic change in the way Python imports work that could make the codebase less intuitive. Achieving the perfect equilibrium was a constant consideration, and was ensured by limiting the impact of semantic changes to only the relevant parts that could be thoroughly tested.

Ensuring seamless interaction with the existing codebase required meticulous testing and adjustments. The task was particularly intricate when dealing with complex, multifaceted ML models, where the implications of deferred imports needed to be thoroughly considered. We ultimately opted for enabling Lazy Imports only during the startup and preparation phases and disabling it before the first batch started.

Learning curve

Adopting new paradigms like Lazy Imports can introduce a learning curve for the development team. Training ML engineers, infra engineers, and system engineers to adapt to the new approach, understand its nuances, and implement it effectively is a process in itself.

What is next for Lazy Imports at Meta?

The adoption of Lazy Imports and Cinder represented a meaningful enhancement in Meta’s AI key workloads. It came with its share of ups and downs, but ultimately demonstrated that Lazy Imports can be a game changer in expediting ML development. The TTFB wins, DevX improvements, and reduced kernel startup times are all tangible results of this initiative. With Lazy Imports, Meta’s ML developers are now equipped to work more efficiently, experiment more rapidly, and achieve results faster.

While we’ve achieved remarkable success with the adoption of Lazy Imports, our journey is far from over. So, what’s next for us? Here’s a glimpse into our future endeavors:

Streamlining developer onboarding

The learning curve associated with Lazy Imports can be a challenge for newcomers. We’re investing in educational resources and onboarding materials to make it easier for developers to embrace this game-changing approach. 

Enhancing tooling

Debugging code with deferred imports can be intricate. We’re working on developing tools and techniques that simplify the debugging and troubleshooting process, ensuring that developers can quickly identify and resolve issues.

Community collaboration

The power of Lazy Imports lies in its adaptability and versatility. We’re eager to collaborate with the Python community – sharing insights, best practices, and addressing challenges together. Building a robust community that helps supporting paradigms and patterns that play well with Lazy Imports is one of our future priorities.