The field of Natural Language Processing has undergone a significant transformation with the advent of Large Language models. The pursuit of more advanced models has resulted in challenges, such as the need to scale up the models. Scaling up large language models refers to enhancing their size and capacity to improve their performance, which is usually measured by their ability to understand, generate, and interact with natural language.

However, before exploring details, let’s cover some basics.

Why Scaling up of Models is needed?

• Improving the performance of models in NLP tasks such as language understanding, translation, question-answering, and creative generation.
• Improving the model’s ability to understand context and complexities in the language
• A scaled-up model can be more versatile in different applications, ranging from simple tasks like classification to complex ones like generating coherent and contextually relevant text.

How Models are Scalled Up?

This scaling can be achieved in several ways, each affecting the model’s capabilities and efficiency:

1. Increasing the Number of Parameters: A standard method of scaling up LLMs is increasing the number of parameters. Parameters in neural networks are the parts of the model that are learned from training data. More parameters typically mean a greater ability to learn complex patterns and nuances in language.
2. Expanding the Depth or Width of the Nural Network: This involves adding more layers (Depth) or increasing the number of units per layer (width). A deeper network can capture more complex features and hierarchical representations of language, while a wider network can capture a broader range of features.
3. Enhancing Training Data: Scaling up can also mean using larger and more diverse datasets for training. A model trained on a more extensive and varied dataset can better understand and generate a wide array of linguistic styles and subjects.
4. Improving Computational Resources: Scaling up often requires more computational power. This includes using more powerful processors, such as GPUs or TPUs, and optimizing computational efficiency.
5. Advanced Architectures: Implementing advanced neural network architectures or techniques, like transformers, Depth Up-Scaling, or Mixture of Experts (MoE), can also be a form of scaling up. These architectures enable the model to process and generate language more effectively.

This paper introduces a new technique called Depth Up-Scaling (DUS) for scaling up models and its compared to Mixture of Experts (MoE) for scaling optimization. Before we explore DUS lets have a quick peak at MoE

What is Mixture of Experts (MoE)

In MoE, a neural network comprises numerous sub-networks known as ‘experts.’ Each expert is typically a smaller neural network that handles a specific task or data pattern.
There is a Gating network that analyzes each input and decides which of the experts are best suited to process it.
The gating network effectively routes different parts of the input data to different experts. Unlike traditional neural networks, where all parts of the network are used for every input, MoE dynamically allocates the input to only a subset of experts. This selective engagement of experts is what makes MoE efficient.

Challenges of MoE 

• Complexity: The architecture and training of MoE models are more complex than standard models. The gating mechanism adds an additional layer of complexity in both the forward and backward passes during training.
• Balancing Experts: Ensuring that all experts are trained effectively and that no single expert becomes a ‘jack-of-all-trades’, thereby defeating the purpose of specialization, is a challenge.
• Resource Management: While MoE can be more efficient per task, the overall resource requirements (such as memory and processing power) for training and maintaining such models can be substantial.

What is Depth Up-Scaling(DUS)

Depth Up-Scaling is a straightforward yet powerful method introduced in this paper to enhance language models’ capabilities. It focuses on adding more layers to the network to increase its Depth.

Key Concepts of DUS

• Increasing Depth: The primary strategy in DUS is to increase the number of layers in a neural network. Each layer in a neural network can be thought of as a level of processing, with each subsequent layer building on the previous ones to extract and refine features from the input data.
• Sequential Processing: In deep neural networks, information is processed sequentially through these layers. Each layer transforms the input in some way, often extracting higher-level features or representations as the data moves deeper into the network.

Advantages of DUS 

• Enhanced Learning Capability: Adding more layers can potentially increase the model’s ability to learn complex patterns and relationships in the data. This is particularly important in language models, where nuances and context are critical.
• Simplicity in Scaling: Compared to other scaling methods like width scaling (adding more neurons per layer) or complex architectures like Mixture of Experts (MoE), DUS maintains a relatively simple and linear structure.
• Compatibility with Existing Frameworks: Since DUS doesn’t radically alter the basic architecture of neural networks, it often remains compatible with existing training algorithms and frameworks.

Challenges and Considerations of DUS

• Vanishing/Exploding Gradients: Deeper networks can suffer from issues like vanishing or exploding gradients, where the gradients (used in training the network) become too small or too large to be effective.
• Training Difficulty: Training very deep networks can be challenging due to the increased complexity and the need for more sophisticated optimization techniques.
• Resource Requirements: While DUS might be more straightforward than other scaling methods, it still requires significant computational resources, especially as the number of layers grows.

What is SOLAR 10.7B Model

The authors of this paper created a model, SOLAR 10.7B, a large language model with 10.7 billion parameters to demonstrate the possibilities of DUS. The model is also available as SOLAR 10.7B-Instruct, which is fine-tuned for instruction-following capabilities. This model has shown superior performance as compared to the models with similar parameters.

Why Is This Significant?

• Efficiency: The performance of this model shows that it’s possible to scale up LLMs effectively while maintaining simplicity and computational efficiency using techniques like DUS.
• Performance: SOLAR 10.7B demonstrates superior performance in various NLP tasks compared to existing models of similar size, such as Llama2 and Mistral 7B, in reasoning, mathematics, and MMLU Framework.

Open Source Access: Released under the Apache 2.0 license, SOLAR 10.7B promotes broader access and application in the LLM field, facilitating further research and development.

Its Depth Up-Scaling approach to model scaling, and the trained models compatibility with existing frameworks make it a valuable contribution to the NLP community. The paper showcases the potential of Depth Up-Scaling in overcoming some of the challenges associated with scaling up LLMs, paving the way for more accessible and efficient language models in the future.

Key Links 

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
HuggingFace Page
Upstate.AI

Paper Authors : Dahyun Kim , Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim Mikyoung Cha, Hwalsuk Lee, Sunghun Kim