.png){kind=logo}
Modern LLMs can generate novels in human-like texts but cannot remember and process long pieces of information. LLMs excel at predicting what comes next in a sentence within a fixed memory size of words and their relationship. When new information adds to this context window, it loses track of earlier details.
While models like GPT-4o and LLamaA3 extended their context window to cover a long research paper or multiple chapters in a book. They often struggled to maintain the accuracy, when working with longer text inputs.
Microsoft just introduced LongRoPE2 to keep the accuracy intact in extended LLM context windows. This is a game changer for document processing, long conversations and multi-step reasoning AI applications, where memory limitations previously caused performance issues.
LLMs rely on Rotatory Positional Embeddings (RoPE) technique to understand word order and their relationship. However, when RoPE extends beyond its fixed-size limit, it loses accuracy. The novel length of texts affects the loss of data from the earlier parts of the input. The new input text causes distortions in the representation of words
Why This Matters
Conversational AI: Chatbots and AI have to remember long conversations from past interactions to offer credible customer resolution.
Software Development: Developers need a large codebase without repeating context.
Scientific Research: Technical discussion requires lengthy and accurate information.
Legal and Financial Documents: Credible decisions are dependent on the ability to remember long contracts and reports without missing any details.
The Challenge? Previous attempts by other LLMs, such as YaRN, NTK, and LongRoPE1, focused on shorter input sizes to extend the context window size, making them impractical for real-world applications.
Microsoft’s LongRoPE2 innovation solves these problems:
Distorted Information: LongRoPE2 can remember two different words differently. Keywords will be given more attention to prevent loss of accuracy. While filler words' attention will only be remembered around their contextual meaning.
Perplexity Evaluation: Perplexity is a measure of prediction of the next word in a sentence. LongRoPE2 memory prioritizes difficult or context-heavy words over other words.
Mixed Context Training: Traditional LLMs are trained on short and long sentences separately. LongRoPE2 trains on both forms of sentences. Allowing the model to train beyond any current limit.
Microsoft’s test results are impressive. Its test with other major LLMs, such as Llama3-8B and Phi3-mini3.8B reveal:
97.6% accurate in short context: No longer need to sacrifice short-term memory for long term gains.
80% more efficient than Meta: Only 10 billion tokens were used for training, whereas Meta’s approach required 800 billion tokens.
0% loss in retrieval tasks: AI models can now extract information even buried deep in long documents.
These findings indicate a fundamental shift in LLMs ability to process large amounts of text.
The ability to process long contexts opens up new possibilities in AI applications, which are well integrated with our daily lives:
Better AI-Assistants: Meaningful Conversations and responses without forgetting past interactions
Smarter Enterprise Applications: Improved efficiency for financial modelling, legal research and compliance.
Search and Summarization: Improved research and data analysis workflow
Microsoft’s bold moves indicate a clear strategy to build a smarter, more context-aware model. The LongRoPE2 model is challenging the scaling tactics of OpenAI, Google DeepMind, and Anthropic in increasing token limits, without losing accuracy.
The next wave of AI innovation will tell whether this will be the bedrock or baby steps for future LLMs.
