Sequential-NIAH

Sequential-NIAH extends the Needle-in-a-Haystack test to the extraction of multiple connected pieces of information from long contexts.

What is Sequential-NIAH?

For practical AI use, Sequential-NIAH is more relevant than the simple Needle-in-a-Haystack test: in everyday situations, AI models often need to combine multiple scattered facts from long documents — for example for research, analysis, or summaries. Models that perform well on Sequential-NIAH are better suited for complex tasks like evaluating extensive data or synthesizing information from different sources.

Sequential-NIAH (S-NIAH) is a more demanding variant of the Needle-in-a-Haystack test that presents more realistic requirements to LLMs. While the classic NIAH test only hides a single “needle” in context, S-NIAH places multiple connected pieces of information at different locations. The model must find all parts and logically connect them — a task that is significantly closer to real-world use cases.

A typical example: three facts about a project are distributed across a 100,000-token document — the budget at position 20,000, the timeline at position 60,000, and the responsibilities at position 85,000. The LLM must extract all three pieces of information and combine them into a coherent response. S-NIAH shows that even models that pass the simple NIAH test lose significant accuracy when dealing with multiple distributed pieces of information.

For companies, S-NIAH results are especially relevant because business questions almost always require multiple information sources. The insights flow directly into Haystack Engineering: related information should be placed as close together as possible. Benchmarks like NOLIMA and S-NIAH help select the right models for complex retrieval tasks.