Lightnews — Scholar-powered news

LightNews

Sumit

@reachsumit.com

190 followers 36 following 1.8K posts

Senior MLE at Meta. Trying to keep up with the Information Retrieval domain! Blog: https://blog.reachsumit.com/ Newsletter: https://recsys.substack.com/

blog.reachsumit.com

Posts Media Videos Starter Packs

Pinned

Sumit @reachsumit.com · 3d

LLMs often don't know what they don't know. They'll confidently generate wrong answers rather than admit uncertainty. This miscalibration makes it difficult to determine when external retrieval is actually necessary.

blog.reachsumit.com/posts/2025/0...

Probing LLMs' Knowledge Boundary: Adaptive RAG, Part 3

This post introduces techniques that probe the LLM’s internal confidence and knowledge boundaries. We explore prompt-based confidence detection, consistency-based uncertainty estimation, and internal ...

blog.reachsumit.com

Sumit @reachsumit.com · 1d

A2Search: Ambiguity-Aware Question Answering with Reinforcement Learning

Introduces an annotation-free framework that automatically detects ambiguous questions and gathers alternative answers.

📝 arxiv.org/abs/2510.07958
👨🏽‍💻 github.com/zfj1998/A2Se...

A$^2$Search: Ambiguity-Aware Question Answering with Reinforcement Learning

Recent advances in Large Language Models (LLMs) and Reinforcement Learning (RL) have led to strong performance in open-domain question answering (QA). However, existing models still struggle with ques...

Sumit @reachsumit.com · 1d

Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Window

Generates complex QA pairs from web sources and uses dynamic sliding window management to enable sustained interactions of nearly 100 turns within 32k context.

📝 arxiv.org/abs/2510.08276

Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Window

While recent advances in reasoning models have demonstrated cognitive behaviors through reinforcement learning, existing approaches struggle to invoke deep reasoning capabilities in multi-turn agents ...

Sumit @reachsumit.com · 1d

QAgent: A modular Search Agent with Interactive Query Understanding

Alibaba introduces a modular search agent that optimizes retrieval through interactive query understanding and multi-round reasoning.

📝 arxiv.org/abs/2510.08383
👨🏽‍💻 github.com/OpenStellarT...

GitHub - OpenStellarTeam/QAgent: the open-source code of QAgent

the open-source code of QAgent. Contribute to OpenStellarTeam/QAgent development by creating an account on GitHub.

Sumit @reachsumit.com · 1d

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Presents a benchmark built on Wikipedia's hyperlink network that evaluates how retrieval strategies affect LLM long-context performance.

📝 arxiv.org/abs/2510.07414
👨🏽‍💻 github.com/Graph-COM/Ha...

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Modern long-context large language models (LLMs) perform well on synthetic "needle-in-a-haystack" (NIAH) benchmarks, but such tests overlook how noisy contexts arise from biased retrieval and agentic ...

Sumit @reachsumit.com · 1d

Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems

Meta introduces a survey-based measure that captures users' intent to return for similar content, outperforming engagement signals in predicting next-day retention.

📝 arxiv.org/abs/2510.07621

Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems

Recommendation systems have traditionally relied on short-term engagement signals, such as clicks and likes, to personalize content. However, these signals are often noisy, sparse, and insufficient fo...

Sumit @reachsumit.com · 1d

PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations

Google adapts pre-trained LLMs for large-scale recommendation through Semantic ID tokenization, continued pre-training, and generative retrieval without embedding tables.

📝 arxiv.org/abs/2510.07784

PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations

Large Language Models (LLMs) pose a new paradigm of modeling and computation for information tasks. Recommendation systems are a critical application domain poised to benefit significantly from the se...

Sumit @reachsumit.com · 1d

HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation

Presents a training methodology using fine-grained process rewards to optimize search behavior in RAG systems.

📝 arxiv.org/abs/2510.07794
👨🏽‍💻 github.com/qualidea1217...

HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation

Agentic RAG is a powerful technique for incorporating external information that LLMs lack, enabling better problem solving and question answering. However, suboptimal search behaviors exist widely, su...

Sumit @reachsumit.com · 1d

Multilingual Generative Retrieval via Cross-lingual Semantic Compression

Introduces a framework that unifies semantically equivalent multilingual keywords into shared atoms, improving cross-lingual retrieval accuracy.

📝 arxiv.org/abs/2510.07812
👨🏽‍💻 github.com/simengggg/MG...

Multilingual Generative Retrieval via Cross-lingual Semantic Compression

Generative Information Retrieval is an emerging retrieval paradigm that exhibits remarkable performance in monolingual scenarios.However, applying these methods to multilingual retrieval still encount...

Sumit @reachsumit.com · 1d

STEPER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models

Proposes a step-wise KD method that trains student models to learn distinct reasoning abilities at each retrieval step.

📝 arxiv.org/abs/2510.07923

STEPER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models

Answering complex real-world questions requires step-by-step retrieval and integration of relevant information to generate well-grounded responses. However, existing knowledge distillation methods ove...

Sumit @reachsumit.com · 1d

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

Taobao presents an adaptive guided RL framework that addresses reward sparsity in e-commerce search through rule-aware reward shaping and replay mechanisms.

📝 arxiv.org/abs/2510.08048

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly ...

Sumit @reachsumit.com · 1d

VersionRAG: Version-Aware Retrieval-Augmented Generation for Evolving Documents

Proposes a framework that explicitly models document versioning through hierarchical graphs.

📝 arxiv.org/abs/2510.08109
👨🏽‍💻 github.com/danielhuwile...

VersionRAG: Version-Aware Retrieval-Augmented Generation for Evolving Documents

Retrieval-Augmented Generation (RAG) systems fail when documents evolve through versioning-a ubiquitous characteristic of technical documentation. Existing approaches achieve only 58-64% accuracy on v...

Sumit @reachsumit.com · 1d

ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval

Introduces a text embedding model for reasoning-intensive retrieval, using ReMixer data synthesis and Redapter self-adaptive learning.

📝 arxiv.org/abs/2510.08252
👨🏽‍💻 github.com/VectorSpaceL...

ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval

In this paper, we introduce ReasonEmbed, a novel text embedding model developed for reasoning-intensive document retrieval. Our work includes three key technical contributions. First, we propose ReMix...

Sumit @reachsumit.com · 2d

The Upside of Bias: Personalizing Long-Tail Item Recommendations with Biased Sampling

Introduces a training algorithm that prioritizes tail items in user interaction histories through biased sampling.

📝 dl.acm.org/doi/10.1145/...
👨🏽‍💻 github.com/lkp411/Biase...

The Upside of Bias: Personalizing Long-Tail Item Recommendations with Biased Sampling | ACM Transactions on Recommender Systems

Recommendation systems drive user engagement across social media, streaming platforms, and e-commerce by learning from past interactions. The relevance of a recommended item depends on the quality of ...

Sumit @reachsumit.com · 2d

Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them

Identifies four key reasoning behaviors for agentic search and proposes Behavior Priming that instills these behaviors via supervised fine-tuning before RL.

📝 arxiv.org/abs/2510.06534

Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them

Agentic search leverages large language models (LLMs) to interpret complex user information needs and execute a multi-step process of planning, searching, and synthesizing information to provide answe...

Sumit @reachsumit.com · 2d

Search-R3: Unifying Reasoning and Embedding Generation in Large Language Models

Adapts LLMs to generate search embeddings through CoT reasoning, combining supervised learning with RL for improved retrieval performance.

📝 arxiv.org/abs/2510.07048
👨🏽‍💻 github.com/ytgui/Search...

Search-R3: Unifying Reasoning and Embedding Generation in Large Language Models

Despite their remarkable natural language understanding capabilities, Large Language Models (LLMs) have been underutilized for retrieval tasks. We present Search-R3, a novel framework that addresses t...

Sumit @reachsumit.com · 2d

LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding

Oracle AI constructs symbolic document graphs to capture layout structure and cross-page dependencies, enabling dynamic retrieval for visually-rich documents

📝 arxiv.org/abs/2510.07233

LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding

Question answering over visually rich documents (VRDs) requires reasoning not only over isolated content but also over documents' structural organization and cross-page dependencies. However, conventi...

Sumit @reachsumit.com · 2d

PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs

Introduces a dynamic evaluation protocol that generates meaning-preserving paraphrases at evaluation time to assess embedding model robustness.

📝 arxiv.org/abs/2510.06730

PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs

Current evaluations of sentence embedding models typically rely on static test beds such as the Massive Text Embedding Benchmark (MTEB). While invaluable, repeated tuning on a fixed suite can inflate ...

Sumit @reachsumit.com · 2d

LLM-Powered Nuanced Video Attribute Annotation for Enhanced Recommendations

Google uses LLMs as annotators to achieve nuanced content understanding at scale for video recommendations, significantly improving user participation and satisfaction in production systems.

📝 arxiv.org/abs/2510.06657

LLM-Powered Nuanced Video Attribute Annotation for Enhanced Recommendations

This paper presents a case study on deploying Large Language Models (LLMs) as an advanced "annotation" mechanism to achieve nuanced content understanding (e.g., discerning content "vibe") at scale wit...

Sumit @reachsumit.com · 2d

Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

Exposes vulnerabilities in LLM-based reranking systems through a two-stage token optimization method.

📝 arxiv.org/abs/2510.06732
👨🏽‍💻 github.com/glad-lab/RAF

Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

Large language models (LLMs) are increasingly used as rerankers in information retrieval, yet their ranking behavior can be steered by small, natural-sounding prompts. To expose this vulnerability, we...

Sumit @reachsumit.com · 3d

We examine approaches ranging from simple prompt engineering to sophisticated internal state analysis and establish that each method has distinct trade-offs between accuracy, cost, latency, complexity, and model access requirements.

blog.reachsumit.com/posts/2025/0...

Probing LLMs' Knowledge Boundary: Adaptive RAG, Part 3

This post introduces techniques that probe the LLM’s internal confidence and knowledge boundaries. We explore prompt-based confidence detection, consistency-based uncertainty estimation, and internal ...

blog.reachsumit.com

Sumit @reachsumit.com · 3d

Instead of just analyzing the query, what if we could directly probe the model's own confidence? Part 3 of the Adaptive RAG series explores methods that directly assess LLM confidence and knowledge gaps.

Sumit @reachsumit.com · 3d

LLMs often don't know what they don't know. They'll confidently generate wrong answers rather than admit uncertainty. This miscalibration makes it difficult to determine when external retrieval is actually necessary.

blog.reachsumit.com/posts/2025/0...

Probing LLMs' Knowledge Boundary: Adaptive RAG, Part 3

This post introduces techniques that probe the LLM’s internal confidence and knowledge boundaries. We explore prompt-based confidence detection, consistency-based uncertainty estimation, and internal ...

blog.reachsumit.com

Sumit @reachsumit.com · 3d

Demystifying Deep Search: A Holistic Evaluation with Hint-Free Multi-Hop Questions and Factorised Metrics

Presents a benchmark with hint-free multi-hop questions and controlled Wikipedia sandbox.

📝 arxiv.org/abs/2510.05137

Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics

RAG (Retrieval-Augmented Generation) systems and web agents are increasingly evaluated on multi-hop deep search tasks, yet current practice suffers from two major limitations. First, most benchmarks l...

Sumit @reachsumit.com · 3d

Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents

Introduces Stratified GRPO to address cross-stratum bias in training LLM search agents.

📝 arxiv.org/abs/2510.06214

Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents

Large language model (LLM) agents increasingly rely on external tools such as search engines to solve complex, multi-step problems, and reinforcement learning (RL) has become a key paradigm for traini...

Sumit @reachsumit.com · 3d

RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-Style Contexts

Investigate how LLM-based guardrails handle RAG-style contexts with retrieved documents. Finds that adding benign documents changes guardrail judgments in ~11% of input cases.

📝 arxiv.org/abs/2510.05310

RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts

With the increasing adoption of large language models (LLMs), ensuring the safety of LLM systems has become a pressing concern. External LLM-based guardrail models have emerged as a popular solution t...