FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering

Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specified: Meta Platforms' "operating income" is $46.75B consolidated but $62.87B for the Family of Apps segment, and each reading is exactly verifiable against the filing. A capable agent should recognize the ambiguity and ask, rather than commit to a plausible but unintended reading. Existing financial benchmarks cannot measure this, because one gold answer per question cannot separate agents that resolve the ambiguity from those that guess the common reading, a blind spot we call the single-gold illusion. We release FinInteract, a bilingual (English/Chinese) benchmark of 173 instances that pairs each question with a default and an intended interpretation across a five-category ambiguity taxonomy, and grades whether an agent elicits the right clarification and then integrates it. Re-grading identical outputs against the default rather than the intended reading inflates GPT-4o's accuracy by 3.1 times, confirming the illusion. Beyond it, we find that models answer above 90% once the interpretation is supplied but at most 28.9% when they must elicit it themselves, that targeting is uneven across a taxonomy well powered for entity scope and metric definition and exploratory elsewhere, and that conditioning on the ambiguity category improves resolution at both inference and training time.

1. 9 Oct 2024

   GPT-4o as the Gold Standard: A Scalable and General Purpose Approach to Filter Language Model Pretraining Data

   Data Filtering System with GPT-4o Accuracy: The article introduces SIEVE, a cost-effective method for filtering web-scale data that matches the accuracy of GPT-4o and is efficient in curating large datasets for language model training.

   Machine learning

   2cites
2. 7 Aug 2024

   Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

   The research investigates enhancing Large Language Models' (LLMs) performance using more test-time computation, suggesting a compute-optimal scaling strategy based on prompt difficulty.

   Machine learning

   2,189cites
3. 12 Dec 2024

   Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

   The paper presents InternVL 2.5, a sophisticated multimodal large language model that performs well on various benchmarks, exceeding 70% on the MMMU benchmark.

   Machine learning

   1,775cites
4. 24 Jul 2024

   AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration

   The study suggests Activation-aware Weight Quantization (AWQ), a hardware-friendly method for quantizing large language models that reduces error and improves performance on various benchmarks.

   Machine learningIn GetMobile: Mobile Computing and Communications

   1,800cites
5. 16 Oct 2023

   MemGPT: Towards LLMs as Operating Systems

   Extended Context in LLMs: MemGPT is a system that manages different memory levels, providing extended context within large language models' limited context windows, enhancing document analysis and multi-session chat performance.

   Machine learningFeatured 2×

   1,373cites
6. 16 Oct 2023

   SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

   Hallucination Detection for LLMs: The paper presents SelfCheckGPT, a new approach for fact-checking black-box model responses without an external database, proving its superior ability to detect and rank factual and non-factual sentences.

   Machine learningIn Conference on Empirical Methods in Natural Language ProcessingFeatured 2×

   1,236cites