LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

In finance, the answer that matters is usually the current one: prices, rates, and disclosures hold only as of a given day, so retrieving up-to-date facts is a core capability of a financial agent. Yet most financial benchmarks overlook timeliness. They fix each answer at annotation time, so the gold goes stale as markets move and, once leaked into training data, rewards recall rather than search and analysis. We introduce LiveSearch, a live benchmark that stores each question as executable verifier code: a typed program of calls to a professional financial data service, followed by a declarative extractor. The question is frozen, while its gold answer is determined by executing the program against professional financial data at evaluation time, and a daily job re-executes it with no human or language model in the loop. A memorized answer is thus likely outdated, and a model must reach the current one through tools and analysis. Construction is equally automatic: a language model only verbalizes programs enumerated from real tool responses and never chooses an answer. Under general web search, six contemporary tool-calling models reach 70.9% to 76.7% answer accuracy. Over three consecutive days, 70% of served questions change their gold value, and even under tolerance-aware matching, a literally reused earlier snapshot fails on a majority of questions within a single day. Per-question tracking uncovers dynamic shifts in correctness masked by aggregate accuracy.

1. 9 Oct 2024

   GPT-4o as the Gold Standard: A Scalable and General Purpose Approach to Filter Language Model Pretraining Data

   Data Filtering System with GPT-4o Accuracy: The article introduces SIEVE, a cost-effective method for filtering web-scale data that matches the accuracy of GPT-4o and is efficient in curating large datasets for language model training.

   Machine learning

   3cites
2. 13 Mar 2024

   The Financial Uncertainty of Climate-Related Assets

   The paper identifies a factor linked to climate news indices and commodity markets that contributes to the excess volatility of climate-sensitive assets.

   SSRN

   1cites
3. 25 Sep 2026

   FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering

   A benchmark reveals that language models answer financial questions above 90 percent with clarification but only 28.9 percent when they must elicit it themselves, exposing model ambiguity resolution.

   arXiv

   0cites
4. 18 Sep 2024

   Investor Sentiment in Malaysian Government Bonds

   The COVID-19 pandemic has changed investment strategies in bond markets, with Malaysian government bonds showing equity-like traits and deviating from their usual safe-haven role, with gold prices being the key factor affecting bond yields.

   SSRNFeatured 3×

   64shares
5. 23 Jan 2024

   Media Hype and Fake News Impact on Commodity Prices

   The study shows that media hype and fake news greatly influence commodity prices, especially during COVID-19. It also found that bi-directional long-short-term memory is useful in predicting these impacts.

   RePEc

   10shares
6. 14 Jun 2023

   Large Datasets and Hybrid Models

   Study explores using machine learning and sentiment analysis to forecast foreign exchange rates and commodity prices.

   RePEcFeatured 3×

   31shares