Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.

1. 15 May 2024

   AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

   AI Evaluation in Clinical Environments: The paper introduces AgentClinic, a benchmark for assessing large language models in simulated clinical environments, highlighting the significant impact of biases on diagnostic accuracy and patient interactions.

   Machine learning

   256cites
2. 1 May 2024

   Learning Performance-Improving Code Edits

   The research presents a framework for optimizing programs using large language models, achieving a mean speedup of 6.86, outperforming average individual programmers.

   Machine learningIn International Conference on Learning Representations

   164cites
3. 13 Dec 2023

   Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia

   Concordia is a library designed to help build and operate Generative Agent-Based Models (GABMs), using Large Language Models (LLMs) to simulate physical or digital environments.

   Machine learning

   164cites
4. 28 Aug 2024

   Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

   Transformer-Mamba Models: Jamba-1.5 is a new large language model with enhanced conversational and instruction-following capabilities, featuring a unique quantization technique for cost-effective inference.

   Machine learning

   56cites
5. 31 Jul 2024

   MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

   Mimicking Human Minds for Search: MindSearch is a Large Language Model-based framework that simulates human cognitive processes for web information seeking, greatly enhancing response quality.

   Machine learningIn International Conference on Learning Representations

   80cites
6. 13 Sep 2025

   From Digital Distrust to Codified Honesty: Experimental Evidence on Generative AI in Credence Goods Markets

   Large language models (LLMs) in expert services have pros and cons, with human markets being more efficient, but LLMs potentially reducing trust and overshadowing experts' preferences, while also improving experts' communication of their goals.

   arXiv

   3cites