Same Text, Different Numbers: The Divergence of LLM-Based Measures

Researchers increasingly use generative large language models (LLMs) to convert corporate text into empirical variables. We examine the extent to which LLM-based textual measures are invariant to model choice using thirteen measures, including sentiment, management clarity, uncertainty, answer specificity, and climate and political risk. Seven LLMs from different providers score earnings call transcripts of S&P 500 companies on these constructs. Cross-model rank correlations average only 0.52, and transcript-level differences common across providers account for only 34% of total score variation. Cross-model disagreement does not predict subsequent analyst or market disagreement, consistent with a substantial model-specific component rather than common ambiguity in the underlying disclosure. Model choice significantly affects downstream inference, with coefficient magnitudes, signs, and statistical significance varying substantially across models. Averaging across providers makes transcript rankings more stable for most constructs, but score levels remain sensitive to the models included in the ensemble. LLM-generated variables should therefore be treated as model-contingent measurements and validated across providers.

1. 31 Jul 2024

   The Impact of the ESG Factor on Industrial Performance An Analysis using Machine Learning Techniques

   The study applies machine learning to explore the link between ESG performance and corporate earnings, using data from over 850 European and US firms from 2007-2021.

   SSRN

   4cites
2. 6 Mar 2024

   A Yield-Based Asset Ratio to Boost Minimum Investment Returns

   The article proposes a strategy to improve weak medium-term returns in retirement portfolios by adjusting the stock percentage based on the earnings yield of stock and the current yield of bonds, with caution needed when stock prices exceed sustainable levels.

   SSRN

   0cites
3. 15 May 2024

   AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

   AI Evaluation in Clinical Environments: The paper introduces AgentClinic, a benchmark for assessing large language models in simulated clinical environments, highlighting the significant impact of biases on diagnostic accuracy and patient interactions.

   Machine learning

   263cites
4. 1 May 2024

   Learning Performance-Improving Code Edits

   The research presents a framework for optimizing programs using large language models, achieving a mean speedup of 6.86, outperforming average individual programmers.

   Machine learningIn International Conference on Learning Representations

   167cites
5. 13 Dec 2023

   Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia

   Concordia is a library designed to help build and operate Generative Agent-Based Models (GABMs), using Large Language Models (LLMs) to simulate physical or digital environments.

   Machine learning

   166cites
6. 28 Aug 2024

   Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

   Transformer-Mamba Models: Jamba-1.5 is a new large language model with enhanced conversational and instruction-following capabilities, featuring a unique quantization technique for cost-effective inference.

   Machine learning

   56cites