Machine learningLLMs & Text

Block Verification Accelerates Speculative Decoding

Speculative Decoding: The paper presents Block Verification, a new verification algorithm for large language models that checks a whole block of tokens at once, offering slight but consistent speed improvements over the standard token verification algorithm without adding to code complexity.