Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Current LLM systems are increasingly equipped with a code interpreter that executes generated code to obtain results. This works serially: the model first generates the complete code, then an interpreter executes it. This sequential workflow leaves the executor idle during generation and the generator idle during execution, resulting in unnecessary end-to-end latency. Our key observation is that an LLM, unlike a human developer, emits code tokens left to right and does not backtrack over what it has already written. This makes it possible to start executing a piece of code while later tokens are still being generated. We formalize this parallel execution paradigm, modeling it as a three-s...
| Authors | Zhensu Sun, Zhihao Lin, Zhi Chen et al. |
|---|---|
| Primary category | cs.PL |
| Published | 1 April 2026 |
| Last updated at source | 22 June 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.