Abstract

Scripting languages are widely used to compose external calls such as native libraries and network services. In such scripts, execution time is often dominated by waiting for these external calls, rendering traditional single-language optimizations ineffective. To address this, we propose a novel opportunistic evaluation strategy for scripting languages based on a core lambda calculus that automatically dispatches independent external calls in parallel and streams their results. We prove that our approach is confluent, ensuring that it preserves the programmer’s original intent, and that it eventually executes every external call. We implement this approach in a scripting language called Opal. We demonstrate the versatility and performance of Opal, focusing on programs that invoke heavy external computation through the use of large language models (LLMs) and other APIs. Across five scripts, we compare to several state-of-the-art baselines and show that opportunistic evaluation improves total running time (up to 6.2×) and latency (up to 12.7×) compared to standard sequential Python, while performing very close (between 1.3% and 18.5% running time overhead) to hand-tuned manually optimized asynchronous Rust. For Tree-of-Thoughts, a prominent LLM reasoning approach, we achieve a 6.2× performance improvement over the authors’ own implementation.

Formats available

You can view the full content in the following formats:

References

[1]

2022. Guidance. https://github.com/guidance-ai/guidance

[2]

2023. Official Repo of Tree of Thoughts. https://github.com/princeton-nlp/tree-of-thought-llm

[3]

Duane A. Adams. 1968. A Computation Model with Data-Sequenced Control. Stanford University. Technical Report CGTM 45

[4]

Duane A. Adams. 1969. A Computation Model with Data Flow Sequencing. Ph. D. Dissertation.

[5]

[17]

Bradford L Chamberlain, David Callahan, and Hans P Zima. 2007. Parallel programmability and the chapel language. The International Journal of High Performance Computing Applications, 21, 3 (2007), 291–312.

[18]

Harrison Chase. 2022. LangChain. https://github.com/langchain-ai/langchain

[19]

Alan L. Davis and Robert M. Keller. 1982. Data Flow Program Graphs. Computer, 15, 02 (1982), 2, 26–41.

[20]

Jack B. Dennis. 1974. First version of a data flow procedure language. In Programming Symposium, B. Robinet (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg. 362–376. isbn:978-3-540-37819-8

[21]

John T. Feo, David C. Cann, and Rodney R. Oldehoeft. 1990. A report on the sisal language project. J. Parallel and Distrib. Comput., 10, 4 (1990), 349–366. issn:0743-7315 https://doi.org/10.1016/0743-7315(90)90035-N Data-flow Processing

[22]

Cormac Flanagan, Amr Sabry, Bruce F Duba, and Matthias Felleisen. 1993. The Essence of Compiling with Continuations. In Proceedings of the ACM SIGPLAN 1993 conference on Programming Language Design and Implementation (PLDI). 237–247.

[23]

Matteo Frigo, Charles E. Leiserson, and Keith H. Randall. 1998. The implementation of the Cilk-5 multithreaded language. SIGPLAN Not., 33, 5 (1998), may, 212–223. issn:0362-1340 https://doi.org/10.1145/277652.277725

[24]

Kanishk Gandhi. 2022. Synchromesh. https://github.com/kanishkg/synchromesh

[25]

Jean-Yves Girard. 1987. Linear logic. Theoretical computer science, 50, 1 (1987), 1–101.

[26]

Robert H. Halstead. 1985. MULTILISP: a language for concurrent symbolic computation. ACM Trans. Program. Lang. Syst., 7, 4 (1985), Oct., 501–538. issn:0164-0925 https://doi.org/10.1145/4472.4478

[27]

Nick P Johnson, Hanjun Kim, Prakash Prabhu, Ayal Zaks, and David I August. 2012. Speculative separation for privatization and reductions. ACM SIGPLAN Notices, 47, 6 (2012), 359–370.

[28]

Simon Peyton Jones. 2001. Tackling the awkward squad: monadic input/output, concurrency, exceptions, and foreign-language calls in Haskell. NATO SCIENCE SERIES SUB SERIES III COMPUTER AND SYSTEMS SCIENCES, 180 (2001), 47–96.

[29]

Simon Peyton Jones, Andrew Gordon, and Sigbjorn Finne. 1996. Concurrent haskell. In POPL. 96, 295–308.

[30]

Konstantinos Kallas, Tammam Mustafa, Jan Bielak, Dimitris Karnikis, Thurston HY Dang, Michael Greenberg, and Nikos Vasilakis. 2022. Practically correct,Just-in-Time shell script parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 769–785.

[31]

Richard M. Karp and Raymond Miller. 1966. Properties of a Model for Parallel Computation: Determinacy, Termination, Queueing. SlAM J. of Applied Mathematics, 14, 6 (1966), 11, 1390–1411.

[32]

Richard M. Karp and Raymond Miller. 1969. Parallel Program Schemata. J. Comput. System Sci., 3 (1969), 147–195.

[33]

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714.

[34]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33 (2020), 9459–9474.

[35]

Georgios Liargkovas, Konstantinos Kallas, Michael Greenberg, and Nikos Vasilakis. 2023. Executing Shell Scripts in the Wrong Order, Correctly. In Proceedings of the 19th Workshop on Hot Topics in Operating Systems. ACM, 103–109. isbn:9798400701955 https://doi.org/10.1145/3593856.3595891

[36]

Jerry Liu. 2022. LlamaIndex. https://github.com/jerryjliu/llama_index

[37]

JW Maessen, RS Nikhil, and JE Stoy. 1996. S: an Implicitly Parallel-Calculus with Letrec, Synchronization and Side-E ects.

[38]

Shail Aditya Arvind Jan-Willem Maessen, Lennart Augustsson, and Rishiyur S Nikhil. 1995. Semantics of pH: A parallel dialect of Haskell. In In Proceedings from the Haskell Workshop (at FPCA 95). 35–49.

[39]

Simon Marlow. 2013. Parallel and concurrent programming in Haskell: Techniques for multicore and multithreaded programming. “ O’Reilly Media, Inc.“.

[40]

Simon Marlow, Louis Brandy, Jonathan Coens, and Jon Purdy. 2014. There is no fork: an abstraction for efficient, concurrent, and concise data access. SIGPLAN Not., 49, 9 (2014), aug, 325–337. issn:0362-1340 https://doi.org/10.1145/2692915.2628144

[41]

James R McGraw. 1982. The VAL language: Description and analysis. ACM Transactions on Programming Languages and Systems (TOPLAS), 4, 1 (1982), 44–82.

[42]

Stephen Mell. 2025. Artifact for Opportunistically Parallel Lambda Calculus. https://doi.org/10.5281/zenodo.16929280

[43]

Stephen Mell, Osbert Bastani, and Steve Zdancewic. 2023. Ideograph: A Language for Expressing and Manipulating Structured Data. Electronic Proceedings in Theoretical Computer Science, 377 (2023), April, 65–84. issn:2075-2180 https://doi.org/10.4204/eptcs.377.4

[44]

Tammam Mustafa, Konstantinos Kallas, Pratyush Das, and Nikos Vasilakis. 2023. DiSh: Dynamic Shell-Script Distribution. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, 341–356. isbn:978-1-939133-33-5 https://www.usenix.org/conference/nsdi23/presentation/mustafa

[45]

J.K. Ousterhout. 1998. Scripting: higher level programming for the 21st Century. Computer, 31, 3 (1998), 23–30. https://doi.org/10.1109/2.660187

[46]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019).

[47]

Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201.11227.

[48]

Jorge E. Rodriguez. 1969. A Graph Model for Parallel Computations. Ph. D. Dissertation. MIT-LCS-TR64

[49]

Mohammed Saeed, Nicola De Cao, and Paolo Papotti. 2023. Querying large language models with SQL. arXiv preprint arXiv:2304.00472.

[50]

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36 (2024).

[51]

Enrico Shippole. 2023. ReAct. https://github.com/conceptofmind/toolformer

[52]

Keenneth R Traub. 1988. Sequential implementation of lenient programming languages.

[53]

G. Tremblay. 2000. Lenient evaluation is neither strict nor lazy. Computer Languages, 26, 1 (2000), 43–66. issn:0096-0551 https://doi.org/10.1016/S0096-0551(01)00006-6

[54]

G. Tremblay and B. Malenfant. 2000. Lenient evaluation and parallelism. Computer Languages, 26, 1 (2000), 27–41. issn:0096-0551 https://doi.org/10.1016/S0096-0551(01)00007-8

[55]

Nikos Vasilakis, Konstantinos Kallas, Konstantinos Mamouras, Achilles Benetopoulos, and Lazar Cvetković. 2021. PaSh: light-touch data-parallel shell processing. In Proceedings of the Sixteenth European Conference on Computer Systems (EuroSys ’21). Association for Computing Machinery, New York, NY, USA. 49–66. isbn:9781450383349 https://doi.org/10.1145/3447786.3456228

[56]

Paul G. Whiting and Robert S. V. Pascoe. 1994. A history of data-flow languages. IEEE Annals of the History of Computing, 16 (1994), 38–59. https://api.semanticscholar.org/CorpusID:7384421

[57]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. Association for Computational Linguistics, 38–45. https://www.aclweb.org/anthology/2020.emnlp-demos.6

[58]

Shunyu Yao. 2023. ReAct. https://github.com/ysymyth/ReAct

[59]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36 (2024).

[60]

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023. Efficiently programming large language models using sglang. arXiv preprint arXiv:2312.07104.