<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us"><generator uri="https://gohugo.io/" version="0.170.0">Hugo</generator><title type="html">Nextest on sunshowers</title><subtitle type="html">thoughts and ideas</subtitle><link href="https://sunshowers.io/tags/nextest/" rel="alternate" type="text/html" title="html"/><link href="https://sunshowers.io/tags/nextest/index.xml" rel="alternate" type="application/rss+xml" title="rss"/><link href="https://sunshowers.io/tags/nextest/atom.xml" rel="self" type="application/atom+xml" title="atom"/><updated>2026-11-15T23:03:09+00:00</updated><rights>© Rain 2020-present. Licensed under CC BY-NC-SA 4.0 unless marked otherwise.</rights><author><name>Rain</name></author><id>https://sunshowers.io/tags/nextest/</id><entry><title type="html">Designing nextest's install scripts</title><link href="https://sunshowers.io/posts/nextest-install-scripts/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="related" type="text/html" title="Why nextest is process-per-test"/><link href="https://sunshowers.io/posts/nextest-and-tokio/?utm_source=atom_feed" rel="related" type="text/html" title="How (and why) nextest uses tokio"/><id>https://sunshowers.io/posts/nextest-install-scripts/</id><published>2026-11-14T00:00:00+00:00</published><updated>2026-11-14T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>How I built the scripts carefully using coding agents.</blockquote>
  <figure class="floatright narrow" >
    
      <a href="/images/nextest-install-scripts-card.png">
    
    <img src="/images/nextest-install-scripts-card.png"  alt="The cargo-nextest home page, with the headline &#39;A next-generation test runner for Rust&#39; above tabs for &#39;Linux, macOS, WSL&#39; and &#39;Windows (PowerShell)&#39;. The selected tab shows the command &#39;curl -LsSf https://get.nexte.st/install.sh | sh&#39;."   />
    
      </a>
    
    
      <figcaption class="center" >Now leading with install scripts!</figcaption>
    
  </figure>


<p>Four and a half years after I released the first version of <a href="https://nexte.st/">cargo-nextest</a>, it now has install scripts! This has been a long time coming, and I&rsquo;m really happy to finally have support for them.</p>
<p>On Unix-like platforms, including WSL, <a href="https://www.cygwin.com/">Cygwin</a>, and <a href="https://gitforwindows.org/">Git Bash for Windows</a> (<a href="https://www.msys2.org/">MSYS</a>), you can now run <a href="https://get.nexte.st/install.sh"><code>install.sh</code></a>:</p>
<pre tabindex="0"><code>curl -LsSf https://get.nexte.st/install.sh | sh
</code></pre><p>On Windows, you can also use PowerShell with <a href="https://get.nexte.st/install.ps1"><code>install.ps1</code></a>:</p>
<pre tabindex="0"><code>irm https://get.nexte.st/install.ps1 | iex
</code></pre><p>I wanted to write a bit about why it took so long to publish them, how they work, and how I used coding agents to meet a high quality bar.</p>
<h2 id="background">Background</h2>
<p>We&rsquo;ve published release binaries for the nextest project since <a href="https://nexte.st/changelog/#0.9.4">its very earliest versions</a> in 2022. When initially designing support for release binaries, I spent some time looking at the options for how to deliver them to users. The usual approach is an install script that you <code>curl | sh</code> or <code>irm | iex</code>, but I determined that:</p>
<ul>
<li>Writing install scripts is quite difficult. You have to maintain separate Unix and Windows scripts, each with its own hazards, and test in a myriad of different situations. My general philosophy while writing infrastructure is to build things in a way that I don&rsquo;t have to think about them again, and I felt that any install script would have a long tail of issues I&rsquo;d have to chase down. I didn&rsquo;t think I had the skills required to write one that lived up to my own standards.</li>
<li>Many users (quite understandably) do not like <code>curl | sh</code>, and it would be a mistake for that to be the <em>only</em> way to install release binaries<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</li>
</ul>
<p>Based on this, I came up with an alternate approach: produce tarballs (and a zip file for Windows), and ask people to <a href="https://nexte.st/docs/installation/pre-built-binaries/#downloading-and-installing-from-the-release-urls">extract it themselves</a>. For example, on Linux, you would run:</p>
<pre tabindex="0"><code>curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin
</code></pre><p>Some notes:</p>
<ul>
<li>The URLs were backed by a redirect service (originally Netlify, now <a href="https://github.com/nextest-rs/release-meta/blob/main/_redirects">Cloudflare</a>), with the intent to keep them relatively short.</li>
<li>The binaries were built in <a href="https://github.com/nextest-rs/nextest/blob/8f2d970b36d8be5b7aafbf201c07f0f87506b8e3/.github/workflows/release.yml#L184">GitHub Actions</a> and hosted on <a href="https://github.com/nextest-rs/nextest/releases/tag/cargo-nextest-0.9.152">GitHub Releases</a>, and do not change after initial publication. (Originally, this was policy. Nowadays this is cryptographically attested via GitHub&rsquo;s <a href="https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases">immutable releases</a> feature.)</li>
<li>The code to manage all this lived in a little release manager I wrote for nextest called <a href="https://github.com/nextest-rs/mukti">mukti</a><sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</li>
</ul>
<p>But over time, the cracks started to show: while the Unix shell instructions were a single line, the Windows instructions were a <a href="https://nexte.st/docs/installation/pre-built-binaries/#powershell">five-line mess</a>. There wasn&rsquo;t a single command that worked on any platform, so nothing that could be shown on the home page. Also, on Linux, there was no way to select a statically-linked musl build automatically if the system version of glibc was too old or not present.</p>
<p>I felt that install scripts were going to be needed in the long term. This year, over a month of intermittent work, I finally put together a pretty solid pair of scripts. They&rsquo;re tested to work in a wide variety of situations under many different failure modes, and I feel quite confident they meet the quality bar set by the rest of nextest. (The manual archive extraction path will continue to work as well.)</p>
<h2 id="how-the-install-scripts-work">How the install scripts work</h2>
<p>Mukti has been extended to produce a text file containing archive URLs and checksums for each version, called an <em>install manifest</em> (<a href="https://get.nexte.st/latest/install-manifest.txt">example for the latest version</a>). The manifest is designed to be easily parseable with standard word-splitting utilities present on every system, like <a href="https://pubs.opengroup.org/onlinepubs/9799919799/utilities/read.html"><code>read</code></a> in Unix shells and <a href="https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_split?view=powershell-7.6"><code>-split</code></a> in PowerShell.</p>
<p>The shell script works with any Bourne shell, and the PowerShell script runs on both the legacy (but installed by default on Windows) version 5.1 and the new PowerShell 7. They follow a series of steps:</p>
<ol>
<li>Download the install manifest for the specified version, which is always at <code>https://get.nexte.st/&lt;version&gt;/install-manifest.txt</code>. (<code>&lt;version&gt;</code> can also be a series like <code>0.9</code>, or the string <code>latest</code> for the latest release; this is carried forward from the release URLs.)</li>
<li>Examine the local system and determine the best candidates for the release.</li>
<li>Download the best available candidate, verify its checksum, and extract the binary into the right spot.</li>
</ol>
<p>As I expected, a production-ready install script has to deal with a surprising amount of complexity. Some examples:</p>
<ul>
<li>On Linux, if the system version of glibc is too old or not present, a statically-linked musl build is selected. (We do not install the musl build by default, originally because nextest on musl <a href="https://github.com/rust-lang/rust/issues/99740">used <code>fork</code>/<code>exec</code></a> over the far more efficient <code>posix_spawn</code> to spawn processes. This has <a href="https://github.com/rust-lang/rust/pull/131851">since been fixed</a>, but musl is still slightly slower than glibc, likely due to a less performant memory allocator.)</li>
<li>On current versions of macOS, the default <code>sh</code> is the <a href="https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-in-macos-catalina-and-beyond/">very outdated</a> bash 3.2. The script works around a number of defects in this version of bash, such as a command inside an <code>if</code> that fails to exec reporting exit status 1 instead of 126, an issue fixed in upstream bash <a href="https://github.com/gitGNU/gnu_bash/commit/66119cfc34d33cc1de7f760c636f7c1f3e74e70e#diff-deade331ca123fb3869937232daa4d8d4d6410c5df0dbfcfb67023c445968abf">back in 2014</a>.</li>
<li>On Windows, one could be using a shell emulated to run on a different architecture, such as an x86_64 Git Bash on aarch64 Windows. The scripts detect such situations and install the correct version of nextest for the native architecture.</li>
</ul>
<h2 id="so-lets-talk-agentic-coding">So… let&rsquo;s talk agentic coding</h2>
<p>The install process, such as the manifest format and location at well-known URLs, was very much designed by me. I had a ton of implementation and testing help from Claude Code and Codex, though.</p>
<p>One way to use them, which I&rsquo;m going to call the <em>vibed</em> approach, is:</p>
<ul>
<li>Tell Claude, GPT, or some other LLM, &ldquo;write me an install script, make no mistakes&rdquo;.</li>
<li>Test it out locally and report a few bugs. Have the LLM dutifully fix those bugs by adding more code.</li>
<li>Ask the LLM to add more features, adding even more code.</li>
<li>Put it out into the world, and when others report defects, have the LLM fix them by adding yet more code.</li>
</ul>
<p>This is how you end up with install scripts with hundreds of lines of repetitive sections. Earlier this year, <a href="https://github.com/openclaw/openclaw/blob/2f90c703bdf55ed99219100832b2e4c85ad5b0fc/scripts/install.sh">a popular project&rsquo;s install script</a> became a somewhat notorious example of this pattern.</p>
<p>But this vibed approach is not how I develop software meant to be used by others. For one, nextest is not my full-time job, and I don&rsquo;t want to be chasing down a steady stream of issues filed against it. More importantly, I find this approach to building infrastructure-grade software, whatever technical merits it might or might not have, to be against my values. I&rsquo;m not going to give up my values in service of getting something like this out quicker.</p>
<p>So how did I end up using LLMs for this? Very carefully, and in a way that aimed to <a href="https://oxide-and-friends.transistor.fm/episodes/engineering-rigor-in-the-llm-age">increase the level of rigor</a> compared to writing the script by hand.</p>
<p>On the LLM side, by far the most effort went into validation. From my experience with <a href="https://github.com/oxidecomputer/dropshot-api-manager">projects at work</a>, I knew that LLMs can be very thorough at building test harnesses that would otherwise require weeks of sustained effort. So, very early on, I decided to build an automated test harness for the installer script. The test harness defines <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/cases.rs">dozens of cases</a> with expected inputs and behaviors, and runs them on a <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/shells.rs#L19-L34">variety of shells</a>. The harness also has:</p>
<ul>
<li>An <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/server.rs">HTTP server</a> to serve artifacts from, with a <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/tls.rs">TLS frontend</a>;</li>
<li>A <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/test-helpers/mukti-test-helper.rs">test helper multicall binary</a> to inject a variety of faults, and also to <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/test-helpers/mukti-test-helper.rs#L71">hold open</a> a running binary on Windows to ensure that the PowerShell script can replace it;</li>
<li>A set of <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/standalone.rs">standalone tests</a> for edge cases, including tests for <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/standalone.rs#L38">truncated scripts</a> and cleanup on <a href="https://github.com/nextest-rs/mukti/blob/19cb9698a5c5c2b28d7b562ef5f181eebe400de9/integration-tests/tests/install-scripts/standalone.rs#L417">interrupted downloads</a>.</li>
</ul>
<p>All of these are well within my capacity to build in principle, though not in any reasonable amount of time. But once the high-level properties one wants to test, such as timeouts on a stalled download, are defined, LLMs are very good at this kind of fill-in-the-blanks semantic infill.</p>
<p>The other large bit of work was on the human side, with design review. Over time, as I&rsquo;ve gained experience with agentic tools, I&rsquo;ve settled on a fairly particular — some might say peculiar — way of using them. I first run any code through automated review<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> with two different LLMs. After that, I spend a lot of time manually reviewing the code.</p>
<p>How do I make sure I pay attention to the code I&rsquo;m reviewing? For me, the most effective technique has been to <a href="https://gist.github.com/sunshowers/61eb834c6b62abb916eb3c9c23fc3f02">ban the agents</a> from writing comments <a href="https://gist.github.com/sunshowers/d522a672329d4206b229e1a259a8dceb">without a special marker</a>. To be honest, I <em>can&rsquo;t stand</em> LLM-authored technical writing, and the comments these tools produce are uniformly terrible. But also, when I&rsquo;m forced to write about how or why some code is written the way it is, I&rsquo;ve consistently found a <em>startling</em> number of issues. Some that I found in this effort:</p>
<ul>
<li>
<p>The tests were incredibly comprehensive at validating all the possible non-default options, but there were no tests for the most basic no-option happy path. I realized this around a third of the way into the review, and did a major rework of the test harness based on it.</p>
</li>
<li>
<p>In the test harness, SHA-256 hashes were represented as Rust <code>String</code>s, and the tests did a weird case-insensitive comparison. I replaced it with the <a href="https://docs.rs/byte-wrapper">byte-wrapper</a> crate I wrote for Oxide, which provides a <code>HexArray&lt;N&gt;</code> newtype<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>.</p>
</li>
<li>
<p>The scripts initially permitted HTTPS to HTTP redirects, and the tests validated that this scenario works. That is completely wrong, of course. Such redirects are now banned.</p>
</li>
<li>
<p>The Unix tests validated that signals performed cleanup, but there were originally no Ctrl-C tests on Windows. There are now tests for that.</p>
</li>
</ul>
<p>A common theme across many of the defects is that an LLM will often pick a hill to climb, then climb it as thoroughly as possible, without asking whether it is the right hill to climb in the first place. The HTTPS to HTTP redirects are a rather striking example of that: at some point, Claude decided that this is what the script should do, then encoded it in the test suite. The automated review done by subsequent Claude and GPT instances likely led them to conclude &ldquo;oh, the tests verify that such redirects work — clearly this must be intended behavior&rdquo;. It requires human wisdom to judge that <em>no, this is wrong</em>.</p>
<p>All in all, the manual review took around 25 hours over 4 weekends. The final result was a change of around <a href="https://github.com/nextest-rs/mukti/pull/943">10,000 lines of code</a>, most of which is tests — very different from the initial work done by the LLM, and something I believe I can stand behind. I&rsquo;m sure there will be a few bugfixes here and there, and maybe major feature work at some point, but overall I consider this work to be completed.</p>
<p>The work is <a href="https://github.com/nextest-rs/mukti">generically implemented in mukti</a> and reusable by other projects, too. I&rsquo;m not totally sure mukti is fit for public consumption yet, but it&rsquo;s not too far off. Given how well this works, it might finally be time to push mukti over the line. Maybe over the holidays!</p>
<h2 id="conclusion">Conclusion</h2>
<p>If a thing is worth doing, is it worth <a href="https://www.bookbrowse.com/expressions/detail/index.cfm/expression_number/537/if-a-jobs-worth-doing-its-worth-doing-well">doing well</a>? Or should you <a href="https://proverbhunter.com/if-a-thing-is-worth-doing-it-is-worth-doing-well/">do it badly</a>? The software industry has gone through a remarkable transformation over the last year. To my dismay, a lot of people who really should know better have chosen the latter. I&rsquo;m not saying there isn&rsquo;t a place for that, but I think infrastructure that others depend on demands a measured, careful approach, and I try to live up to that whenever I can.</p>
<p>Practically speaking, whenever I use LLMs, I cannot help but tread slowly and carefully with them. I&rsquo;ve found them to be invaluable at achieving a level of rigor that I could previously only dream about, but the fact that models don&rsquo;t learn too much between sessions means they continue to make elementary mistakes. My work process makes me far slower in the moment than someone who one-shots software and moves on, but in the long run it lets me be <em>done</em> with a component, trusting that it works, and helps me build reliable systems out of well-designed parts. I hope that if you choose to use agents, this post inspires you to have a similar mindset.</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>From a systems design perspective, the release binaries are already arbitrary code, so <code>curl | sh</code> from the same origin doesn&rsquo;t introduce a <em>new</em> trust boundary. One could reasonably critique the process of publishing release binaries at all. I think that on balance, publishing them in an auditable, transparent fashion is the best approach.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p><em>Mukti</em> is a Sanskrit, and later Hindi, word that means freedom or liberation. It has a spiritual meaning of release similar to <em>moksha</em> or <em>nirvana</em>, but also a more temporal meaning; e.g., FOSS is sometimes called <em>mukt software</em> in Hindi.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>My go-to is Claude Code&rsquo;s bundled <code>/simplify</code> skill and the <a href="https://claude.com/marketplace/plugins/pr-review-toolkit"><code>pr-review-toolkit</code> plugin</a>, though I also ask the agents to do an architectural/negative space review.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>As a result of this process, I also built an inventory of crates that were written by myself or Oxide, or that I otherwise trust — the LLMs are now instructed to use crates from this inventory when possible. They manage to follow it most of the time.&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/nextest" term="nextest" label="nextest"/></entry><entry><title type="html">Why nextest is process-per-test</title><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/nextest-and-tokio/?utm_source=atom_feed" rel="related" type="text/html" title="How (and why) nextest uses tokio"/><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="related" type="text/html" title="Open and closed universes"/><id>https://sunshowers.io/posts/nextest-process-per-test/</id><published>2025-01-07T00:00:00+00:00</published><updated>2025-01-07T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>Why nextest runs each test in its own process: a game-theoretic view.</blockquote><p><em>I&rsquo;m often asked why the Rust test runner I maintain, <a href="https://nexte.st/">cargo-nextest</a>, runs every test in a separate process. Here&rsquo;s my best attempt at explaining the rationale behind it.</em></p>
<p><em>This document is cross-posted from the canonical copy at <a href="https://nexte.st/docs/design/why-process-per-test/">the nextest site</a>. Benchmarks <a href="https://nexte.st/docs/benchmarks/">are available</a> there as well, though a large part of this document talks about non-benchmarkable reliability improvements.</em></p>
<h1 id="why-process-per-test">Why process-per-test?</h1>
<p>A key factor distinguishing nextest from <code>cargo test</code> is that nextest runs <strong>each test in a separate process</strong>. With nextest, the default execution model is now, and will always be, process-per-test. This document explains why.</p>
<p>Some terms used in this document:</p>
<ul>
<li><em>Process-per-test:</em> Each test lives in its own process.</li>
<li><em>Shared-process:</em> Several tests share the same process.</li>
</ul>
<h2 id="the-ecosystem-coordination-challenge">The ecosystem coordination challenge</h2>
<p>The Rust ecosystem is now staggeringly large. There are millions of Rust developers and billions of lines of Rust code in production today. A key challenge with driving adoption of any new technology, like a new test runner, is <strong>coordination</strong>: everyone needs to agree on how to run and manage them.</p>
<ul>
<li>There are vast numbers of existing tests, representing millions of developer-hours of effort, that must continue to work</li>
<li>These tests integrate with all sorts of different libraries, including many not written in Rust, or with other unusual requirements</li>
<li>Test runners need to be able to run tests reliably, and rely on certain properties—for example, that killing one test will have no effect on others</li>
</ul>
<h2 id="schelling-points-in-game-theory">Schelling points in game theory</h2>

  <figure class="floatright" >
    
    <img src="/images/schelling.jpg"  alt="Photo of Nobel laureate Thomas Schelling, who first described focal (or Schelling) points."   />
    
    
      <figcaption class="center" >Thomas Schelling won the 2005 Nobel Prize in Economics for his work on focal points. <a href="https://www.bostonglobe.com/metro/obituaries/2016/12/17/thomas-schelling-nobel-winning-economist-who-influenced-nuclear-policy/1iMPQdz8NQFB75HwAPwWdM/story.html"><em>Boston Globe</em></a></figcaption>
    
  </figure>


<p><a href="https://en.wikipedia.org/wiki/Game_theory">Game theory</a> offers a powerful framework to study these kinds of coordination problems. Consider this classic problem: if two people need to meet in London on a particular day but, for whatever reason, can&rsquo;t coordinate the time and place, they&rsquo;re likely to choose noon at <a href="https://en.wikipedia.org/wiki/Big_Ben">Big Ben</a>. This becomes what game theorists call a <a href="https://en.wikipedia.org/wiki/Focal_point_%28game_theory%29"><em>focal point</em></a>, also known as a <em>Schelling point</em>—a natural default that everyone can assume without discussion.</p>
<p>The process-per-test model is powerful because <strong>it serves as the Schelling point</strong>, the Big Ben, of how to run tests. Just like it&rsquo;s commonly agreed that the Great Clock of Westminster is the most well-known landmark in London, software developers have all agreed that processes form natural fault boundaries. The process-per-test model forms a universal protocol that everyone in the ecosystem can rely on without explicit coordination.</p>
<h2 id="the-benefits-of-separate-processes">The benefits of separate processes</h2>
<p>This game-theoretic lens provides fresh insight into the benefits of process-per-test:</p>
<ul>
<li>
<p><strong>Test authors and libraries don&rsquo;t need to coordinate global state usage.</strong> There are a surprising number of real-world use cases that <em>require</em> per-test isolation—particularly integration tests against global state:</p>
<ul>
<li>Tests against a graphical API like <a href="https://www.khronos.org/egl">EGL</a>, which only allows one thread in a process to create a GPU context. <a href="https://github.com/gfx-rs/wgpu"><code>wgpu</code></a> uses nextest for this reason</li>
<li>Tests that <a href="https://nexte.st/docs/configuration/extra-args">must be run on the main thread</a> of their corresponding process, like tests against certain macOS frameworks</li>
<li>Tests that require altering environment variables or other global context</li>
</ul>
<p>Even if particular tests don&rsquo;t <em>require</em> per-test isolation, they may often <em>benefit</em> from it. For example, singletons become separated per-test. (You may argue that singletons aren&rsquo;t desirable, or should not be used for tests—but instead, think of per-test isolation as a principled, zero-coordination solution to this class of issues.)</p>
</li>
<li>
<p><strong>Internal memory handling doesn&rsquo;t need to be coordinated across tests.</strong> Memory corruption in one test doesn&rsquo;t cause others to behave erratically. One test segfaulting does not take down a bunch of other tests.</p>
</li>
<li>
<p><strong>Test runners don&rsquo;t need a custom protocol.</strong> With process-per-test, test runners can rely on universal OS primitives, like the ability to kill processes. For more discussion, see <a href="/posts/nextest-process-per-test/#the-costs-of-coordination"><em>The costs of coordination</em></a> below.</p>
</li>
</ul>
<p>Process-per-test is not free, though. It&rsquo;s worth acknowledging some costs:</p>
<ul>
<li>
<p>Tests might want to share in-memory state. They might have an in-memory semaphore for rate-limiting, a shared in-memory immutable cache that is primed on first access, or an in-memory server to which tests communicate via message passing. Process-per-test makes these valuable patterns <strong>harder to achieve</strong>: semaphores must <a href="https://nexte.st/docs/configuration/test-groups">be managed by</a> the test runner, in-memory state must be stored on disk, and shared servers must live out-of-process. Adapting existing tests to process-per-test may require a significant engineering effort.</p>
</li>
<li>
<p>Another downside that&rsquo;s noticeable in some situations is process creation <strong>performance</strong>. Creating processes is very fast on Linux and most other Unix-like systems. It is quite slow on Windows, though, and it can be slow on macOS <a href="https://nexte.st/docs/installation/macos#gatekeeper">if anti-malware protections are interfering</a>.</p>
<p>How much this is an issue in practice depends on how long individual tests take to run. For many projects on Windows, nextest is overall faster than <code>cargo test</code> anyway because it <a href="https://nexte.st/docs/design/how-it-works">handles long-pole tests</a> better.</p>
</li>
</ul>
<h2 id="the-costs-of-coordination">The costs of coordination</h2>
<p>Process-per-test enables rich lifecycle management with zero coordination via universal OS primitives. What kinds of coordination would enable shared-process tests?</p>
<p>Here&rsquo;s a partial list of operations that nextest performs using OS primitives:</p>
<ol>
<li><strong>Start a test.</strong> In particular, start a test at a specific moment, and not before then; and also, do not start a test if it is filtered out or the run is cancelled. With a process-per-test model, this is natural: start the test process.</li>
<li><strong>Know when a test is done, as it&rsquo;s done.</strong> With process-per-test, wait for the process to exit.</li>
<li><strong>Measure test times.</strong> Not just after a test is done, but while it is running. With process-per-test, use wall-clock time.</li>
<li><strong>Terminate tests that timed out.</strong> In particular, one test timing out should not cause others to be killed. With process-per-test, this is <a href="https://nexte.st/docs/design/architecture/signal-handling">solved</a> via signals on Unix and job objects on Windows.</li>
<li><strong>Retry failed tests.</strong> This is an extension to point 1: retrying tests, and marking tests as flaky if they succeeded later.</li>
<li><strong>Cancel tests on an input.</strong> On a test failure or receiving a signal, the test runner may decide to leave tests running but no longer schedule new ones, or to cancel existing tests.</li>
<li><strong>Gather test output.</strong> With process-per-test, nextest can read standard output and standard error for the process.</li>
</ol>
<p>Compare this to <code>cargo test</code>&rsquo;s shared-process model. <code>cargo test</code> works by embedding a small runner into each test binary which manages rate-limiting and filtering.</p>
<ul>
<li>
<p>However, as of early 2025, each runner works in isolation: there is no global orchestrator that can schedule tests across multiple binaries.</p>
</li>
<li>
<p>There is also no way to configure timeouts for tests, or to cancel some (but not all) running tests. For more, see <a href="/posts/nextest-process-per-test/#appendix">the appendix</a>.</p>
</li>
<li>
<p>If a test takes down the process, all of the other tests in the binary are effectively cancelled as well. In this case, getting a full accounting of all tests and what happened to them is extremely difficult. <a href="https://nexte.st/docs/machine-readable/junit">JUnit reports</a> wouldn&rsquo;t be nearly as valuable if they were missing many tests!</p>
</li>
</ul>
<p>Effective and robust management of tests in a shared-process model requires coordination between three entities:</p>
<ul>
<li><strong>The test runner:</strong> There must be an overall orchestrator, and for robustness it must live in a separate process from any of the test binaries.</li>
<li><strong>A component within the test binary:</strong> These requirements preclude any kind of one-shot model where the list of tests is provided upfront and run to completion. Instead, there must be an in-memory component that&rsquo;s dynamic and event-driven. There needs to be a rich, bidirectional communication protocol between the test runner and this component. (Simpler protocols like <a href="https://docs.rs/jobserver">the GNU make jobserver</a> can do rate-limiting, but not cancellation, retries, or time measurement.)</li>
<li><strong>The test itself:</strong> For cancellation in particular, the test must cooperate. For more on why, see <a href="/posts/nextest-process-per-test/#appendix">the appendix</a>.</li>
</ul>
<p>This represents a <em>lot</em> of extra work! Not just the technical kind (though that is certainly quite involved), but also the coordination kind: building support for such a protocol and getting developers to adopt it. And it would still not solve every problem. (For example, segfaults would still take down the whole process.)</p>
<h2 id="conclusion">Conclusion</h2>

  <figure class="floatright" >
    
    <img src="/images/big-ben.jpg"  alt="Photo of the clock on the Great Clock of Westminster, also known as Big Ben."   />
    
    
      <figcaption class="center" >Big Ben, the most recognizable landmark in London. <a href="https://unsplash.com/photos/big-ben-london-MdJq0zFUwrw"><em>Unsplash / Henry Be</em></a></figcaption>
    
  </figure>


<p>There are many technical benefits to the process-per-test model, but the biggest benefit is in (the lack of) coordination: the process-per-test model acts as a Schelling point that all participants can agree on by default. This is the key reason that nextest commits to this model being the default in perpetuity.</p>
<p>Nevertheless, we&rsquo;re excited to see developments in this space. We&rsquo;ll consider opt-ins for newer patterns that can deliver feature-rich and reliable test running at scale like nextest does today.</p>
<hr>
<h2 id="appendix">Appendix: thread cancellation is hard</h2>
<p>Many nextest users rely on its ability to cancel running tests due to timeouts. This section talks about why terminating threads is much more difficult than terminating processes.</p>
<ul>
<li>With process-per-test, nextest sends a <code>SIGTERM</code> signal to the test process, and if it doesn&rsquo;t exit within 10 seconds, a <code>SIGKILL</code>.</li>
<li>With a shared-process model, assuming each test is in a separate thread, a timeout would require the thread to be killed rather than the process.</li>
</ul>
<p>Unlike process termination, though, thread termination is inherently hazardous. The Windows <a href="https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminatethread"><code>TerminateThread</code></a> function explicitly warns against ever using it:</p>
<blockquote>
<p>TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination.</p>
</blockquote>
<p>With <code>TerminateThread</code> and the POSIX equivalent, <a href="https://man7.org/linux/man-pages/man3/pthread_cancel.3.html"><code>pthread_cancel</code></a>, the behavior of synchronization primitives like <code>std::sync::Mutex</code> is undefined. Certainly, Rust mutexes will not be marked <a href="https://doc.rust-lang.org/beta/std/sync/struct.Mutex.html#poisoning">poisoned</a>. It&rsquo;s possible (even likely) that mutexes are held forever and never released. Thread-killing operations would be devastating to test reliability.</p>
<p>Practically speaking, the only reliable way to cancel synchronous Rust code is with some kind of cooperation from the test itself. This could mean the test checking a flag every so often, or by the test regularly calling into a library that panics with a special payload on cancellation.</p>
<p>With async Rust, cancellation is <a href="https://docs.rs/tokio/latest/tokio/task/struct.JoinHandle.html#method.abort">somewhat easier</a>, because yield points are cancellation points. However, in an eerie similarity with <code>pthread_cancel</code>, <a href="https://docs.rs/tokio/latest/tokio/sync/struct.Mutex.html">Tokio mutexes</a> are <a href="https://docs.rs/cancel-safe-futures/0.1/cancel_safe_futures/sync/struct.RobustMutex.html"><em>also</em> not marked poisoned</a> on a task or future cancellation<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<p>It can be illuminating to contrast process termination with thread termination:</p>
<ul>
<li>Threads do not generally expect other threads to just disappear without warning, but processes are expected to sometimes exit abnormally.</li>
<li>Synchronization across processes is possible via functions like <a href="http://linux.die.net/man/2/flock"><code>flock</code></a>, but is often advisory and generally uncommon. Code that uses these tools is generally prepared to encounter protected data in an invalid state.</li>
<li>Cross-process mutexes are quite rare (message-passing is much more common), and shared-memory code is usually written with great care.</li>
</ul>
<p>These examples make clear how Schelling points manifest and perpetuate: we&rsquo;ve all generally agreed that other processes might behave in strange ways, but we assume that other threads within our process are going to behave reliably.</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>Based on our experiences with cancellation in other Rust projects, we strongly recommend Tokio mutexes be treated as a feature of last resort.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/nextest" term="nextest" label="nextest"/><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/></entry><entry><title type="html">How (and why) nextest uses tokio</title><link href="https://sunshowers.io/posts/nextest-and-tokio/?utm_source=atom_feed" rel="alternate" type="text/html"/><id>https://sunshowers.io/posts/nextest-and-tokio/</id><published>2022-10-03T00:00:00+00:00</published><updated>2022-11-03T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>How my open source test runner uses async Rust.</blockquote><div class="sectioned-post">
<h2 id="1-introduction">1. Introduction</h2>
<p><em>Update 2022-11-03: This was originally the first in a series of two blog posts. I&rsquo;ve now marked this as a standalone post. Content originally slated for part 2 will now be published as followup posts.</em></p>
<p>I&rsquo;m the primary author and maintainer of <a href="https://nexte.st/">cargo-nextest</a>, a next-generation test
runner for Rust released earlier this year. (It reached a thousand stars on GitHub recently; <a href="https://github.com/nextest-rs/nextest">go
star it here!</a>)</p>
<div class="center">
<p><a href="https://github.com/nextest-rs/nextest"><img src="https://img.shields.io/github/stars/nextest-rs/nextest?style=social" alt="GitHub stars shield"></a></p>
</div>
<p>Nextest is <a href="https://nexte.st/book/benchmarks.html">faster than <code>cargo test</code></a> for most Rust projects,
and provides a number of extra features, such as <a href="https://nexte.st/book/retries">test retries</a>,
<a href="https://nexte.st/book/reusing-builds">reusing builds</a>, and <a href="https://nexte.st/book/partitioning">partitioning (sharding) test
runs</a>. To power these features, nextest uses
<a href="https://tokio.rs/">Tokio</a>, the leading Rust async runtime for writing network applications.</p>
<p><em>Wait, what?</em> Nextest isn&rsquo;t a network app at all<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. However, nextest has to juggle
multiple concurrent tests, each of which can be in the process of starting, printing output,
failing, timing out, being cancelled or even forcibly stopped. In this post, I&rsquo;ll try to convince
you that an async runtime is actually the perfect approach to handle a large number of heterogenous
events like this.</p>
<h3 id="theres-more-than-one-kind-of-distributed-system">There&rsquo;s more than one kind of distributed system</h3>
<p>The challenges posed by interprocess communication <em>within a system</em> are really similar to those
posed by remote communication <em>across systems</em>. Your computer is <a href="http://catern.com/compdist.html">a distributed
system</a>. Just like most network services, nextest spends almost all
of its time waiting rather than computing.</p>
<p>In other words, nextest provides the prototypical use-case for <em>asynchronous concurrency</em>, one of
the two main paradigms for non-sequential code.</p>

  <figure class="center" >
    
    <img src="/images/io-compute-bound.png"  alt="A simple diagram showing the two main paradigms for non-sequential code. To the left is &#39;I/O-bound&#39; with &#39;Async concurrency (Tokio)&#39;, and to the right is &#39;Compute-bound&#39; with &#39;Data Parallelism (Rayon)&#39;"   />
    
    
  </figure>


<ul>
<li>The async concurrency style is best for I/O-bound applications, where most time is spent waiting
on other parts of the overall system to respond. An I/O-bound application becomes faster if I/O
becomes faster&mdash;for example, if your network link becomes faster, or you replace a spinning disk
hard drive with a solid-state drive.</li>
<li>On the other end of the spectrum, <em>data parallelism</em> with libraries like
<a href="https://github.com/rayon-rs/rayon">Rayon</a> is best for compute-bound applications, where you&rsquo;re
running a lot of computations in parallel on a set of input data. These become faster if you replace
your CPU with a faster one.</li>
</ul>
<p>Why is an async runtime like Tokio good for I/O-bound applications? The answer has to do with
<em>heterogenous selects</em><sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p>
<h3 id="what-are-heterogenous-selects">What are heterogenous selects?</h3>
<p>You might have heard of the <em>select</em> operation: in computer parlance, it means to observe
two or more operations, and then to proceed with the first one that completes.</p>
<p>The name of the operation comes from the Unix
<a href="https://man7.org/linux/man-pages/man2/select.2.html"><code>select</code></a> syscall. With <code>select</code> and its successors
like <a href="https://man7.org/linux/man-pages/man7/epoll.7.html"><code>epoll</code></a>, you can typically only wait on one of
several I/O read or write operations (represented through file descriptors). But real-world
I/O-bound programs need to do much more than that! Here are some other kinds of operations they may
need to wait on:</p>
<ul>
<li>Timeouts or recurring timers</li>
<li><a href="https://tokio.rs/tokio/tutorial/channels">Channel reads and writes</a> when using message-passing,
including bounded channels with backpressure</li>
<li>Process exits</li>
<li>Unix signals</li>
</ul>
<p>I/O-bound programs often need to wait across several of these sources at the same time. For example,
it&rsquo;s common for services to make a network request, then select across:</p>
<ul>
<li>a network response</li>
<li>a timeout</li>
<li>a termination signal, so any pending network operations can be canceled</li>
</ul>
<p>This is an example of a <em>heterogenous select</em> (or, spelled slightly differently, a <em>heterogeneous select</em>): a select operation that can span several different
kinds of sources.</p>
<hr>
<p>We also run into portability considerations: <code>select</code> is Unix-specific, and <code>epoll</code> is for the most
part Linux-only<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>. Other operating systems have their own <code>epoll</code>-like abstractions, each
with their own peculiarities:</p>
<ul>
<li>BSD-like platforms have <code>kqueue</code> (<a href="https://www.freebsd.org/cgi/man.cgi?kqueue">FreeBSD</a>, <a href="https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/kqueue.2.html">macOS</a>).</li>
<li>Windows has <a href="https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports">I/O completion ports</a>.</li>
</ul>
<p>Some platforms may also support selecting across some kinds of heterogenous sources. For example,
Linux has <a href="https://man7.org/linux/man-pages/man2/signalfd.2.html"><code>signalfd</code></a> and
<a href="https://man7.org/linux/man-pages/man2/timerfd_create.2.html"><code>timerfd</code></a>. To the extent that these
are supported, there are significant differences in the APIs available on each platform. And all of
them make selecting across channels and other user code quite difficult.</p>
<p>The divergence in platform APIs and capabilities may not be a huge deal for certain kinds of network
services that only target one platform, but is a real problem for a cross-platform developer tool
like nextest.</p>
<hr>
<p>So there are two separate but related sets of problems here:</p>
<ol>
<li>The need to select over arbitrary kinds of heterogenous sources, not just I/O operations.</li>
<li>The need to do so in a cross-platform manner.</li>
</ol>
<p>Async runtimes like Tokio solve both sets of problems in one go<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>. In fact, I&rsquo;d like to make a bold claim here:</p>
<p><strong>The ability to do heterogenous selects is <em>the point of async Rust</em>.</strong></p>
<div class="framed">
<h3 id="async-runtimes-vs-asyncawait">Async runtimes vs async/await</h3>
<p>At this point it&rsquo;s worth mentioning that there are two related, but distinct, things here:</p>
<ol>
<li>An <em>async runtime</em>: a framework that manages an event loop, handling events from heterogenous sources and associating them with individual tasks.</li>
<li>The <em><code>async</code> and <code>await</code> keywords</em>, which provide an easy way to write tasks that wait on events handled by an async runtime.</li>
</ol>
<p>It is definitely possible to leverage async runtimes without ever using the <code>async</code> and <code>await</code> keywords&mdash;in fact, I&rsquo;ve written large chunks of a production service in Rust that used an async runtime before <code>async</code>/<code>await</code> was ever a thing.</p>
<p>However, without syntactic support it is often very difficult to do so, since you may need to hand-write complicated state machines. The <code>async</code> and <code>await</code> keywords dramatically simplify the experience of writing state machines. David Tolnay talks about this more <a href="https://docs.rs/dtolnay/latest/dtolnay/macro._01__await_a_minute.html#native-control-flow">in his blog post</a>. (The <code>poll_next</code>/<code>EncodeState</code> example in his blog post was part of the aforementioned production service, and it was my code that he rewrote.)</p>
</div>
<h2 id="2-the-initial-implementation">2. The initial implementation</h2>
<p>With this background, we&rsquo;re now ready to start looking at how nextest&rsquo;s current architecture came about.</p>
<p>The first versions of nextest all ran tests using &ldquo;normal&rdquo; threads. Here&rsquo;s the general structure
they used for their <a href="https://nexte.st/book/how-it-works.html">execution model</a>:</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-flowchart.png"  alt="A flowchart showing the main ways data moves around among nextest&#39;s components. See below for a text description."   />
    
    
      <figcaption class="center" >The main ways data moves around among nextest&rsquo;s components.</figcaption>
    
  </figure>


<p><em>The run pool:</em></p>
<p>Tests were run via a thread pool.</p>
<ol>
<li>Nextest first <a href="https://nexte.st/book/how-it-works.html#the-nextest-model">built up</a> a list of tests.</li>
<li>It then spun up a thread pool with the number of tests<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>, called the <em>run pool</em>.</li>
<li>For each test in the list, nextest used a thread in the run pool. The thread ran the test using
the <a href="https://docs.rs/duct/">duct</a> library, then blocked until the test exited. (nextest used duct for convenience&mdash;we could have used <code>std::process::Command</code> instead, which wouldn&rsquo;t have changed much.)</li>
<li>To send information about test events (the start, end, success/failure, and time taken for
individual tests) to the main thread, nextest used a <a href="https://docs.rs/crossbeam-channel/0.5/crossbeam_channel/">crossbeam MPSC
channel</a><sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup>: we&rsquo;ll call this the
<em>test event channel</em>.</li>
</ol>
<p><em>The signal thread:</em></p>
<p>To handle signals such as Ctrl-C cleanly, nextest spun up a separate thread, and if a signal is
detected, wrote to another channel: the <em>signal event channel</em>.</p>
<p><em>The main thread:</em></p>
<p>The main thread was dedicated to monitoring the two event channels. This thread would
<a href="https://docs.rs/crossbeam-channel/0.5/crossbeam_channel/macro.select.html">select</a> across
the <a href="https://github.com/nextest-rs/nextest/blob/ce98c9405bfe512e2f411d78bfe04468722ae69b/nextest-runner/src/runner.rs#L267-L287"><em>test event</em> and <em>signal event</em>
channels</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// InternalEvent has this definition:
</span></span></span><span class="line"><span class="cl"><span class="cp">#[derive(Debug)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">enum</span> <span class="nc">InternalEvent</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Test</span><span class="p">(</span><span class="n">InternalTestEvent</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Signal</span><span class="p">(</span><span class="n">SignalEvent</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Note that receiving an Err() from a receiver (a RecvError) means that there are
</span></span></span><span class="line"><span class="cl"><span class="c1">// no more senders.
</span></span></span><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">crossbeam_channel</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">recv</span><span class="p">(</span><span class="n">run_receiver</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">internal_event</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// internal_event is a Result&lt;InternalTestEvent, RecvError&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Ok</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Test</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// All senders were dropped -- the test run completed.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">recv</span><span class="p">(</span><span class="n">signal_receiver</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">internal_event</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// internal_event is a Result&lt;SignalEvent, RecvError&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Ok</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Signal</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// Ignore the signal thread being dropped. This is done for
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// noop signal handlers.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">continue</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ... process internal_event
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<ul>
<li>If this loop received a test event, it would bubble that up to the reporter (the component that
prints nextest&rsquo;s output). If a test failed, it would set a cancellation flag which would cause no
further tests to be scheduled.</li>
<li>If the loop received a signal event, it would set the cancellation flag.</li>
</ul>
<p>Importantly, <code>crossbeam_channel::select!</code> only lets you select over crossbeam channels, not
heterogenous sources in general. Anything else, including pipe reads or timers, must be translated
over to crossbeam channels somehow.</p>
<h3 id="the-first-issues-with-the-model">The first issues with the model</h3>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-with-wait.png"  alt="A flowchart showing how nextest&#39;s runner loop detected slow tests. See below for a text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s original runner loop detected slow tests.</figcaption>
    
  </figure>


<p>One of the first demands placed on nextest was the need to identify <a href="https://nexte.st/book/slow-tests">slow
tests</a>. Note that duct&rsquo;s
<a href="https://docs.rs/duct/0.13/duct/struct.Handle.html#method.wait"><code>Handle::wait</code></a>
method doesn&rsquo;t have any notion of a timeout: it just blocks until the process is completed.</p>
<p>Luckily, <code>crossbeam_channel</code> has a <a href="https://docs.rs/crossbeam-channel/0.5/crossbeam_channel/struct.Receiver.html#method.recv_timeout"><code>recv_timeout</code>
method</a>.
So the solution was straightforward: set up another thread pool, called the <em>wait pool</em>, which would wait on test processes to exit.</p>
<p>So now this becomes:</p>
<ol>
<li>Start the test process using a thread in the run pool.</li>
<li>Spawn a wait pool thread to wait on the test.</li>
<li>In the run pool, wait on the thread in the wait pool with a timeout.
<ul>
<li>If the test finishes before the timeout, mark it done.</li>
<li>If the timeout is hit, mark the test slow and go back to waiting on the test.</li>
</ul>
</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/5b82369cb42209c7f513ee686faa3dd340d971b7/nextest-runner/src/runner.rs#L358-L389">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cmd</span><span class="p">.</span><span class="n">start</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="bp">self</span><span class="p">.</span><span class="n">wait_pool</span><span class="p">.</span><span class="n">in_place_scope</span><span class="p">(</span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">sender</span><span class="p">,</span><span class="w"> </span><span class="n">receiver</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">crossbeam_channel</span>::<span class="n">bounded</span>::<span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">wait_handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="n">handle</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Spawn a task on the threadpool that waits for the test to finish.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">s</span><span class="p">.</span><span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">|</span><span class="n">_</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">wait_handle</span><span class="p">.</span><span class="n">wait</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">sender</span><span class="p">.</span><span class="n">send</span><span class="p">(());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Continue waiting for the test to finish with a timeout, logging at
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// slow-timeout intervals.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">while</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">error</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">receiver</span><span class="p">.</span><span class="n">recv_timeout</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">slow_timeout</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">match</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">RecvTimeoutError</span>::<span class="n">Timeout</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// ... bubble up the slow event so the reporter can print out a
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// SLOW line
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">RecvTimeoutError</span>::<span class="n">Disconnected</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="fm">unreachable!</span><span class="p">(</span><span class="s">&#34;Waiting thread should never drop the sender&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">into_output</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ... process output
</span></span></span></code></pre></div></details>
<p>This works, but you can already see the cracks starting to show. To solve this issue we had to
introduce a whole new thread pool on top of the existing run pool.</p>
<h3 id="terminating-tests-using-a-polling-loop">Terminating tests using a polling loop</h3>
<p>A few months after nextest was released, a user requested another feature: the ability to terminate
tests that take more than a certain amount of time.</p>
<p>Now, nextest <em>could</em> terminate tests immediately with the unblockable <a href="https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html"><code>SIGKILL</code>
signal</a><sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup>
(aka <code>kill -9</code>). But it&rsquo;s generally a good idea to give tests a bit of a grace period. So we chose
to first send tests a <code>SIGTERM</code> signal, then wait a bit for them to clean up before sending
<code>SIGKILL</code>.</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-with-wait-and-timeout.png"  alt="A flowchart to show how nextest&#39;s runner loop implemented test timeouts. See below for text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s original runner loop implemented test timeouts.</figcaption>
    
  </figure>


<p>To do this, we defined two timeouts:</p>
<ol>
<li>A <em>slow timeout</em>, after which tests are marked as slow. Nextest&rsquo;s default slow timeout is 60 seconds.</li>
<li>A <em>termination timeout</em>, after which tests are killed. Nextest doesn&rsquo;t have a default termination timeout, but a typical one users might set is 5 minutes (300 seconds).</li>
</ol>
<p>The test execution flow is then:</p>
<ol>
<li>Start the test process using a thread in the run pool.</li>
<li>Spawn a wait pool thread to wait on the test.</li>
<li>In the run pool, wait on the thread in the wait pool with the slow timeout.
<ul>
<li>If the test finishes before the slow timeout, mark it done.</li>
<li>If the slow timeout is hit and the termination timeout hasn&rsquo;t been hit yet, mark the test as slow and go back to waiting on it.</li>
<li>If the termination timeout is hit, set up a termination loop.</li>
</ul>
</li>
</ol>
<p>And the termination loop:</p>
<ol>
<li>Send the process <code>SIGTERM</code>.</li>
<li>In a loop, for up to 10 seconds:
<ul>
<li>Check if the test process exited.</li>
<li>If it did, break out of the loop.</li>
<li>Otherwise, wait 100ms and check the test&rsquo;s status again.</li>
</ul>
</li>
<li>If the test hasn&rsquo;t exited by the end of the loop, send it <code>SIGKILL</code>.</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/ce98c9405bfe512e2f411d78bfe04468722ae69b/nextest-runner/src/runner.rs#L412-L473">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">match</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">RecvTimeoutError</span>::<span class="n">Timeout</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// ... bubble up the slow event to the reporter
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">terminate_after</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slow_timeout</span><span class="p">.</span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">if</span><span class="w"> </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// Attempt to terminate the slow test.
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// As there is a race between shutting down a slow test and its
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// own completion, we silently ignore errors to avoid printing
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// false warnings.
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="cp">#[cfg(unix)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="kd">let</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">use</span><span class="w"> </span><span class="n">duct</span>::<span class="n">unix</span>::<span class="n">HandleExt</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">use</span><span class="w"> </span><span class="n">libc</span>::<span class="no">SIGTERM</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">send_signal</span><span class="p">(</span><span class="no">SIGTERM</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// give the process a grace period of 10s
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="k">match</span><span class="w"> </span><span class="p">((</span><span class="mi">0</span><span class="o">..</span><span class="mi">100</span><span class="p">).</span><span class="n">contains</span><span class="p">(</span><span class="o">&amp;</span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">try_wait</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="n">_</span><span class="p">,</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Some</span><span class="p">(</span><span class="n">_</span><span class="p">)))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">None</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">None</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="c1">// in case of an error we&#39;ll send SIGKILL nonetheless
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="n">_</span><span class="p">,</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="n">i</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="n">std</span>::<span class="n">thread</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_millis</span><span class="p">(</span><span class="mi">100</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="cp">#[cfg(not(unix))]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="kd">let</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">exited</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">kill</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ExecutionResult</span>::<span class="n">Timeout</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<p>You can already start seeing the code becoming long and rather janky, with its 100ms polling loop.</p>
<h3 id="detecting-leaked-handles">Detecting leaked handles</h3>
<p>Some tests spawn server or other subprocesses, and sometimes those subprocesses don&rsquo;t get cleaned up
at the end of the test. In particular, nextest is concerned about processes that
<a href="https://doc.rust-lang.org/std/process/struct.Stdio.html#method.inherit"><em>inherit</em></a> standard output
and/or standard error. Here&rsquo;s an example test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[test]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">test_subprocess_doesnt_exit</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Spawn a subprocess that sleeps for 2 minutes, then exits.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cmd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span>::<span class="n">process</span>::<span class="n">Command</span>::<span class="n">new</span><span class="p">(</span><span class="s">&#34;sleep&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">cmd</span><span class="p">.</span><span class="n">arg</span><span class="p">(</span><span class="s">&#34;120&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// This sleep command inherits standard output and standard error from the test.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">cmd</span><span class="p">.</span><span class="n">spawn</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Since <code>sleep</code> inherits standard output and standard error from the test, it keeps those file
descriptors open until it exits. Previously, nextest would simply wait until those subprocesses
exited. In the worst case, the subprocesses wouldn&rsquo;t exit at all: think of a server process that
gets started but never shut down. That can result in nextest simply hanging, which isn&rsquo;t great.</p>
<p>How can nextest detect <a href="https://nexte.st/book/leaky-tests.html">these sorts of <em>leaky</em> tests</a>? One
solution is to wait for standard output and standard error to be closed until a short duration (say
100ms) passes. While this is conceptually simple, <em>implementing it</em> is surprisingly complicated.
Waiting on one handle being closed is nontrivial; waiting on <em>two</em> handles being closed (standard
output and standard error) adds an extra challenge.</p>
<p>I spent some time looking at how to implement it manually, and came away from it with the
understanding that I&rsquo;d have to maintain separate implementations for Windows and Unix. Not only
that, I&rsquo;d just be duplicating a lot of the work Tokio had already done to abstract over platform
differences. So I started looking at porting over nextest to Tokio.</p>
<h2 id="3-using-tokio">3. Using Tokio</h2>
<p>I spent a weekend porting over nextest to use async Rust with Tokio. After a bit of iteration, I
settled on this general architecture:</p>
<ol>
<li>Create a <a href="https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html">Tokio <code>Runtime</code></a> in the
beginning. The <code>Runtime</code> lasts for as long as the runner loop does.</li>
<li>Turn the list of tests into a
<a href="https://docs.rs/futures/0.3/futures/stream/trait.Stream.html"><code>Stream</code></a>, then for each test,
create a future that executes the test. Use
<a href="https://docs.rs/futures/0.3/futures/stream/trait.StreamExt.html#method.buffer_unordered"><code>buffer_unordered</code></a>
to control concurrency<sup id="fnref:8"><a href="#fn:8" class="footnote-ref" role="doc-noteref">8</a></sup>.</li>
<li>Switch over crossbeam channels to <a href="https://docs.rs/tokio/1/tokio/sync/mpsc/index.html">tokio
channels</a> as needed, and
<code>crossbeam_channel::select!</code> invocations <a href="https://docs.rs/tokio/1/tokio/macro.select.html">to
<code>tokio::select!</code></a>.</li>
</ol>
<p>Here&rsquo;s the bit that <a href="https://github.com/nextest-rs/nextest/blob/8703af50b649fc882e601d4eb01f7c2c104c0395/nextest-runner/src/runner.rs#L325-L347">selects over the test event and signal event channels</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">signals_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">run_receiver</span><span class="p">.</span><span class="n">recv</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// internal_event is an Option&lt;InternalTestEvent&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Some</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Test</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">None</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// All senders were dropped -- the test run completed.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">signal_handler</span><span class="p">.</span><span class="n">recv</span><span class="p">(),</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">signals_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// internal_event is an Option&lt;SignalEvent&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Some</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Signal</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">None</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="n">signals_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">continue</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ... process internal_event
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<p>This is really similar to the crossbeam-based implementation above, with just one real change: The
code tracks a <code>signals_done</code> state to see if the signal event channel has been dropped. This is to
ensure that the loop doesn&rsquo;t repeatedly get <code>None</code> values from the signal handler, wasting cycles.
(This is more important with futures that, once completed, cannot be waited on further, as we&rsquo;ll
see in the next section.)</p>
<h3 id="waiting-on-process-completion-with-tokio">Waiting on process completion with Tokio</h3>
<p>Where this gets really interesting is with the loop nextest uses to wait for tests to complete. It
turns out that <em>all</em> the code we wrote above can be replaced with a set of Rust and Tokio
primitives, written in a roughly declarative style.</p>
<p>After starting the process, nextest <a href="https://github.com/nextest-rs/nextest/blob/e11c35547511e728a422cabdcd04ea542df3bef5/nextest-runner/src/runner.rs#L600-L648">sets
up</a>
a timer, and futures to read from standard output and standard error:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Set up a timer to produce events at predefined intervals.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">time</span>::<span class="n">interval</span><span class="p">(</span><span class="n">slow_timeout</span><span class="p">.</span><span class="n">period</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// The first tick is immediate.
</span></span></span><span class="line"><span class="cl"><span class="n">interval</span><span class="p">.</span><span class="n">tick</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">child_stdout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child</span><span class="p">.</span><span class="n">stdout</span><span class="p">.</span><span class="n">take</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">child_stderr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child</span><span class="p">.</span><span class="n">stderr</span><span class="p">.</span><span class="n">take</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stdout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bytes</span>::<span class="n">BytesMut</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stderr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bytes</span>::<span class="n">BytesMut</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">read_all_to_bytes</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">bytes</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">bytes</span>::<span class="n">BytesMut</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">mut</span><span class="w"> </span><span class="n">input</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">(</span><span class="k">dyn</span><span class="w"> </span><span class="n">AsyncRead</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Unpin</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">std</span>::<span class="n">io</span>::<span class="nb">Result</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Reborrow it as AsyncReadExt::read_buf expects
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Sized self.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">input</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">bytes</span><span class="p">.</span><span class="n">reserve</span><span class="p">(</span><span class="mi">4096</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">bytes_read</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="n">read_buf</span><span class="p">(</span><span class="n">bytes</span><span class="p">).</span><span class="k">await</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="n">bytes_read</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Set up futures for reading from stdout and stderr.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">stdout_fut</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stdout</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child_stdout</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">read_all_to_bytes</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stdout</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stdout</span><span class="p">).</span><span class="k">await</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">tokio</span>::<span class="fm">pin!</span><span class="p">(</span><span class="n">stdout_fut</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">stderr_fut</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stderr</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child_stderr</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">read_all_to_bytes</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stderr</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stderr</span><span class="p">).</span><span class="k">await</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">tokio</span>::<span class="fm">pin!</span><span class="p">(</span><span class="n">stderr_fut</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div></details>
<p>Some notes about the code above:</p>
<ul>
<li>On being polled, the futures repeatedly read standard output and standard error into their
corresponding buffers in 4 KiB chunks. They only exit when 0 bytes are read, which corresponds
to the handle being closed.</li>
<li>The code needs to maintain <code>stdout_done</code> and <code>stderr_done</code> flags: this is similar to
<code>signals_done</code> above, except this time it&rsquo;s necessary because async blocks cannot be polled once
completed: if done so, they panic with the message &ldquo;<code>async fn</code> resumed after completion&rdquo;.</li>
<li>The futures borrow data from the stack. This is possible thanks to async Rust&rsquo;s integration with the
borrow checker.</li>
<li>The futures are <a href="https://docs.rs/tokio/1/tokio/macro.pin.html"><em>pinned</em> to the stack</a>, which means
that they cannot be moved. This is a <a href="https://rust-lang.github.io/async-book/04_pinning/01_chapter.html">fundamental
part</a> of how zero-overhead
async Rust works. An async block must be pinned before it can be polled.</li>
</ul>
<p>Remember how we talked about heterogenous selects in part 1 of this post? This is where we start
using them. Notice how <code>crossbeam_channel::select!</code> only works on crossbeam channels. On the other
hand, Tokio&rsquo;s select works on arbitrary, heterogenous sources of asynchronicity.</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-tokio.png"  alt="A flowchart to show how nextest&#39;s Tokio-based runner loop executes tests. See below for text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s Tokio-based runner loop executes tests.</figcaption>
    
  </figure>


<p>Here&rsquo;s how nextest now waits on tests:</p>
<p><em>Setup:</em></p>
<ol>
<li>Start the test process, which sets up a future that completes when the test exits.</li>
<li>Set up a timer for the slow timeout.</li>
<li>Set up futures to read from standard output and standard error.</li>
</ol>
<p>Then, select across the futures listed above in a loop, using Tokio&rsquo;s ability to handle heterogenous
selects with <code>tokio::select!</code>.</p>
<p><em>If data is available via the standard output or standard error futures:</em></p>
<ol>
<li>Read data into the corresponding buffer.</li>
<li>Loop back and select again.</li>
</ol>
<p><em>If the timer fires:</em></p>
<ol>
<li>If the terminate timeout is hit, terminate the test and exit the loop.</li>
<li>If not, mark as slow, loop back and select again.</li>
</ol>
<p><em>If the process exits:</em></p>
<ol>
<li>Exit the execution loop.</li>
<li>Proceed to the leak detection loop below.</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/8703af50b649fc882e601d4eb01f7c2c104c0395/nextest-runner/src/runner.rs#L533-L570">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// res is an std::io::Result&lt;ExitStatus&gt;.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stdout_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stderr_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child</span><span class="p">.</span><span class="n">wait</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// The test finished executing.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="n">res</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">interval</span><span class="p">.</span><span class="n">tick</span><span class="p">(),</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">status</span><span class="p">.</span><span class="n">is_none</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">is_slow</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// ... bubble up the slow event to the reporter
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">terminate_after</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slow_timeout</span><span class="p">.</span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">//  ... attempt to terminate the slow test
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ExecutionResult</span>::<span class="n">Timeout</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<hr>
<p>And then, how does nextest detect leaked handles? After the main loop ends, nextest just runs another little loop
for that:</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-tokio-leaks.png"  alt="A flowchart to show how nextest&#39;s Tokio-based runner loop detects leaked handles in tests. See below for text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s Tokio-based runner loop detects leaked handles in tests.</figcaption>
    
  </figure>


<ol>
<li>Set up a short (by default 100ms) leak timer.</li>
<li>Select across the leak timer and the standard output and error futures from before, in a loop.</li>
</ol>
<p><em>If data is available via the standard output or standard error futures:</em></p>
<ol>
<li>Read data into the corresponding buffer.</li>
<li>Loop back, and select again.</li>
</ol>
<p><em>If both the standard output and standard error futures have completed:</em></p>
<ol>
<li>Note that the test didn&rsquo;t leak.</li>
<li>Conclude test execution.</li>
</ol>
<p><em>If the timer fires before both futures have completed:</em></p>
<ol>
<li>Note that the test leaked handles.</li>
<li>Conclude test execution.</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/8703af50b649fc882e601d4eb01f7c2c104c0395/nextest-runner/src/runner.rs#L582-L609">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Once the process is done executing, wait up to leak_timeout for the pipes to shut
</span></span></span><span class="line"><span class="cl"><span class="c1">// down. Previously, this used to hang if spawned grandchildren inherited
</span></span></span><span class="line"><span class="cl"><span class="c1">// stdout/stderr but didn&#39;t shut down properly. Now, nextest detects those cases and
</span></span></span><span class="line"><span class="cl"><span class="c1">// marks them as leaked.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">leaked</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">sleep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">time</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">leak_timeout</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stdout_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stderr_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">()</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sleep</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="p">(</span><span class="n">stdout_done</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="n">stderr_done</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// stdout and/or stderr haven&#39;t completed yet. In this case, break
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// the loop and mark this as leaked.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">else</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span></code></pre></div></details>
<p>That&rsquo;s it! We&rsquo;ve replaced all of that unwieldy code with a couple of declarative <code>select!</code> loops.</p>
<div class="framed">
<h3 id="if-it-compiles-does-it-work">If it compiles, does it work?</h3>
<p>One of the promises of Rust&rsquo;s type system is that it&rsquo;ll catch most kinds of bugs before you ever run
the program. In my experience, this promise is <em>mostly</em> upheld, even when dealing with async code.
For a few compile-time issues like pinning, it was easy to find
<a href="https://tokio.rs/tokio/tutorial/select#resuming-an-async-operation">solutions</a>.</p>
<p>I ran into two runtime issues:</p>
<ol>
<li>A panic with the message &ldquo;<code>async fn</code> resumed after completion&rdquo;. The solution to this
was <a href="https://tokio.rs/tokio/tutorial/select#modifying-a-branch">easy to find</a>.</li>
<li>The main loop that <a href="/posts/nextest-and-tokio/#cutting-the-knot">selected across</a> the test event and signal event channels
hung during longer test runs; this was caused by a <a href="https://docs.rs/tokio/latest/tokio/macro.select.html#fairness"><code>biased</code>
statement</a>. Removing the <code>biased</code>
fixed it, but I mostly found that out by trial and error. I&rsquo;m not sure how to debug this kind of
issue in a principled manner.</li>
</ol>
</div>
<h2 id="4-conclusion">4. Conclusion</h2>
<p>There&rsquo;s a bit of a meme in the Rust world: that if you can &ldquo;just use&rdquo; threads, <a href="https://news.ycombinator.com/item?id=31601973">you
should</a>; and that async code is really only &ldquo;worth
it&rdquo; for network services that <a href="https://www.reddit.com/r/rust/comments/v6rx10/comment/ibh2nwz">need to be massively
concurrent</a>. (I don&rsquo;t mean to pick on
the individual authors here. It&rsquo;s a general view in the Rust community.)</p>
<p>While I agree that async Rust isn&rsquo;t <em>easy</em>, I believe nextest&rsquo;s experience using tokio directly
challenges that general view. The point of async, even more than the concurrency, is to make it
easy to operate across arbitrary sources of asynchronicity. This is exactly what nextest needs to
do. Any Rust program that is I/O-bound, or any program that needs to do heterogenous selects,
should strongly consider using an async runtime like Tokio.</p>
<h3 id="thanks">Thanks</h3>
<p>Thanks to <a href="https://twitter.com/munin">Fiona</a>, <a href="https://twitter.com/inanna_malick">Inanna</a>,
<a href="https://twitter.com/mycoliza/">Eliza</a>, <a href="https://twitter.com/bmwill_">Brandon</a>,
<a href="https://twitter.com/PredragGruevski">Predrag</a>, and <a href="https://twitter.com/mgattozzi">Michael</a> for
reviewing drafts of this article. And thank you for reading it to the end.</p>
<p>The flowcharts are available in <a href="https://drive.google.com/drive/folders/1znKowkG_58v7oo4zoTxHPF0wBd6OkTyt?usp=sharing">this Google folder</a>, and are licensed under <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>.</p>
<h4 id="changelog">Changelog</h4>
<p><em>2022-10-06:</em></p>
<ul>
<li>Add a note that platform-specific APIs like <code>signalfd</code> and <code>timerfd</code> can provide heterogenous selects to some extent, though not as much as Tokio can.</li>
<li>Updated <code>stdout_fut</code> and <code>stderr_fut</code> implementations to match changes in <a href="https://github.com/nextest-rs/nextest/pull/560">#560</a>.</li>
<li>I&rsquo;d forgotten to add Michael Gattozzi to the reviewers list. Fixed, sorry!</li>
</ul>
</div>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>Other than its <a href="https://nexte.st/book/updating.html">self-update</a> functionality,
which isn&rsquo;t relevant here.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>Full credit for the term and framing goes to <a href="https://twitter.com/mycoliza">Eliza Weisman</a>.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>Some other systems have added <code>epoll</code> for compatibility with Linux: for example,
<a href="https://illumos.org/man/7/epoll">illumos</a>.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>The platform-specific implementations are usually abstracted away with a low-level library
like <a href="https://github.com/tokio-rs/mio">Mio</a>.&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5">
<p>The runner loop needed a scoped thread pool because it relied on borrowed data&mdash;it
used <a href="https://github.com/rayon-rs/rayon">rayon</a>&rsquo;s
<a href="https://docs.rs/rayon/1/rayon/struct.ThreadPool.html"><code>ThreadPool</code></a> for that purpose.&#160;<a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6">
<p>MPSC stands for &ldquo;multi-producer, single consumer&rdquo;. This kind of channel lets many threads
(in this case, the threads running the tests) send data to a single &ldquo;status&rdquo; thread.&#160;<a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7">
<p>Or its equivalent on Windows,
<a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess">TerminateProcess</a>.&#160;<a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:8">
<p>Why <code>buffer_unordered</code>? Because we&rsquo;d like tests to be <em>started</em> in lexicographic order,
but results to be reported in the order tests finish in. That is exactly the behavior
<code>buffer_unordered</code> provides.</p>
<p>One alternative was to spawn every test as a separate future and use a
<a href="https://docs.rs/tokio/1/tokio/sync/struct.Semaphore.html">semaphore</a> to control concurrency,
but that might have resulted in tests being started in arbitrary order.&#160;<a href="#fnref:8" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/nextest" term="nextest" label="nextest"/><category scheme="https://sunshowers.io/tags/tokio" term="tokio" label="tokio"/><category scheme="https://sunshowers.io/tags/case-study" term="case-study" label="case-study"/></entry></feed>