<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us"><generator uri="https://gohugo.io/" version="0.170.0">Hugo</generator><title type="html">Case-Study on sunshowers</title><subtitle type="html">thoughts and ideas</subtitle><link href="https://sunshowers.io/tags/case-study/" rel="alternate" type="text/html" title="html"/><link href="https://sunshowers.io/tags/case-study/index.xml" rel="alternate" type="application/rss+xml" title="rss"/><link href="https://sunshowers.io/tags/case-study/atom.xml" rel="self" type="application/atom+xml" title="atom"/><updated>2026-11-15T23:03:09+00:00</updated><rights>© Rain 2020-present. Licensed under CC BY-NC-SA 4.0 unless marked otherwise.</rights><author><name>Rain</name></author><id>https://sunshowers.io/tags/case-study/</id><entry><title type="html">How (and why) nextest uses tokio</title><link href="https://sunshowers.io/posts/nextest-and-tokio/?utm_source=atom_feed" rel="alternate" type="text/html"/><id>https://sunshowers.io/posts/nextest-and-tokio/</id><published>2022-10-03T00:00:00+00:00</published><updated>2022-11-03T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>How my open source test runner uses async Rust.</blockquote><div class="sectioned-post">
<h2 id="1-introduction">1. Introduction</h2>
<p><em>Update 2022-11-03: This was originally the first in a series of two blog posts. I&rsquo;ve now marked this as a standalone post. Content originally slated for part 2 will now be published as followup posts.</em></p>
<p>I&rsquo;m the primary author and maintainer of <a href="https://nexte.st/">cargo-nextest</a>, a next-generation test
runner for Rust released earlier this year. (It reached a thousand stars on GitHub recently; <a href="https://github.com/nextest-rs/nextest">go
star it here!</a>)</p>
<div class="center">
<p><a href="https://github.com/nextest-rs/nextest"><img src="https://img.shields.io/github/stars/nextest-rs/nextest?style=social" alt="GitHub stars shield"></a></p>
</div>
<p>Nextest is <a href="https://nexte.st/book/benchmarks.html">faster than <code>cargo test</code></a> for most Rust projects,
and provides a number of extra features, such as <a href="https://nexte.st/book/retries">test retries</a>,
<a href="https://nexte.st/book/reusing-builds">reusing builds</a>, and <a href="https://nexte.st/book/partitioning">partitioning (sharding) test
runs</a>. To power these features, nextest uses
<a href="https://tokio.rs/">Tokio</a>, the leading Rust async runtime for writing network applications.</p>
<p><em>Wait, what?</em> Nextest isn&rsquo;t a network app at all<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. However, nextest has to juggle
multiple concurrent tests, each of which can be in the process of starting, printing output,
failing, timing out, being cancelled or even forcibly stopped. In this post, I&rsquo;ll try to convince
you that an async runtime is actually the perfect approach to handle a large number of heterogenous
events like this.</p>
<h3 id="theres-more-than-one-kind-of-distributed-system">There&rsquo;s more than one kind of distributed system</h3>
<p>The challenges posed by interprocess communication <em>within a system</em> are really similar to those
posed by remote communication <em>across systems</em>. Your computer is <a href="http://catern.com/compdist.html">a distributed
system</a>. Just like most network services, nextest spends almost all
of its time waiting rather than computing.</p>
<p>In other words, nextest provides the prototypical use-case for <em>asynchronous concurrency</em>, one of
the two main paradigms for non-sequential code.</p>

  <figure class="center" >
    
    <img src="/images/io-compute-bound.png"  alt="A simple diagram showing the two main paradigms for non-sequential code. To the left is &#39;I/O-bound&#39; with &#39;Async concurrency (Tokio)&#39;, and to the right is &#39;Compute-bound&#39; with &#39;Data Parallelism (Rayon)&#39;"   />
    
    
  </figure>


<ul>
<li>The async concurrency style is best for I/O-bound applications, where most time is spent waiting
on other parts of the overall system to respond. An I/O-bound application becomes faster if I/O
becomes faster&mdash;for example, if your network link becomes faster, or you replace a spinning disk
hard drive with a solid-state drive.</li>
<li>On the other end of the spectrum, <em>data parallelism</em> with libraries like
<a href="https://github.com/rayon-rs/rayon">Rayon</a> is best for compute-bound applications, where you&rsquo;re
running a lot of computations in parallel on a set of input data. These become faster if you replace
your CPU with a faster one.</li>
</ul>
<p>Why is an async runtime like Tokio good for I/O-bound applications? The answer has to do with
<em>heterogenous selects</em><sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p>
<h3 id="what-are-heterogenous-selects">What are heterogenous selects?</h3>
<p>You might have heard of the <em>select</em> operation: in computer parlance, it means to observe
two or more operations, and then to proceed with the first one that completes.</p>
<p>The name of the operation comes from the Unix
<a href="https://man7.org/linux/man-pages/man2/select.2.html"><code>select</code></a> syscall. With <code>select</code> and its successors
like <a href="https://man7.org/linux/man-pages/man7/epoll.7.html"><code>epoll</code></a>, you can typically only wait on one of
several I/O read or write operations (represented through file descriptors). But real-world
I/O-bound programs need to do much more than that! Here are some other kinds of operations they may
need to wait on:</p>
<ul>
<li>Timeouts or recurring timers</li>
<li><a href="https://tokio.rs/tokio/tutorial/channels">Channel reads and writes</a> when using message-passing,
including bounded channels with backpressure</li>
<li>Process exits</li>
<li>Unix signals</li>
</ul>
<p>I/O-bound programs often need to wait across several of these sources at the same time. For example,
it&rsquo;s common for services to make a network request, then select across:</p>
<ul>
<li>a network response</li>
<li>a timeout</li>
<li>a termination signal, so any pending network operations can be canceled</li>
</ul>
<p>This is an example of a <em>heterogenous select</em> (or, spelled slightly differently, a <em>heterogeneous select</em>): a select operation that can span several different
kinds of sources.</p>
<hr>
<p>We also run into portability considerations: <code>select</code> is Unix-specific, and <code>epoll</code> is for the most
part Linux-only<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>. Other operating systems have their own <code>epoll</code>-like abstractions, each
with their own peculiarities:</p>
<ul>
<li>BSD-like platforms have <code>kqueue</code> (<a href="https://www.freebsd.org/cgi/man.cgi?kqueue">FreeBSD</a>, <a href="https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/kqueue.2.html">macOS</a>).</li>
<li>Windows has <a href="https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports">I/O completion ports</a>.</li>
</ul>
<p>Some platforms may also support selecting across some kinds of heterogenous sources. For example,
Linux has <a href="https://man7.org/linux/man-pages/man2/signalfd.2.html"><code>signalfd</code></a> and
<a href="https://man7.org/linux/man-pages/man2/timerfd_create.2.html"><code>timerfd</code></a>. To the extent that these
are supported, there are significant differences in the APIs available on each platform. And all of
them make selecting across channels and other user code quite difficult.</p>
<p>The divergence in platform APIs and capabilities may not be a huge deal for certain kinds of network
services that only target one platform, but is a real problem for a cross-platform developer tool
like nextest.</p>
<hr>
<p>So there are two separate but related sets of problems here:</p>
<ol>
<li>The need to select over arbitrary kinds of heterogenous sources, not just I/O operations.</li>
<li>The need to do so in a cross-platform manner.</li>
</ol>
<p>Async runtimes like Tokio solve both sets of problems in one go<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>. In fact, I&rsquo;d like to make a bold claim here:</p>
<p><strong>The ability to do heterogenous selects is <em>the point of async Rust</em>.</strong></p>
<div class="framed">
<h3 id="async-runtimes-vs-asyncawait">Async runtimes vs async/await</h3>
<p>At this point it&rsquo;s worth mentioning that there are two related, but distinct, things here:</p>
<ol>
<li>An <em>async runtime</em>: a framework that manages an event loop, handling events from heterogenous sources and associating them with individual tasks.</li>
<li>The <em><code>async</code> and <code>await</code> keywords</em>, which provide an easy way to write tasks that wait on events handled by an async runtime.</li>
</ol>
<p>It is definitely possible to leverage async runtimes without ever using the <code>async</code> and <code>await</code> keywords&mdash;in fact, I&rsquo;ve written large chunks of a production service in Rust that used an async runtime before <code>async</code>/<code>await</code> was ever a thing.</p>
<p>However, without syntactic support it is often very difficult to do so, since you may need to hand-write complicated state machines. The <code>async</code> and <code>await</code> keywords dramatically simplify the experience of writing state machines. David Tolnay talks about this more <a href="https://docs.rs/dtolnay/latest/dtolnay/macro._01__await_a_minute.html#native-control-flow">in his blog post</a>. (The <code>poll_next</code>/<code>EncodeState</code> example in his blog post was part of the aforementioned production service, and it was my code that he rewrote.)</p>
</div>
<h2 id="2-the-initial-implementation">2. The initial implementation</h2>
<p>With this background, we&rsquo;re now ready to start looking at how nextest&rsquo;s current architecture came about.</p>
<p>The first versions of nextest all ran tests using &ldquo;normal&rdquo; threads. Here&rsquo;s the general structure
they used for their <a href="https://nexte.st/book/how-it-works.html">execution model</a>:</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-flowchart.png"  alt="A flowchart showing the main ways data moves around among nextest&#39;s components. See below for a text description."   />
    
    
      <figcaption class="center" >The main ways data moves around among nextest&rsquo;s components.</figcaption>
    
  </figure>


<p><em>The run pool:</em></p>
<p>Tests were run via a thread pool.</p>
<ol>
<li>Nextest first <a href="https://nexte.st/book/how-it-works.html#the-nextest-model">built up</a> a list of tests.</li>
<li>It then spun up a thread pool with the number of tests<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>, called the <em>run pool</em>.</li>
<li>For each test in the list, nextest used a thread in the run pool. The thread ran the test using
the <a href="https://docs.rs/duct/">duct</a> library, then blocked until the test exited. (nextest used duct for convenience&mdash;we could have used <code>std::process::Command</code> instead, which wouldn&rsquo;t have changed much.)</li>
<li>To send information about test events (the start, end, success/failure, and time taken for
individual tests) to the main thread, nextest used a <a href="https://docs.rs/crossbeam-channel/0.5/crossbeam_channel/">crossbeam MPSC
channel</a><sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup>: we&rsquo;ll call this the
<em>test event channel</em>.</li>
</ol>
<p><em>The signal thread:</em></p>
<p>To handle signals such as Ctrl-C cleanly, nextest spun up a separate thread, and if a signal is
detected, wrote to another channel: the <em>signal event channel</em>.</p>
<p><em>The main thread:</em></p>
<p>The main thread was dedicated to monitoring the two event channels. This thread would
<a href="https://docs.rs/crossbeam-channel/0.5/crossbeam_channel/macro.select.html">select</a> across
the <a href="https://github.com/nextest-rs/nextest/blob/ce98c9405bfe512e2f411d78bfe04468722ae69b/nextest-runner/src/runner.rs#L267-L287"><em>test event</em> and <em>signal event</em>
channels</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// InternalEvent has this definition:
</span></span></span><span class="line"><span class="cl"><span class="cp">#[derive(Debug)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">enum</span> <span class="nc">InternalEvent</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Test</span><span class="p">(</span><span class="n">InternalTestEvent</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Signal</span><span class="p">(</span><span class="n">SignalEvent</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Note that receiving an Err() from a receiver (a RecvError) means that there are
</span></span></span><span class="line"><span class="cl"><span class="c1">// no more senders.
</span></span></span><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">crossbeam_channel</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">recv</span><span class="p">(</span><span class="n">run_receiver</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">internal_event</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// internal_event is a Result&lt;InternalTestEvent, RecvError&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Ok</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Test</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// All senders were dropped -- the test run completed.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">recv</span><span class="p">(</span><span class="n">signal_receiver</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">internal_event</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// internal_event is a Result&lt;SignalEvent, RecvError&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Ok</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Signal</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// Ignore the signal thread being dropped. This is done for
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// noop signal handlers.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">continue</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ... process internal_event
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<ul>
<li>If this loop received a test event, it would bubble that up to the reporter (the component that
prints nextest&rsquo;s output). If a test failed, it would set a cancellation flag which would cause no
further tests to be scheduled.</li>
<li>If the loop received a signal event, it would set the cancellation flag.</li>
</ul>
<p>Importantly, <code>crossbeam_channel::select!</code> only lets you select over crossbeam channels, not
heterogenous sources in general. Anything else, including pipe reads or timers, must be translated
over to crossbeam channels somehow.</p>
<h3 id="the-first-issues-with-the-model">The first issues with the model</h3>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-with-wait.png"  alt="A flowchart showing how nextest&#39;s runner loop detected slow tests. See below for a text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s original runner loop detected slow tests.</figcaption>
    
  </figure>


<p>One of the first demands placed on nextest was the need to identify <a href="https://nexte.st/book/slow-tests">slow
tests</a>. Note that duct&rsquo;s
<a href="https://docs.rs/duct/0.13/duct/struct.Handle.html#method.wait"><code>Handle::wait</code></a>
method doesn&rsquo;t have any notion of a timeout: it just blocks until the process is completed.</p>
<p>Luckily, <code>crossbeam_channel</code> has a <a href="https://docs.rs/crossbeam-channel/0.5/crossbeam_channel/struct.Receiver.html#method.recv_timeout"><code>recv_timeout</code>
method</a>.
So the solution was straightforward: set up another thread pool, called the <em>wait pool</em>, which would wait on test processes to exit.</p>
<p>So now this becomes:</p>
<ol>
<li>Start the test process using a thread in the run pool.</li>
<li>Spawn a wait pool thread to wait on the test.</li>
<li>In the run pool, wait on the thread in the wait pool with a timeout.
<ul>
<li>If the test finishes before the timeout, mark it done.</li>
<li>If the timeout is hit, mark the test slow and go back to waiting on the test.</li>
</ul>
</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/5b82369cb42209c7f513ee686faa3dd340d971b7/nextest-runner/src/runner.rs#L358-L389">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cmd</span><span class="p">.</span><span class="n">start</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="bp">self</span><span class="p">.</span><span class="n">wait_pool</span><span class="p">.</span><span class="n">in_place_scope</span><span class="p">(</span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">sender</span><span class="p">,</span><span class="w"> </span><span class="n">receiver</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">crossbeam_channel</span>::<span class="n">bounded</span>::<span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">wait_handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="n">handle</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Spawn a task on the threadpool that waits for the test to finish.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">s</span><span class="p">.</span><span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">|</span><span class="n">_</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">wait_handle</span><span class="p">.</span><span class="n">wait</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">sender</span><span class="p">.</span><span class="n">send</span><span class="p">(());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Continue waiting for the test to finish with a timeout, logging at
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// slow-timeout intervals.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">while</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">error</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">receiver</span><span class="p">.</span><span class="n">recv_timeout</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">slow_timeout</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">match</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">RecvTimeoutError</span>::<span class="n">Timeout</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// ... bubble up the slow event so the reporter can print out a
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// SLOW line
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">RecvTimeoutError</span>::<span class="n">Disconnected</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="fm">unreachable!</span><span class="p">(</span><span class="s">&#34;Waiting thread should never drop the sender&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">into_output</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ... process output
</span></span></span></code></pre></div></details>
<p>This works, but you can already see the cracks starting to show. To solve this issue we had to
introduce a whole new thread pool on top of the existing run pool.</p>
<h3 id="terminating-tests-using-a-polling-loop">Terminating tests using a polling loop</h3>
<p>A few months after nextest was released, a user requested another feature: the ability to terminate
tests that take more than a certain amount of time.</p>
<p>Now, nextest <em>could</em> terminate tests immediately with the unblockable <a href="https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html"><code>SIGKILL</code>
signal</a><sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup>
(aka <code>kill -9</code>). But it&rsquo;s generally a good idea to give tests a bit of a grace period. So we chose
to first send tests a <code>SIGTERM</code> signal, then wait a bit for them to clean up before sending
<code>SIGKILL</code>.</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-with-wait-and-timeout.png"  alt="A flowchart to show how nextest&#39;s runner loop implemented test timeouts. See below for text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s original runner loop implemented test timeouts.</figcaption>
    
  </figure>


<p>To do this, we defined two timeouts:</p>
<ol>
<li>A <em>slow timeout</em>, after which tests are marked as slow. Nextest&rsquo;s default slow timeout is 60 seconds.</li>
<li>A <em>termination timeout</em>, after which tests are killed. Nextest doesn&rsquo;t have a default termination timeout, but a typical one users might set is 5 minutes (300 seconds).</li>
</ol>
<p>The test execution flow is then:</p>
<ol>
<li>Start the test process using a thread in the run pool.</li>
<li>Spawn a wait pool thread to wait on the test.</li>
<li>In the run pool, wait on the thread in the wait pool with the slow timeout.
<ul>
<li>If the test finishes before the slow timeout, mark it done.</li>
<li>If the slow timeout is hit and the termination timeout hasn&rsquo;t been hit yet, mark the test as slow and go back to waiting on it.</li>
<li>If the termination timeout is hit, set up a termination loop.</li>
</ul>
</li>
</ol>
<p>And the termination loop:</p>
<ol>
<li>Send the process <code>SIGTERM</code>.</li>
<li>In a loop, for up to 10 seconds:
<ul>
<li>Check if the test process exited.</li>
<li>If it did, break out of the loop.</li>
<li>Otherwise, wait 100ms and check the test&rsquo;s status again.</li>
</ul>
</li>
<li>If the test hasn&rsquo;t exited by the end of the loop, send it <code>SIGKILL</code>.</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/ce98c9405bfe512e2f411d78bfe04468722ae69b/nextest-runner/src/runner.rs#L412-L473">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">match</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">RecvTimeoutError</span>::<span class="n">Timeout</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// ... bubble up the slow event to the reporter
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">terminate_after</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slow_timeout</span><span class="p">.</span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">if</span><span class="w"> </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// Attempt to terminate the slow test.
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// As there is a race between shutting down a slow test and its
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// own completion, we silently ignore errors to avoid printing
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// false warnings.
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="cp">#[cfg(unix)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="kd">let</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">use</span><span class="w"> </span><span class="n">duct</span>::<span class="n">unix</span>::<span class="n">HandleExt</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">use</span><span class="w"> </span><span class="n">libc</span>::<span class="no">SIGTERM</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">send_signal</span><span class="p">(</span><span class="no">SIGTERM</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// give the process a grace period of 10s
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="k">match</span><span class="w"> </span><span class="p">((</span><span class="mi">0</span><span class="o">..</span><span class="mi">100</span><span class="p">).</span><span class="n">contains</span><span class="p">(</span><span class="o">&amp;</span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">try_wait</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="n">_</span><span class="p">,</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Some</span><span class="p">(</span><span class="n">_</span><span class="p">)))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">None</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">None</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="c1">// in case of an error we&#39;ll send SIGKILL nonetheless
</span></span></span><span class="line"><span class="cl"><span class="w">                            </span><span class="p">(</span><span class="n">_</span><span class="p">,</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="n">i</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="n">std</span>::<span class="n">thread</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_millis</span><span class="p">(</span><span class="mi">100</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="cp">#[cfg(not(unix))]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="kd">let</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">exited</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">handle</span><span class="p">.</span><span class="n">kill</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ExecutionResult</span>::<span class="n">Timeout</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<p>You can already start seeing the code becoming long and rather janky, with its 100ms polling loop.</p>
<h3 id="detecting-leaked-handles">Detecting leaked handles</h3>
<p>Some tests spawn server or other subprocesses, and sometimes those subprocesses don&rsquo;t get cleaned up
at the end of the test. In particular, nextest is concerned about processes that
<a href="https://doc.rust-lang.org/std/process/struct.Stdio.html#method.inherit"><em>inherit</em></a> standard output
and/or standard error. Here&rsquo;s an example test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[test]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">test_subprocess_doesnt_exit</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Spawn a subprocess that sleeps for 2 minutes, then exits.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cmd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span>::<span class="n">process</span>::<span class="n">Command</span>::<span class="n">new</span><span class="p">(</span><span class="s">&#34;sleep&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">cmd</span><span class="p">.</span><span class="n">arg</span><span class="p">(</span><span class="s">&#34;120&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// This sleep command inherits standard output and standard error from the test.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">cmd</span><span class="p">.</span><span class="n">spawn</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Since <code>sleep</code> inherits standard output and standard error from the test, it keeps those file
descriptors open until it exits. Previously, nextest would simply wait until those subprocesses
exited. In the worst case, the subprocesses wouldn&rsquo;t exit at all: think of a server process that
gets started but never shut down. That can result in nextest simply hanging, which isn&rsquo;t great.</p>
<p>How can nextest detect <a href="https://nexte.st/book/leaky-tests.html">these sorts of <em>leaky</em> tests</a>? One
solution is to wait for standard output and standard error to be closed until a short duration (say
100ms) passes. While this is conceptually simple, <em>implementing it</em> is surprisingly complicated.
Waiting on one handle being closed is nontrivial; waiting on <em>two</em> handles being closed (standard
output and standard error) adds an extra challenge.</p>
<p>I spent some time looking at how to implement it manually, and came away from it with the
understanding that I&rsquo;d have to maintain separate implementations for Windows and Unix. Not only
that, I&rsquo;d just be duplicating a lot of the work Tokio had already done to abstract over platform
differences. So I started looking at porting over nextest to Tokio.</p>
<h2 id="3-using-tokio">3. Using Tokio</h2>
<p>I spent a weekend porting over nextest to use async Rust with Tokio. After a bit of iteration, I
settled on this general architecture:</p>
<ol>
<li>Create a <a href="https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html">Tokio <code>Runtime</code></a> in the
beginning. The <code>Runtime</code> lasts for as long as the runner loop does.</li>
<li>Turn the list of tests into a
<a href="https://docs.rs/futures/0.3/futures/stream/trait.Stream.html"><code>Stream</code></a>, then for each test,
create a future that executes the test. Use
<a href="https://docs.rs/futures/0.3/futures/stream/trait.StreamExt.html#method.buffer_unordered"><code>buffer_unordered</code></a>
to control concurrency<sup id="fnref:8"><a href="#fn:8" class="footnote-ref" role="doc-noteref">8</a></sup>.</li>
<li>Switch over crossbeam channels to <a href="https://docs.rs/tokio/1/tokio/sync/mpsc/index.html">tokio
channels</a> as needed, and
<code>crossbeam_channel::select!</code> invocations <a href="https://docs.rs/tokio/1/tokio/macro.select.html">to
<code>tokio::select!</code></a>.</li>
</ol>
<p>Here&rsquo;s the bit that <a href="https://github.com/nextest-rs/nextest/blob/8703af50b649fc882e601d4eb01f7c2c104c0395/nextest-runner/src/runner.rs#L325-L347">selects over the test event and signal event channels</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">signals_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">run_receiver</span><span class="p">.</span><span class="n">recv</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// internal_event is an Option&lt;InternalTestEvent&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Some</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Test</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">None</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">// All senders were dropped -- the test run completed.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">break</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">internal_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">signal_handler</span><span class="p">.</span><span class="n">recv</span><span class="p">(),</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">signals_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// internal_event is an Option&lt;SignalEvent&gt;.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">match</span><span class="w"> </span><span class="n">internal_event</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">Some</span><span class="p">(</span><span class="n">event</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">InternalEvent</span>::<span class="n">Signal</span><span class="p">(</span><span class="n">event</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nb">None</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="n">signals_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="k">continue</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ... process internal_event
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<p>This is really similar to the crossbeam-based implementation above, with just one real change: The
code tracks a <code>signals_done</code> state to see if the signal event channel has been dropped. This is to
ensure that the loop doesn&rsquo;t repeatedly get <code>None</code> values from the signal handler, wasting cycles.
(This is more important with futures that, once completed, cannot be waited on further, as we&rsquo;ll
see in the next section.)</p>
<h3 id="waiting-on-process-completion-with-tokio">Waiting on process completion with Tokio</h3>
<p>Where this gets really interesting is with the loop nextest uses to wait for tests to complete. It
turns out that <em>all</em> the code we wrote above can be replaced with a set of Rust and Tokio
primitives, written in a roughly declarative style.</p>
<p>After starting the process, nextest <a href="https://github.com/nextest-rs/nextest/blob/e11c35547511e728a422cabdcd04ea542df3bef5/nextest-runner/src/runner.rs#L600-L648">sets
up</a>
a timer, and futures to read from standard output and standard error:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Set up a timer to produce events at predefined intervals.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">time</span>::<span class="n">interval</span><span class="p">(</span><span class="n">slow_timeout</span><span class="p">.</span><span class="n">period</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// The first tick is immediate.
</span></span></span><span class="line"><span class="cl"><span class="n">interval</span><span class="p">.</span><span class="n">tick</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">child_stdout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child</span><span class="p">.</span><span class="n">stdout</span><span class="p">.</span><span class="n">take</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">child_stderr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child</span><span class="p">.</span><span class="n">stderr</span><span class="p">.</span><span class="n">take</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stdout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bytes</span>::<span class="n">BytesMut</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stderr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bytes</span>::<span class="n">BytesMut</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">read_all_to_bytes</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">bytes</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">bytes</span>::<span class="n">BytesMut</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">mut</span><span class="w"> </span><span class="n">input</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">(</span><span class="k">dyn</span><span class="w"> </span><span class="n">AsyncRead</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Unpin</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">std</span>::<span class="n">io</span>::<span class="nb">Result</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Reborrow it as AsyncReadExt::read_buf expects
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Sized self.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">input</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">bytes</span><span class="p">.</span><span class="n">reserve</span><span class="p">(</span><span class="mi">4096</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">bytes_read</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="n">read_buf</span><span class="p">(</span><span class="n">bytes</span><span class="p">).</span><span class="k">await</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="n">bytes_read</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Set up futures for reading from stdout and stderr.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">stdout_fut</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stdout</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child_stdout</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">read_all_to_bytes</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stdout</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stdout</span><span class="p">).</span><span class="k">await</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">tokio</span>::<span class="fm">pin!</span><span class="p">(</span><span class="n">stdout_fut</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">stderr_fut</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stderr</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child_stderr</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">read_all_to_bytes</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stderr</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">child_stderr</span><span class="p">).</span><span class="k">await</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">tokio</span>::<span class="fm">pin!</span><span class="p">(</span><span class="n">stderr_fut</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div></details>
<p>Some notes about the code above:</p>
<ul>
<li>On being polled, the futures repeatedly read standard output and standard error into their
corresponding buffers in 4 KiB chunks. They only exit when 0 bytes are read, which corresponds
to the handle being closed.</li>
<li>The code needs to maintain <code>stdout_done</code> and <code>stderr_done</code> flags: this is similar to
<code>signals_done</code> above, except this time it&rsquo;s necessary because async blocks cannot be polled once
completed: if done so, they panic with the message &ldquo;<code>async fn</code> resumed after completion&rdquo;.</li>
<li>The futures borrow data from the stack. This is possible thanks to async Rust&rsquo;s integration with the
borrow checker.</li>
<li>The futures are <a href="https://docs.rs/tokio/1/tokio/macro.pin.html"><em>pinned</em> to the stack</a>, which means
that they cannot be moved. This is a <a href="https://rust-lang.github.io/async-book/04_pinning/01_chapter.html">fundamental
part</a> of how zero-overhead
async Rust works. An async block must be pinned before it can be polled.</li>
</ul>
<p>Remember how we talked about heterogenous selects in part 1 of this post? This is where we start
using them. Notice how <code>crossbeam_channel::select!</code> only works on crossbeam channels. On the other
hand, Tokio&rsquo;s select works on arbitrary, heterogenous sources of asynchronicity.</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-tokio.png"  alt="A flowchart to show how nextest&#39;s Tokio-based runner loop executes tests. See below for text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s Tokio-based runner loop executes tests.</figcaption>
    
  </figure>


<p>Here&rsquo;s how nextest now waits on tests:</p>
<p><em>Setup:</em></p>
<ol>
<li>Start the test process, which sets up a future that completes when the test exits.</li>
<li>Set up a timer for the slow timeout.</li>
<li>Set up futures to read from standard output and standard error.</li>
</ol>
<p>Then, select across the futures listed above in a loop, using Tokio&rsquo;s ability to handle heterogenous
selects with <code>tokio::select!</code>.</p>
<p><em>If data is available via the standard output or standard error futures:</em></p>
<ol>
<li>Read data into the corresponding buffer.</li>
<li>Loop back and select again.</li>
</ol>
<p><em>If the timer fires:</em></p>
<ol>
<li>If the terminate timeout is hit, terminate the test and exit the loop.</li>
<li>If not, mark as slow, loop back and select again.</li>
</ol>
<p><em>If the process exits:</em></p>
<ol>
<li>Exit the execution loop.</li>
<li>Proceed to the leak detection loop below.</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/8703af50b649fc882e601d4eb01f7c2c104c0395/nextest-runner/src/runner.rs#L533-L570">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// res is an std::io::Result&lt;ExitStatus&gt;.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stdout_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stderr_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">child</span><span class="p">.</span><span class="n">wait</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// The test finished executing.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="n">res</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">interval</span><span class="p">.</span><span class="n">tick</span><span class="p">(),</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">status</span><span class="p">.</span><span class="n">is_none</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">is_slow</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// ... bubble up the slow event to the reporter
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">terminate_after</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slow_timeout</span><span class="p">.</span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="n">timeout_hit</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="n">terminate_after</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="c1">//  ... attempt to terminate the slow test
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">ExecutionResult</span>::<span class="n">Timeout</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div></details>
<hr>
<p>And then, how does nextest detect leaked handles? After the main loop ends, nextest just runs another little loop
for that:</p>

  <figure class="floatright" >
    
    <img src="/images/nextest-runner-tokio-leaks.png"  alt="A flowchart to show how nextest&#39;s Tokio-based runner loop detects leaked handles in tests. See below for text description."   />
    
    
      <figcaption class="center" >How nextest&rsquo;s Tokio-based runner loop detects leaked handles in tests.</figcaption>
    
  </figure>


<ol>
<li>Set up a short (by default 100ms) leak timer.</li>
<li>Select across the leak timer and the standard output and error futures from before, in a loop.</li>
</ol>
<p><em>If data is available via the standard output or standard error futures:</em></p>
<ol>
<li>Read data into the corresponding buffer.</li>
<li>Loop back, and select again.</li>
</ol>
<p><em>If both the standard output and standard error futures have completed:</em></p>
<ol>
<li>Note that the test didn&rsquo;t leak.</li>
<li>Conclude test execution.</li>
</ol>
<p><em>If the timer fires before both futures have completed:</em></p>
<ol>
<li>Note that the test leaked handles.</li>
<li>Conclude test execution.</li>
</ol>
<p><a href="https://github.com/nextest-rs/nextest/blob/8703af50b649fc882e601d4eb01f7c2c104c0395/nextest-runner/src/runner.rs#L582-L609">Here&rsquo;s the code</a>:</p>
<details>
      <summary>Click to expand code sample</summary>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Once the process is done executing, wait up to leak_timeout for the pipes to shut
</span></span></span><span class="line"><span class="cl"><span class="c1">// down. Previously, this used to hang if spawned grandchildren inherited
</span></span></span><span class="line"><span class="cl"><span class="c1">// stdout/stderr but didn&#39;t shut down properly. Now, nextest detects those cases and
</span></span></span><span class="line"><span class="cl"><span class="c1">// marks them as leaked.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">leaked</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">sleep</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">time</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">leak_timeout</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stdout_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stdout_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">stderr_fut</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">stderr_done</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">res</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">()</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sleep</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="p">(</span><span class="n">stdout_done</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="n">stderr_done</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// stdout and/or stderr haven&#39;t completed yet. In this case, break
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// the loop and mark this as leaked.
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">else</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="k">break</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span></code></pre></div></details>
<p>That&rsquo;s it! We&rsquo;ve replaced all of that unwieldy code with a couple of declarative <code>select!</code> loops.</p>
<div class="framed">
<h3 id="if-it-compiles-does-it-work">If it compiles, does it work?</h3>
<p>One of the promises of Rust&rsquo;s type system is that it&rsquo;ll catch most kinds of bugs before you ever run
the program. In my experience, this promise is <em>mostly</em> upheld, even when dealing with async code.
For a few compile-time issues like pinning, it was easy to find
<a href="https://tokio.rs/tokio/tutorial/select#resuming-an-async-operation">solutions</a>.</p>
<p>I ran into two runtime issues:</p>
<ol>
<li>A panic with the message &ldquo;<code>async fn</code> resumed after completion&rdquo;. The solution to this
was <a href="https://tokio.rs/tokio/tutorial/select#modifying-a-branch">easy to find</a>.</li>
<li>The main loop that <a href="/posts/nextest-and-tokio/#cutting-the-knot">selected across</a> the test event and signal event channels
hung during longer test runs; this was caused by a <a href="https://docs.rs/tokio/latest/tokio/macro.select.html#fairness"><code>biased</code>
statement</a>. Removing the <code>biased</code>
fixed it, but I mostly found that out by trial and error. I&rsquo;m not sure how to debug this kind of
issue in a principled manner.</li>
</ol>
</div>
<h2 id="4-conclusion">4. Conclusion</h2>
<p>There&rsquo;s a bit of a meme in the Rust world: that if you can &ldquo;just use&rdquo; threads, <a href="https://news.ycombinator.com/item?id=31601973">you
should</a>; and that async code is really only &ldquo;worth
it&rdquo; for network services that <a href="https://www.reddit.com/r/rust/comments/v6rx10/comment/ibh2nwz">need to be massively
concurrent</a>. (I don&rsquo;t mean to pick on
the individual authors here. It&rsquo;s a general view in the Rust community.)</p>
<p>While I agree that async Rust isn&rsquo;t <em>easy</em>, I believe nextest&rsquo;s experience using tokio directly
challenges that general view. The point of async, even more than the concurrency, is to make it
easy to operate across arbitrary sources of asynchronicity. This is exactly what nextest needs to
do. Any Rust program that is I/O-bound, or any program that needs to do heterogenous selects,
should strongly consider using an async runtime like Tokio.</p>
<h3 id="thanks">Thanks</h3>
<p>Thanks to <a href="https://twitter.com/munin">Fiona</a>, <a href="https://twitter.com/inanna_malick">Inanna</a>,
<a href="https://twitter.com/mycoliza/">Eliza</a>, <a href="https://twitter.com/bmwill_">Brandon</a>,
<a href="https://twitter.com/PredragGruevski">Predrag</a>, and <a href="https://twitter.com/mgattozzi">Michael</a> for
reviewing drafts of this article. And thank you for reading it to the end.</p>
<p>The flowcharts are available in <a href="https://drive.google.com/drive/folders/1znKowkG_58v7oo4zoTxHPF0wBd6OkTyt?usp=sharing">this Google folder</a>, and are licensed under <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>.</p>
<h4 id="changelog">Changelog</h4>
<p><em>2022-10-06:</em></p>
<ul>
<li>Add a note that platform-specific APIs like <code>signalfd</code> and <code>timerfd</code> can provide heterogenous selects to some extent, though not as much as Tokio can.</li>
<li>Updated <code>stdout_fut</code> and <code>stderr_fut</code> implementations to match changes in <a href="https://github.com/nextest-rs/nextest/pull/560">#560</a>.</li>
<li>I&rsquo;d forgotten to add Michael Gattozzi to the reviewers list. Fixed, sorry!</li>
</ul>
</div>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>Other than its <a href="https://nexte.st/book/updating.html">self-update</a> functionality,
which isn&rsquo;t relevant here.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>Full credit for the term and framing goes to <a href="https://twitter.com/mycoliza">Eliza Weisman</a>.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>Some other systems have added <code>epoll</code> for compatibility with Linux: for example,
<a href="https://illumos.org/man/7/epoll">illumos</a>.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>The platform-specific implementations are usually abstracted away with a low-level library
like <a href="https://github.com/tokio-rs/mio">Mio</a>.&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5">
<p>The runner loop needed a scoped thread pool because it relied on borrowed data&mdash;it
used <a href="https://github.com/rayon-rs/rayon">rayon</a>&rsquo;s
<a href="https://docs.rs/rayon/1/rayon/struct.ThreadPool.html"><code>ThreadPool</code></a> for that purpose.&#160;<a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6">
<p>MPSC stands for &ldquo;multi-producer, single consumer&rdquo;. This kind of channel lets many threads
(in this case, the threads running the tests) send data to a single &ldquo;status&rdquo; thread.&#160;<a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7">
<p>Or its equivalent on Windows,
<a href="https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess">TerminateProcess</a>.&#160;<a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:8">
<p>Why <code>buffer_unordered</code>? Because we&rsquo;d like tests to be <em>started</em> in lexicographic order,
but results to be reported in the order tests finish in. That is exactly the behavior
<code>buffer_unordered</code> provides.</p>
<p>One alternative was to spawn every test as a separate future and use a
<a href="https://docs.rs/tokio/1/tokio/sync/struct.Semaphore.html">semaphore</a> to control concurrency,
but that might have resulted in tests being started in arbitrary order.&#160;<a href="#fnref:8" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/nextest" term="nextest" label="nextest"/><category scheme="https://sunshowers.io/tags/tokio" term="tokio" label="tokio"/><category scheme="https://sunshowers.io/tags/case-study" term="case-study" label="case-study"/></entry></feed>