<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us"><generator uri="https://gohugo.io/" version="0.170.0">Hugo</generator><title type="html">Rust on sunshowers</title><subtitle type="html">thoughts and ideas</subtitle><link href="https://sunshowers.io/tags/rust/" rel="alternate" type="text/html" title="html"/><link href="https://sunshowers.io/tags/rust/index.xml" rel="alternate" type="application/rss+xml" title="rss"/><link href="https://sunshowers.io/tags/rust/atom.xml" rel="self" type="application/atom+xml" title="atom"/><updated>2026-11-15T23:03:09+00:00</updated><rights>© Rain 2020-present. Licensed under CC BY-NC-SA 4.0 unless marked otherwise.</rights><author><name>Rain</name></author><id>https://sunshowers.io/tags/rust/</id><entry><title type="html">iddqd, or the hardest kind of unsafe Rust</title><link href="https://sunshowers.io/posts/iddqd-unsafe/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/on-poisoning/?utm_source=atom_feed" rel="related" type="text/html" title="In defense of lock poisoning in Rust"/><link href="https://sunshowers.io/posts/socketaddrv6-not-roundtrip/?utm_source=atom_feed" rel="related" type="text/html" title="`SocketAddrV6` is not roundtrip serializable"/><link href="https://sunshowers.io/posts/cancelling-async-rust/?utm_source=atom_feed" rel="related" type="text/html" title="Cancelling async Rust"/><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="related" type="text/html" title="Why nextest is process-per-test"/><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="related" type="text/html" title="Open and closed universes"/><id>https://sunshowers.io/posts/iddqd-unsafe/</id><published>2026-06-02T00:00:00+00:00</published><updated>2026-06-16T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>How Oxide&rsquo;s Rust collections library defends against adversarial trait implementations.</blockquote><p><em>Mirrored from the canonical version on the <a href="https://oxide.computer/blog/iddqd-unsafe">Oxide blog</a>.</em></p>
<p><em>Update 2026-06-16</em>: I talked about this post on <em>Oxide and Friends</em>, in an episode titled <em>The Hardest Kind of Unsafe Rust</em>. Check it out on <a href="https://www.youtube.com/watch?v=mdcu3dKMjyg">YouTube</a>, <a href="https://oxide-and-friends.transistor.fm/episodes/the-hardest-kind-of-unsafe-rust">Transistor</a>, or wherever else you get your podcasts.</p>
<hr>
<p>I&rsquo;m the main author of <a href="https://docs.rs/iddqd"><code>iddqd</code></a>, a Rust library for maps (named after the <a href="https://doomwiki.org/wiki/Doom_cheat_codes#All_Doom_engine_versions">Doom cheat code</a>) where keys are borrowed from values. At <a href="https://oxide.computer/">Oxide</a> we use it extensively in <a href="https://github.com/oxidecomputer/omicron">Omicron</a>, our control plane—the software that <a href="https://rfd.shared.oxide.computer/rfd/0048">sits</a> at the heart of every Oxide rack, provisions resources like compute and storage for our customers, and ensures the rack stays up and running over time. <code>iddqd</code> maintains in-memory indexes of the kinds of large records that show up everywhere in a system like that, such as <a href="https://github.com/oxidecomputer/omicron/blob/2e319eb3b9b9fcffb8cee71cffb9cd13397cb316/sled-agent/config-reconciler/src/internal_disks.rs#L602-L637">disks</a> or <a href="https://github.com/oxidecomputer/omicron/blob/2e319eb3b9b9fcffb8cee71cffb9cd13397cb316/nexus/types/src/inventory.rs#L682-L717">sled inventories</a>. As a result, it must be correct: if it misbehaves, our control plane can malfunction in ways that are unpredictable and hard to diagnose.</p>
<p><code>iddqd</code> consists of a fair amount of unsafe code underneath. There&rsquo;s been <a href="https://github.com/oven-sh/bun/issues/30719">some recent concern</a> over the amount of unsafe code in Rust rewrites, so I thought I&rsquo;d write about some of the unsafe code in <code>iddqd</code> and how we try to tame it.</p>
<h2 id="what-problem-does-iddqd-solve">What problem does iddqd solve?</h2>
<p>With Rust&rsquo;s standard library maps, keys are stored separately from values. Let&rsquo;s say you want to store a map of records keyed by an email address. With <a href="https://doc.rust-lang.org/std/collections/struct.BTreeMap.html"><code>std::collections::BTreeMap</code></a>, you might write something like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Email is typically a newtype which validates that the email address is well-formed.
</span></span></span><span class="line"><span class="cl"><span class="c1">// In this example, we alias it to String for simplicity.
</span></span></span><span class="line"><span class="cl"><span class="k">type</span> <span class="nc">Email</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">String</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">User</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">age</span>: <span class="kt">u8</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">users</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BTreeMap</span>::<span class="o">&lt;</span><span class="n">Email</span><span class="p">,</span><span class="w"> </span><span class="n">User</span><span class="o">&gt;</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">users</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;alice@example.com&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">User</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span>: <span class="s">&#34;Alice&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w"> </span><span class="n">age</span>: <span class="mi">30</span><span class="w"> </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">users</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;bob@example.com&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">User</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span>: <span class="s">&#34;Bob&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w"> </span><span class="n">age</span>: <span class="mi">35</span><span class="w"> </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>This approach has what I consider to be a pretty major downside: the key (email) is not stored in the same struct as the value (the rest of the record). How would you handle this? One way is to pass around both the email and the user, for example with <a href="https://doc.rust-lang.org/std/collections/struct.BTreeMap.html#method.get_key_value"><code>get_key_value</code></a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">process_user</span><span class="p">(</span><span class="n">email</span>: <span class="kp">&amp;</span><span class="nc">Email</span><span class="p">,</span><span class="w"> </span><span class="n">user</span>: <span class="kp">&amp;</span><span class="nc">User</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="cm">/* ... */</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">email</span><span class="p">,</span><span class="w"> </span><span class="n">user</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">users</span><span class="p">.</span><span class="n">get_key_value</span><span class="p">(</span><span class="s">&#34;alice@example.com&#34;</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">process_user</span><span class="p">(</span><span class="n">email</span><span class="p">,</span><span class="w"> </span><span class="n">user</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>As an extension, you could maybe have a struct which combines both at fetch time:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">UserRecord</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">email</span>: <span class="kp">&amp;</span><span class="na">&#39;a</span> <span class="nc">Email</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">user</span>: <span class="kp">&amp;</span><span class="na">&#39;a</span> <span class="nc">User</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">email</span><span class="p">,</span><span class="w"> </span><span class="n">user</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">users</span><span class="p">.</span><span class="n">get_key_value</span><span class="p">(</span><span class="s">&#34;alice@example.com&#34;</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">record</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UserRecord</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">email</span><span class="p">,</span><span class="w"> </span><span class="n">user</span><span class="w"> </span><span class="p">};</span><span class="w">
</span></span></span></code></pre></div><p>In practice, this gets quite unwieldy when you have lots of different types of records that need this kind of treatment.</p>
<p>Alternatively, you could duplicate the email across the key and the value:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">User</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">email</span>: <span class="nc">Email</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">age</span>: <span class="kt">u8</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">users</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BTreeMap</span>::<span class="o">&lt;</span><span class="n">Email</span><span class="p">,</span><span class="w"> </span><span class="n">User</span><span class="o">&gt;</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">email</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;alice@example.com&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">users</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">email</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">User</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">email</span><span class="p">,</span><span class="w"> </span><span class="n">name</span>: <span class="s">&#34;Alice&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w"> </span><span class="n">age</span>: <span class="mi">30</span><span class="w"> </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>But that has the risk that the emails stored in the key and the value fall out of sync.</p>
<p><code>iddqd</code> provides a better alternative. With <a href="https://docs.rs/iddqd/latest/iddqd/struct.IdOrdMap.html"><code>IdOrdMap</code></a>, you can write something like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">User</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">email</span>: <span class="nc">Email</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">age</span>: <span class="kt">u8</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// You implement a trait which tells iddqd how to extract the key from the
</span></span></span><span class="line"><span class="cl"><span class="c1">// record.
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">IdOrdItem</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">User</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// The key type can borrow from the value!
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">type</span> <span class="nc">Key</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="na">&#39;a</span><span class="w"> </span><span class="n">Email</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">key</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span>::<span class="n">Key</span><span class="o">&lt;</span><span class="nb">&#39;_</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">&amp;</span><span class="bp">self</span><span class="p">.</span><span class="n">email</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// This macro expands into a small amount of boilerplate to work around
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// a Rust borrow checker limitation.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="fm">id_upcast!</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Then, you can insert records:
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">users</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">IdOrdMap</span>::<span class="o">&lt;</span><span class="n">User</span><span class="o">&gt;</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">users</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">insert_unique</span><span class="p">(</span><span class="n">User</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">email</span>: <span class="s">&#34;alice@example.com&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">name</span>: <span class="s">&#34;Alice&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">age</span>: <span class="mi">30</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">users</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">insert_unique</span><span class="p">(</span><span class="n">User</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">email</span>: <span class="s">&#34;bob@example.com&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">name</span>: <span class="s">&#34;Bob&#34;</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">age</span>: <span class="mi">35</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// And look them up by email:
</span></span></span><span class="line"><span class="cl"><span class="fm">assert_eq!</span><span class="p">(</span><span class="o">&amp;</span><span class="n">users</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">&#34;alice@example.com&#34;</span><span class="p">).</span><span class="n">unwrap</span><span class="p">().</span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;Alice&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="fm">assert_eq!</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">&#34;bob@example.com&#34;</span><span class="p">).</span><span class="n">unwrap</span><span class="p">().</span><span class="n">age</span><span class="p">,</span><span class="w"> </span><span class="mi">35</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>At Oxide, this has proven to be an invaluable pattern: many of the records in our control plane are quite large (think database lookups), and <code>iddqd</code> is a great fit for them. It also comes with several other features that directly address pain points we&rsquo;ve dealt with at Oxide. A few worth calling out:</p>
<ul>
<li><a href="https://docs.rs/iddqd/latest/iddqd/#examples">First-class support</a> for complex keys that borrow from more than one field, without having to resort to <a href="https://github.com/sunshowers-code/borrow-complex-key-example">workarounds like dynamic dispatch</a>.</li>
<li>Maps with <a href="https://docs.rs/iddqd/latest/iddqd/struct.BiHashMap.html">two</a> or <a href="https://docs.rs/iddqd/latest/iddqd/struct.TriHashMap.html">three</a> keys per item, each independently indexing the same record, without the usual pattern of maintaining multiple maps by hand.</li>
<li>Serde implementations that <a href="https://docs.rs/iddqd/latest/iddqd/struct.IdOrdMap.html#impl-Serialize-for-IdOrdMap%3CT%3E">serialize</a> as sequences rather than maps, so that non-string keys can be serialized in JSON<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. Importantly, these implementations reject duplicate keys. (For backwards compatibility, serialization as maps is <a href="https://docs.rs/iddqd/latest/iddqd/id_ord_map/struct.IdOrdMapAsMap.html">also supported</a>.)</li>
</ul>
<p>Like many of our other crates, <code>iddqd</code> is built for Oxide&rsquo;s needs but is generally useful to the Rust community. You&rsquo;re <a href="https://crates.io/crates/iddqd">welcome</a> to use it in your own projects as well.</p>
<h2 id="on-unsafe-rust">On unsafe Rust</h2>
<p>Before I move on, I want to talk about what it means for <code>unsafe</code> to exist in a memory-safe language. The big concern is <a href="https://doc.rust-lang.org/reference/behavior-considered-undefined.html"><strong>undefined behavior</strong></a> (UB): a program behaves in an unpredictable way because core assumptions made by the language or compiler have been violated. Rust calls an abstraction <em>sound</em> if no safe code can use it to cause UB, and <em>unsound</em> otherwise.</p>
<p>The vast majority of Rust code is safe, which (assuming that any unsafe used by the safe code is sound) means that no UB can occur. However, due to the fundamental undecidability of static analysis (see <a href="https://en.wikipedia.org/wiki/Rice%27s_theorem">Rice&rsquo;s theorem</a>), it is impossible for the Rust compiler—or any kind of algorithm that terminates—to accept all programs without UB and reject all programs with it. Therefore, when writing such an algorithm, its authors have to make a decision: do they reject some programs without UB, accept some programs with UB, or both? The Rust compiler does the first: within the context of safe Rust, it rejects all programs with UB but also some without UB. (This is the correct choice!)</p>
<p>What if your program is in the no-man&rsquo;s-land where it gets rejected even though you know it doesn&rsquo;t have UB? To express those kinds of programs, the Rust compiler provides an escape hatch: the <code>unsafe</code> keyword. By writing it, the author takes responsibility for soundness: they vouch that no safe code can use this to cause UB, and that the compiler should trust them.</p>
<p>What does it mean for some unsafe code to not have UB? Informally, it must follow <a href="https://cacm.acm.org/research/safe-systems-programming-in-rust/">the rules of Rust</a>—all the rules that the compiler proves for safe Rust. Some examples of these rules are:</p>
<ul>
<li>There are no data races.</li>
<li>There must be no reads of uninitialized or freed memory.</li>
<li>There must not ever be multiple aliases of <code>&amp;mut</code> references to the same region of memory.</li>
<li>Immutable data must not be mutated.</li>
</ul>
<p>The rules of Rust are very hard to reason about! I consider them to be significantly harder than C, for example, particularly due to the rules around mutable aliasing<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>. So judicious use of unsafe Rust usually involves encapsulating it behind a safe abstraction.</p>
<h3 id="unsafe-rust-requires-reasoning-about-safe-rust">Unsafe Rust requires reasoning about safe Rust</h3>
<p>In all but the simplest of cases, it is not possible to be sure that such an abstraction is sound without also reasoning about the surrounding safe Rust. In this section, I&rsquo;ll walk through three examples in increasing order of difficulty.</p>
<p>The first example is the <a href="https://doc.rust-lang.org/std/primitive.slice.html#method.split_at_mut"><code>split_at_mut</code> method</a> on slices. This method splits a mutable slice into two separate parts. <code>split_at_mut</code> is a safe method, but safe Rust lacks the ability to express this kind of partitioning of a slice. Thus, <code>split_at_mut</code> requires unsafe Rust.</p>
<p>The implementation is roughly:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">split_at_mut</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="p">(</span><span class="n">slice</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="n">T</span><span class="p">],</span><span class="w"> </span><span class="n">mid</span>: <span class="kt">usize</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="p">[</span><span class="n">T</span><span class="p">],</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="p">[</span><span class="n">T</span><span class="p">])</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="fm">assert!</span><span class="p">(</span><span class="n">mid</span><span class="w"> </span><span class="o">&lt;=</span><span class="w"> </span><span class="n">slice</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">ptr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slice</span><span class="p">.</span><span class="n">as_mut_ptr</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">remaining</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">slice</span><span class="p">.</span><span class="n">len</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">mid</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">std</span>::<span class="n">slice</span>::<span class="n">from_raw_parts_mut</span><span class="p">(</span><span class="n">ptr</span><span class="p">,</span><span class="w"> </span><span class="n">mid</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">std</span>::<span class="n">slice</span>::<span class="n">from_raw_parts_mut</span><span class="p">(</span><span class="n">ptr</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">mid</span><span class="p">),</span><span class="w"> </span><span class="n">remaining</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>To be sure of the correctness of the unsafe block, you also have to consider several things about the surrounding safe code:</p>
<ul>
<li>that the function was provided a <code>&amp;mut [T]</code>, and not, for example, a <code>&amp;[T]</code></li>
<li>the <code>assert!</code> ensuring that <code>mid</code> is in bounds</li>
<li>the fact that <code>remaining</code> is <code>slice.len() - mid</code>, and not, for example, <code>(slice.len() - mid) + 1</code></li>
</ul>
<p>If any of these invariants (all of which are safe Rust) are not upheld, the safe abstraction becomes unsound.</p>
<p>The next level up is that in many cases you also have to analyze the <em>entire module</em> the unsafe code is present in. Consider this vector type:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">MyVec</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">ptr</span>: <span class="nc">NonNull</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">len</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">cap</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="n">MyVec</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">get</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span>: <span class="kt">usize</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nb">Option</span><span class="o">&lt;&amp;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">len</span><span class="p">).</span><span class="n">then</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">&amp;*</span><span class="bp">self</span><span class="p">.</span><span class="n">ptr</span><span class="p">.</span><span class="n">as_ptr</span><span class="p">().</span><span class="n">add</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The soundness of <code>get</code> relies on <strong>every other method</strong> in the module, safe and unsafe, ensuring that <code>len</code> is correct and in bounds. Encapsulation and privacy (<code>len</code> cannot be mutated from outside the module) are load-bearing here.</p>
<p>The hardest situations, though, are when unsafe code is generic and calls back into user-supplied code. In those cases, the <a href="https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html">fundamental principle</a> is:</p>
<p><em>Safe Rust code, no matter how pathological or adversarially written, must never cause unsafe code to exhibit undefined behavior.</em></p>
<p>In other words, you no longer have the luxury of analyzing just the safe code you see; you must put yourself in the shoes of an attacker, considering every safe Rust program one could write, and be resilient to <em>all</em> of them. For example, it is possible for user code to misbehave in a way that leads to internal tables or data structures being corrupted. Safe but buggy Rust may cause unsafe Rust to be slower, produce incorrect results, or even panic, but it must not cause UB!</p>
<p>(Why assume the worst? You might argue that well-meaning Rust developers are quite unlikely to write adversarial code. One reason for having a firm rule is that guarding against adversarial code means guarding against innocent mistakes as a corollary; a bright line makes it easier to assign responsibility for protection against UB.)</p>
<p>An example is the <a href="https://doc.rust-lang.org/std/iter/trait.ExactSizeIterator.html"><code>ExactSizeIterator</code></a> trait, which has a <code>len</code> method that returns the exact size of an iterator. A developer might be tempted to take advantage of this in unsafe Rust:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">collect_exact</span><span class="o">&lt;</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">I</span>: <span class="nb">ExactSizeIterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">T</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="n">iter</span>: <span class="nc">I</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">MyVec</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">iter</span><span class="p">.</span><span class="n">len</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MyVec</span>::<span class="n">with_capacity</span><span class="p">(</span><span class="n">n</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">item</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">iter</span><span class="p">.</span><span class="n">enumerate</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">std</span>::<span class="n">ptr</span>::<span class="n">write</span><span class="p">(</span><span class="n">v</span><span class="p">.</span><span class="n">ptr</span><span class="p">.</span><span class="n">as_ptr</span><span class="p">().</span><span class="n">add</span><span class="p">(</span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="n">item</span><span class="p">);</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">v</span><span class="p">.</span><span class="n">len</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">n</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">v</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This code does not cause UB if the iterator is well-behaved. But there&rsquo;s no guarantee of such behavior! It is entirely possible to write a broken <code>ExactSizeIterator</code> implementation that returns a bogus result, all in safe Rust:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">BadIterator</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">remaining</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">BadIterator</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">u32</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">next</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nb">Option</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">remaining</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="mi">0</span><span class="p">).</span><span class="n">then</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">remaining</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="mi">42</span><span class="w"> </span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="nb">ExactSizeIterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">BadIterator</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">len</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="kt">usize</span> <span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="mi">0</span><span class="w"> </span><span class="c1">// but this actually yields `self.remaining` items!
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>collect_exact</code> will write into unallocated memory in this case, so it is unsound in general<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>.</p>
<p>Another example is the <a href="https://doc.rust-lang.org/std/io/trait.Read.html"><code>std::io::Read</code> trait</a>, where a safe but buggy <code>Read</code> implementation can return the wrong number of bytes read. See <a href="https://github.com/rust-lang/rfcs/blob/master/text/2930-read-buf.md">Rust RFC 2930</a> for more discussion about this.</p>
<p><code>iddqd</code> is in exactly this situation: it consists of generic data structures which call into user-supplied code, so it is working at this highest level of difficulty<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>.</p>
<h2 id="how-iddqd-works">How iddqd works</h2>
<p><code>iddqd</code>&rsquo;s architecture is pretty straightforward: it uses a combination of a list of items (internally called an <code>ItemSet</code>) and a table with indexes into the slots. For example, <a href="https://docs.rs/iddqd/latest/iddqd/struct.IdHashMap.html"><code>IdHashMap&lt;T&gt;</code></a> consists of:</p>
<ul>
<li>an <code>ItemSet&lt;T&gt;</code>, which (similar to the <a href="https://docs.rs/slab"><code>slab</code> crate</a>) internally has:
<ul>
<li>a <code>Vec&lt;ItemSlot&lt;T&gt;&gt;</code>, where <code>ItemSlot&lt;T&gt;</code> is an enum with two variants:
<ul>
<li><code>Occupied(T)</code></li>
<li><code>Vacant { next: ItemIndex }</code> (where <code>ItemIndex</code> represents an integer index into the <code>Vec&lt;ItemSlot&lt;T&gt;&gt;</code>)</li>
</ul>
</li>
<li>a <code>free_head: ItemIndex</code>, which points to either the most recently freed <code>Vacant</code> slot, or is a sentinel (marked as <code>SENT</code> in the diagrams below) indicating that no free vacant slots are available. (The combination of <code>free_head</code> and the <code>Vacant</code> slots forms a <em>free chain</em>.)</li>
</ul>
</li>
<li>a <a href="https://docs.rs/hashbrown/latest/hashbrown/struct.HashTable.html"><code>hashbrown::HashTable</code></a> over <code>ItemIndex</code> (<code>hashbrown</code> is the fast hash table implementation used by the standard library)</li>
</ul>
<p>For example, an <code>IdHashMap</code> holding three items might look like the diagram below. The top row is the hash table over <code>ItemIndex</code>; the middle row is the <code>Vec&lt;ItemSlot&lt;T&gt;&gt;</code>.</p>

  <figure class="center" >
    
    <img src="/images/iddqd/iddqd-diagram-01.svg"  alt="An IdHashMap holding three items. The top row is a hash table mapping hash(k₁) → 0, hash(k₀) → 3, and hash(k₂) → 4. The middle row is the item vector: slot 0 Occupied(t₁), slot 1 Vacant (next: SENT), slot 2 Vacant (next: 1), slot 3 Occupied(t₀), slot 4 Occupied(t₂). free_head points to slot 2."   />
    
    
  </figure>


<p>Adding a new item consists of checking <code>free_head</code> to see if any <code>Vacant</code> slots are available. If so, we read the <code>next</code> index out of that <code>Vacant</code> slot, overwrite the slot with <code>Occupied(...)</code>, then set <code>free_head</code> to the <code>next</code> value we read. Otherwise, we push a new <code>Occupied</code> slot to the end. Finally, the hash table is updated, fetching the key and computing the hash, then recording the new index in the hash table.</p>
<p>Starting from the state above: Let&rsquo;s say we want to insert a new item <code>t₃</code>. Calling <code>insert(t₃)</code> recycles slot <code>2</code> from the free chain, advances <code>free_head</code> to <code>1</code>, and records the new hash entry:</p>

  <figure class="center" >
    
    <img src="/images/iddqd/iddqd-diagram-02.svg"  alt="The same IdHashMap after inserting t₃. Slot 2 is recycled from the free chain and now holds Occupied(t₃), and a new hash entry hash(k₃) → 2 is added. free_head advances to slot 1."   />
    
    
  </figure>


<p>Removing an item given a key involves this process in reverse: first, consult the hash table using the hash computed from the key, and remove the found index from the hash table. With the index in hand, replace the <code>ItemSlot</code> with <code>Vacant</code>, setting the internal <code>next</code> to the current value of <code>free_head</code>. Finally, set <code>free_head</code> to the index that was just replaced with <code>Vacant</code>.</p>
<p>Continuing from the post-insert state above: Calling <code>remove(k₀)</code> drops the hash entry for <code>k₀</code>, marks slot <code>3</code> as <code>Vacant</code> with its <code>next</code> pointing at the old <code>free_head</code> (<code>1</code>), and updates <code>free_head</code> to <code>3</code>:</p>

  <figure class="center" >
    
    <img src="/images/iddqd/iddqd-diagram-04.svg"  alt="The IdHashMap after removing k₀. The hash entry for k₀ is dropped, slot 3 becomes Vacant with next pointing at the old free_head (1), and free_head updates to 3."   />
    
    
  </figure>


<p>This pattern generalizes to the other map types:</p>
<ul>
<li><a href="https://docs.rs/iddqd/latest/iddqd/struct.IdOrdMap.html"><code>IdOrdMap</code></a> follows a similar structure, except the hash table is replaced with a <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/src/support/btree_table.rs">B-tree index</a><sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>.</li>
<li><a href="https://docs.rs/iddqd/latest/iddqd/struct.BiHashMap.html"><code>BiHashMap</code></a> and <a href="https://docs.rs/iddqd/latest/iddqd/struct.TriHashMap.html"><code>TriHashMap</code></a> store two and three hash tables respectively.</li>
</ul>
<h2 id="unsafe-rust-in-iddqd">Unsafe Rust in iddqd</h2>
<p><code>iddqd</code> has a few different kinds of unsafe Rust within it. Most uses of unsafe follow <a href="https://github.com/oxidecomputer/iddqd/blob/1588c91bb7d120f9175533c0f36fdcf58c84f13c/crates/iddqd/src/support/borrow.rs#L18">well-established patterns</a> also used <a href="https://github.com/rust-lang/rust/blob/c854ac880bf68671f56e6a28e4ff5cb7e92da62c/library/alloc/src/collections/btree/borrow.rs#L14">within Rust&rsquo;s standard library</a>—these are not dependent on user code, and a sufficiently smart borrow checker would make these uses of unsafe unnecessary. But there are a few that stand out.</p>
<p>Perhaps the most challenging one to reason about is mutable iteration over items. For <a href="https://docs.rs/iddqd/latest/iddqd/struct.IdOrdMap.html#method.iter_mut"><code>IdOrdMap::iter_mut</code></a>, iteration is ordered by the key. One might start off by writing something like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">IterMut</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="p">,</span><span class="w"> </span><span class="n">T</span>: <span class="nc">IdOrdItem</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">items</span>: <span class="kp">&amp;</span><span class="na">&#39;a</span> <span class="nc">mut</span><span class="w"> </span><span class="n">ItemSet</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// This is an ordered iterator over the B-tree equivalent to a hash table.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">iter</span>: <span class="nc">btree_table</span>::<span class="n">Iter</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="p">,</span><span class="w"> </span><span class="n">T</span>: <span class="nc">IdOrdItem</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="w"> </span><span class="nb">Iterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">IterMut</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="p">,</span><span class="w"> </span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="na">&#39;a</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">T</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="cp">#[inline]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">next</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nb">Option</span><span class="o">&lt;</span><span class="bp">Self</span>::<span class="n">Item</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Obtain the next index to return, in order.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">iter</span><span class="p">.</span><span class="n">next</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Fetch the item.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">item</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">T</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">items</span><span class="p">[</span><span class="n">index</span><span class="p">];</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Some</span><span class="p">(</span><span class="n">item</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>But this would immediately return an error:</p>
<pre class="chroma rustc-error"><code><span class="level-error">error</span><span class="bold">: lifetime may not live long enough</span>
<span class="info">   --&gt;</span> crates/iddqd/src/id_ord_map/iter.rs:90:9
<span class="info">    |</span>
<span class="info"> 79 |</span> impl&lt;'a, T: IdOrdItem + 'a&gt; Iterator for IterMut&lt;'a, T&gt; {
<span class="info">    |</span>      <span class="info">-- lifetime `'a` defined here</span>
<span class="info">...</span>
<span class="info"> 83 |</span>     fn next(&amp;mut self) -&gt; Option&lt;Self::Item&gt; {
<span class="info">    |</span>             <span class="info">- let's call the lifetime of this reference `'1`</span>
<span class="info">...</span>
<span class="info"> 90 |</span>         Some(item)
<span class="info">    |</span>         <span class="level-error">^^^^^^^^^^ method was supposed to return data with lifetime `'a`</span>
<span class="info">    |</span>                    <span class="level-error">but it is returning data with lifetime `'1`</span></code></pre>
<p>The reason this occurs is that Rust&rsquo;s iterators require that the returned values do not borrow from the iterator itself<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup>. Many uses of iterators do not need this capability—in particular, most for loops only operate on one item at a time:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">IdOrdMap</span>::<span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ... insert some items
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">for</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">item</span><span class="p">.</span><span class="n">value</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>But Rust&rsquo;s iterators also let you create values that outlive the iterator, via e.g. the <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.collect"><code>collect</code> combinator</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">items</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">IdOrdMap</span>::<span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span>::<span class="n">new</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ... insert some items
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">items_mut</span>: <span class="nb">Vec</span><span class="o">&lt;&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">items</span><span class="p">.</span><span class="n">iter_mut</span><span class="p">().</span><span class="n">collect</span><span class="p">();</span><span class="w">
</span></span></span></code></pre></div><p><code>items_mut</code> cannot outlive <code>items</code> itself, but <em>can</em> outlive the iterator returned by <code>items.iter_mut()</code>.</p>
<p>Crucially, this relies on knowing that all of the <code>&amp;mut T</code> returned by the iterator are disjoint—or, in other words, that the iterator never yields the same value twice. (Remember that one of the rules of Rust is that there must never be multiple <code>&amp;mut</code> references to the same memory.) But the Rust borrow checker just sees a succession of accesses into a list, and it has no way of knowing that the indexes are all different. If the human writing the code has knowledge that all of the indexes are different, though, they can use an unsafe pattern called <a href="https://doc.rust-lang.org/stable/std/mem/fn.transmute.html#examples">lifetime extension</a> to tell the borrow checker to let this code through:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">impl</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="p">,</span><span class="w"> </span><span class="n">T</span>: <span class="nc">IdOrdItem</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="na">&#39;a</span><span class="o">&gt;</span><span class="w"> </span><span class="nb">Iterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">IterMut</span><span class="o">&lt;</span><span class="na">&#39;a</span><span class="p">,</span><span class="w"> </span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="na">&#39;a</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">T</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="cp">#[inline]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">next</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nb">Option</span><span class="o">&lt;</span><span class="bp">Self</span>::<span class="n">Item</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Obtain the next index to return, in order.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">iter</span><span class="p">.</span><span class="n">next</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Fetch the item.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">item</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">T</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">items</span><span class="p">[</span><span class="n">index</span><span class="p">];</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Extend the lifetime of the item to &#39;a.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// 
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// SAFETY: The indexes returned by `self.iter` are known to all be different,
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// so successive calls to `IterMut::next` never create mutable aliases to
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// the same memory.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">std</span>::<span class="n">mem</span>::<span class="n">transmute</span>::<span class="o">&lt;&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="na">&#39;a</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">T</span><span class="o">&gt;</span><span class="p">(</span><span class="n">item</span><span class="p">)</span><span class="w"> </span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Some</span><span class="p">(</span><span class="n">item</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><h3 id="a-bug-and-a-fix">A bug and a fix</h3>
<p>In the previous section, we saw how unsafe Rust acts as an escape hatch for facts that the Rust compiler cannot establish by itself—in this case, that the indexes returned by <code>self.iter</code> are all different. But <em>are the indexes actually different</em>? Based on the storage scheme described above, it would seem like that is the case. But this is generic code which calls into user-provided functions, so there&rsquo;s a chance that sufficiently pathological user code can trick the map into storing duplicate indexes.</p>
<p>Let&rsquo;s take one such example we recently found and fixed. Suppose you have an <code>IdOrdMap</code> called <code>items</code> storing five items with integer keys 0 through 4, inserted and stored in order. Here, the top row is the B-tree storing indexes in key order, while the bottom row is the <code>Vec&lt;ItemSlot&lt;T&gt;&gt;</code>:</p>

  <figure class="center" >
    
    <img src="/images/iddqd/iddqd-diagram-05.svg"  alt="An IdOrdMap storing five items with integer keys 0 through 4. The top row is a B-tree of (key, index) pairs in key order; the bottom row is the item vector, with each B-tree entry pointing to its matching Occupied slot."   />
    
    
  </figure>


<p>Now suppose you use the <a href="https://docs.rs/iddqd/latest/iddqd/struct.IdOrdMap.html#method.entry">Entry API</a> (a standard Rust data structure pattern) to fetch an item by index 0:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// In this example, the integer keys are wrapped in another type. We&#39;re going
</span></span></span><span class="line"><span class="cl"><span class="c1">// to swap in a different Ord for Key in the next step.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">entry</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">items</span><span class="p">.</span><span class="n">entry</span><span class="p">(</span><span class="n">Key</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span><span class="w">
</span></span></span></code></pre></div><p>This will successfully look up the item stored at index 0 and return an <a href="https://docs.rs/iddqd/latest/iddqd/id_ord_map/struct.OccupiedEntry.html"><code>OccupiedEntry</code></a>. The <code>OccupiedEntry</code> internally stores the fact that the item was found at index 0.</p>
<p>Now, at this point, let&rsquo;s deliberately switch the <code>Ord</code> implementation for <code>Key</code> to always return <a href="https://doc.rust-lang.org/std/cmp/enum.Ordering.html#variant.Equal"><code>Equal</code></a>, no matter the values inside. There are a few different ways to do this in safe Rust, most of which use some form of <a href="https://doc.rust-lang.org/reference/interior-mutability.html">interior mutability</a>. (This is unlikely in ordinary Rust, but <em>that doesn&rsquo;t matter</em>: unsafe generic Rust must handle arbitrarily pathological Rust!)</p>
<p>Now remove the entry from the map using <a href="https://docs.rs/iddqd/latest/iddqd/id_ord_map/struct.OccupiedEntry.html#method.remove"><code>OccupiedEntry::remove</code></a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">entry</span><span class="p">.</span><span class="n">remove</span><span class="p">();</span><span class="w">
</span></span></span></code></pre></div><p>At this point, <code>iddqd</code> would attempt to remove <em>index</em> 0 from the B-tree, descending into the tree again but comparing only by <em>key</em><sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup>. Because we made comparisons by key always return <code>Equal</code>, the comparison would short-circuit at the first element it compares against. For example, if the first comparison lands on the B-tree entry <code>(key = 2, index = 2)</code>, then that entry would be removed. But the <code>OccupiedEntry</code> was carrying <code>index = 0</code>, so it would also remove item 0 from the item set:</p>

  <figure class="center" >
    
    <img src="/images/iddqd/iddqd-diagram-03.svg"  alt="The IdOrdMap left inconsistent after a buggy removal. The B-tree entry for key=2/index=2 was removed instead of key=0/index=0; slot 0 is now Vacant with free_head = 0 even though its B-tree entry remains, and slot 2 (key=2) is orphaned with nothing pointing to it."   />
    
    
  </figure>


<p>The map is no longer in a consistent state. The index 0 is still in the B-tree index but the corresponding slot in the <code>ItemSet</code> is vacant; meanwhile, item 2 in the <code>ItemSet</code> does not have any pointers to it.</p>
<p>At this point, though, the invariant that there aren&rsquo;t duplicate indexes hasn&rsquo;t been violated. Where it does get violated is with this next step: suppose the <code>Ord</code> implementation now switches back to being honest<sup id="fnref:8"><a href="#fn:8" class="footnote-ref" role="doc-noteref">8</a></sup>, and you insert a new entry with key 1000. Since <code>free_head</code> points to item 0, <code>iddqd</code> will attempt to reuse that slot. It is <em>this</em> step that results in duplicate indexes:</p>

  <figure class="center" >
    
    <img src="/images/iddqd/iddqd-diagram-06.svg"  alt="The IdOrdMap after inserting key=1000 into the reused slot 0. Two B-tree entries, key=0/index=0 and key=1000/index=0, now both point at slot 0, creating duplicate indexes, while slot 2 remains orphaned."   />
    
    
  </figure>


<p>There are now duplicate indexes pointing to the same item, and <code>IterMut</code> becomes unsound.</p>
<p>This was fixed through a combination of two things:</p>
<ol>
<li>
<p><strong>When descending into the B-tree, also check equality against the index.</strong> In other words, if we&rsquo;re looking for a key with a known index in the B-tree, <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/src/support/btree_table.rs#L458-L476">compare against <em>both</em> <code>key</code> and <code>index</code></a>. As covered in the example above, previously, we would just check that the key is the same; the pathological <code>Ord</code> implementation returned <code>Equal</code>, tricking the map into believing that the entry at <code>(key = 2, index = 2)</code> was the one to remove. We changed this to additionally use the index as a tie-breaker. The comparison against <code>(key = 2, index = 2)</code> now resolves to <code>Less</code> (since <code>0 &lt; 2</code>), so the search doesn&rsquo;t spuriously match here.</p>
<p>By construction, the only way the search can now succeed is if the stored index actually equals the one we&rsquo;re looking for.</p>
</li>
<li>
<p><strong>If there are no matches, fall back to a linear scan.</strong> The tie-breaker eliminates spurious matches on the wrong entry, but it doesn&rsquo;t guarantee that the search finds the <em>right</em> entry. (The B-tree is sorted by <em>key</em>, while the tie-breaker compares by <em>index</em>, and in general these two orderings are independent.) If the tree search yields no results, we must still keep the tree and the <code>ItemSet</code> in sync. In that case, we remove the index from the B-tree by <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/src/support/btree_table.rs#L267-L276">doing a linear scan</a> without calling into user code. The remove operation then takes linear rather than logarithmic time, but that is an acceptable tradeoff, since buggy user code is the only way to encounter the fallback.</p>
</li>
</ol>
<p>This is the kind of long-range reasoning that is sometimes required to establish the soundness of a safe abstraction. The overall system relies on all of these moving parts working correctly across arbitrarily complex series of operations.</p>
<h2 id="validating-iddqd">Validating iddqd</h2>
<p>Because <code>iddqd</code> is such a foundational data structure at Oxide, we go through great lengths to validate its correctness, including the soundness of its abstractions. There is a great deal of analytical reasoning performed by experienced Rust authors and reviewers, as described in the section above. But we also empirically validate <code>iddqd</code> along several different dimensions. None of the layers is sufficient by itself, but by rigorously doing all of them we can be more confident that <code>iddqd</code> is correct.</p>
<p>The layers of validation we apply are:</p>
<table>
	<thead>
			<tr>
					<th>Technique</th>
					<th>Catches</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Analytical reasoning</td>
					<td>Soundness errors</td>
			</tr>
			<tr>
					<td>Example-based tests (including doctests)</td>
					<td>Regressions, broken examples</td>
			</tr>
			<tr>
					<td>Pathological tests with Miri</td>
					<td>Adversarial trait impls causing UB</td>
			</tr>
			<tr>
					<td>Model-based tests (PBT with <code>NaiveMap</code> oracle)</td>
					<td>Divergence from oracle over random operation sequences</td>
			</tr>
			<tr>
					<td>Panic safety fault injection</td>
					<td>Invariant violations on mid-operation panics</td>
			</tr>
			<tr>
					<td>LLM adversarial review</td>
					<td>Blind spots in adversarial user code</td>
			</tr>
			<tr>
					<td>Formal verification</td>
					<td>Proofs over arbitrarily adversarial trait impls</td>
			</tr>
	</tbody>
</table>
<p>In this section, I&rsquo;ll provide a brief overview of each layer. Click through the links to see more details and code samples.</p>
<h3 id="analytical-reasoning">Analytical reasoning</h3>
<p>All blocks and patterns of unsafe Rust have been analyzed by at least one human reviewer, and up to three. Thanks to several of my Oxide colleagues for lending their time and expertise to these reviews—these discussions helped sharpen our reasoning considerably.</p>
<p>The reasoning for each unsafe block is captured in a <code>// SAFETY:</code> comment above the block. We use Clippy&rsquo;s <a href="https://rust-lang.github.io/rust-clippy/master/index.html?search=undocumented_unsafe_blocks#undocumented_unsafe_blocks"><code>undocumented_unsafe_blocks</code></a> lint to verify that these comments are present.</p>
<h3 id="example-based-tests">Example-based tests</h3>
<p>The simplest kind of empirical validation is with <strong>example-based tests</strong>: create a map, perform some operations, and ensure that the results of those operations are as expected. <code>iddqd</code> has <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/src/support/btree_table.rs#L639">unit tests for its internals</a> and <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/tests/integration/id_ord_map.rs">integration tests for the public API</a>. These also live as doctests, where they double as documentation.</p>
<h3 id="pathological-tests-with-miri">Pathological tests with Miri</h3>
<p><code>iddqd</code> also has a <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/tests/integration/pathological.rs">battery of <strong>pathological tests</strong></a> which supply buggy implementations of <code>Ord</code> and other traits.</p>
<p>In CI, both regular and pathological tests are run under the <a href="https://github.com/rust-lang/miri/">Miri interpreter</a>, which can detect many classes of undefined behavior. (If you&rsquo;re curious about the details, we run with Miri under both Stacked and <a href="https://www.ralfj.de/blog/2023/06/02/tree-borrows.html">Tree Borrows</a>.) But note that some classes of UB can be detected in the regular test environment as well. For example, in the previous section we established that the tables not containing duplicate indexes is a necessary condition for soundness; this invariant <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/src/support/btree_table.rs#L113-L159">can be verified</a> outside the Miri context as well.</p>
<h3 id="model-based-testing-and-fault-injection">Model-based testing and fault injection</h3>
<p><code>iddqd</code> uses two layers of randomized testing: model-based comparison against a known-correct oracle, and fault injection on top of it.</p>
<p>For data structures, example-based tests alone are generally considered insufficient. A much stronger kind of testing is <strong>model-based testing</strong>, also known as <strong>stateful property-based testing</strong> (stateful PBT): random sequences of operations are applied to an instance of the type, and the results are compared against an <em>oracle</em> that is known to be correct. <code>iddqd</code> has extensive model-based tests against <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd-test-utils/src/naive_map.rs#L9">a <code>NaiveMap</code> oracle</a> that is inefficient but clearly correct. (I&rsquo;ve talked about this style of testing before <a href="https://www.youtube.com/watch?v=CcYK8E1UuJY">on <em>Oxide and Friends</em></a>.)</p>
<p>An extension of model-based testing is <strong>fault injection</strong>, where bugs are randomly inserted into user code. For <code>iddqd</code>, a fruitful avenue for fault injection has been <em>panic safety</em> (or <em>unwind safety</em>): user code panicking in the middle of an operation must not cause the map&rsquo;s invariants to be violated<sup id="fnref:9"><a href="#fn:9" class="footnote-ref" role="doc-noteref">9</a></sup>. We systematically explore fault injection by <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/tests/integration/id_ord_map.rs#L1113-L1255">generating random sequences</a> of map operations, where each operation is associated with a (randomly selected) panic countdown. Each call into user code decrements the countdown by 1; if it reaches 0, the code panics. Randomized panic safety testing found several subtle bugs in <code>iddqd</code> (<a href="https://github.com/oxidecomputer/iddqd/pull/259">example</a>), including some that escalated into unsoundness.</p>
<p>The model-based tests also verify internal invariants after each operation, such as the no-duplicate-index condition described in the previous section. We&rsquo;ve found in practice that model-based tests are too slow to run under Miri, so we verify the invariants on which soundness (and correctness) is known to be dependent instead.</p>
<h3 id="llm-adversarial-review">LLM adversarial review</h3>
<p>Another kind of validation that&rsquo;s recently become available is <strong>LLM-driven adversarial review</strong>. Current-generation frontier models<sup id="fnref:10"><a href="#fn:10" class="footnote-ref" role="doc-noteref">10</a></sup> found several ways to write pathological implementations of user code that would corrupt the map. In <a href="https://github.com/oxidecomputer/iddqd/pull/260">one notable case</a>, an LLM constructed a way for the map&rsquo;s invariants to break on a custom allocator panicking and unwinding. This is a panic safety issue (and a failure mode I hadn&rsquo;t thought about before), but distinct from the existing panic safety tests that only covered panics in regular user code like an <code>Ord</code> implementation.</p>
<p>LLMs can sometimes produce plausible-but-wrong soundness claims. An effective way to guard against that is <a href="https://simonwillison.net/guides/agentic-engineering-patterns/red-green-tdd/">red-green TDD</a>, using Miri as an oracle: For a soundness bug, first have the LLM write a test case demonstrating the bug, running it under Miri to show undefined behavior—the <em>red</em> phase. Then, after fixing the bug, rerun the test to show that it now passes—the <em>green</em> phase.</p>
<h3 id="what-about-formal-methods">What about formal methods?</h3>
<p>To formally verify <code>iddqd</code>, one&rsquo;s first thought would be to use a <a href="https://en.wikipedia.org/wiki/Model_checking">model checker</a> like <a href="https://github.com/model-checking/kani">Kani</a> to establish that the maps don&rsquo;t violate internal invariants. But <code>iddqd</code> is unfortunately a bit too complicated for Kani to handle, and the required proofs blow up beyond what is feasible for the tool.</p>
<p>The <a href="https://creusot.rs/">Creusot</a> deductive verifier can help Rust developers prove their code is free of panics and other errors, but as of this writing it&rsquo;s <a href="https://github.com/creusot-rs/creusot/blob/7a68885eb85a3bea6f65e7cb50b9013e4aa7d1dc/creusot-std/src/std/panicking.rs#L10-L45">unable to prove</a> invariants that must hold even if user code panics or otherwise misbehaves.</p>
<p>The infeasibility of proofs is a common problem with applying formal methods to implementations, so they are often applied to a higher-level specification instead. For <code>iddqd</code>, the closest thing to a specification is <code>NaiveMap</code>, but it can be easily observed to be correct without needing a formal proof.</p>
<p>How do we ensure that the implementation matches the specification? Model-based testing does a lot of heavy lifting here. While they aren&rsquo;t a formal proof that <code>iddqd</code> matches <code>NaiveMap</code>, running model-based tests thousands of times in CI provides fairly high confidence that it does.</p>
<p>We keep an eye on developments in this space; the in-development <a href="https://crates.io/crates/cargo-anneal"><code>cargo-anneal</code></a>, which lets you embed <a href="https://lean-lang.org/">Lean</a> soundness proofs alongside unsafe Rust as doc comments, looks quite interesting.</p>
<p>If you&rsquo;re working on formal verification tooling, <code>iddqd</code> is a great candidate to benchmark against because of its crisp invariants and relatively constrained yet non-trivial scope. We&rsquo;d be especially interested in proofs that hold over arbitrary trait implementations, and of refinement between <code>iddqd</code> and <code>NaiveMap</code>. Please reach out by <a href="https://github.com/oxidecomputer/iddqd/issues/new">filing an issue</a> or <a href="mailto:rain@oxide.computer">emailing me</a>!</p>
<hr>
<p><em>Update 2026-06-16</em>: Since publication, we&rsquo;ve used <a href="https://soteria-tools.com/">Soteria</a> to <a href="https://github.com/oxidecomputer/iddqd/tree/main/crates/iddqd/tests/soteria">verify</a> that <code>iddqd</code>&rsquo;s no-duplicate-index invariant holds under arbitrarily adversarial <code>Hash</code> and <code>Ord</code> implementations. Soteria is similar to Kani in spirit, though with <a href="https://arxiv.org/abs/2511.08729">different internals</a> that make it more compatible with <code>iddqd</code>. Soteria is implemented as a symbolic execution engine that lets you declare points of nondeterminism in your code. (For example, within <code>iddqd</code>, we represent an adversarial <a href="https://github.com/oxidecomputer/iddqd/blob/a794b0a57d18ca3ef5cc261cfbcb6b970301acb4/crates/iddqd/tests/soteria/hasher.rs#L53-L54"><code>Hash</code></a> or <a href="https://github.com/oxidecomputer/iddqd/blob/a794b0a57d18ca3ef5cc261cfbcb6b970301acb4/crates/iddqd/tests/soteria/id_ord_map.rs#L63-L70"><code>Ord</code></a> return value as nondeterministic, since we want to model adversarially bad user code.) Then, it:</p>
<ul>
<li>Compiles the Rust program into a form suitable for formal verification.</li>
<li>Interprets this form until it reaches a point of nondeterminism, treating each such point as a symbol rather than a concrete value.</li>
<li>Branches at each point where control flow depends on a symbol, validating that there is no UB and that the invariants specified by the program are upheld. A lot of the cleverness lies in how the possibility space is collapsed down to something manageable. For example, Soteria doesn&rsquo;t have to enumerate every possible <code>Hash</code> value; rather, it only has to fork in places where control flow depends on the <code>Hash</code> value.</li>
</ul>
<p>For <code>IdOrdMap</code> from the example above, the proof runs against the real B-tree and its unsafe code, and it checks exactly the index tie-breaker and linear search fallback described here. (Remove either of them and the proof fails.)</p>
<p>The benefits and limits of this approach are both apparent. It covers every possible return value at once, but to stay fast enough for CI, we have to restrict the proof to operation sequences of depth 3. (In contrast, the property-based tests generate sequences of hundreds of operations.) So it acts as a complement to the existing layers of validation, not a replacement.</p>
<p>We plan to keep expanding the scope of formal verification within <code>iddqd</code> as the tooling continues to mature.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Writing unsafe generic Rust is extremely difficult. Each invariant that the unsafe code relies upon has to be carefully upheld over arbitrary trait implementations, including adversarial ones. This post covers one such example: how an <code>Ord</code> implementation could be carefully written to trick the map into creating mutable aliases to the same memory, and how we fixed it.</p>
<p>No single technique can hope to catch all bugs, which is why <code>iddqd</code> uses several layers of validation. Humans carefully reasoning about every line of unsafe code goes a long way; example-based, pathological, and randomized tests provide empirical evidence; and frontier models can find new and surprising ways to break code that humans might not have thought of. This is a lot of machinery, but at Oxide <a href="https://oxide.computer/principles">we believe</a> that for foundational infrastructure, this level of rigor is justified. And if you agree, <a href="https://oxide.computer/careers">we&rsquo;re hiring</a>!</p>
<p><em>Diagrams courtesy Ben Leonard. Discuss on <a href="https://news.ycombinator.com/item?id=48371762">Hacker News</a> and <a href="https://lobste.rs/s/6eyhii/iddqd_hardest_kind_unsafe_rust">Lobsters</a>.</em></p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p><code>serde_json</code> also supports integer keys; it serializes and deserializes them as strings.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>In C, the aliasing rules are about <em>accesses</em>: multiple pointers to the same memory are fine, as long as you don&rsquo;t have a data race on them or violate <a href="https://blog.regehr.org/archives/1307">strict aliasing</a>. In Rust, it is UB for multiple <code>&amp;mut</code> references to the same memory to ever <em>exist</em>, even if they&rsquo;re not actively being mutated simultaneously.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>An exercise for the reader. There is a second, entirely different, kind of bug in <code>collect_exact</code>: it can leak memory with some user-supplied iterators. This isn&rsquo;t a soundness bug, but it is a bug nonetheless. Try working out how this can happen and why this isn&rsquo;t a soundness bug. (Hint: think about user code panicking.)&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>For this discussion we&rsquo;re restricting ourselves to single-threaded Rust, since iddqd is not a concurrent data structure. Concurrency amplifies the difficulty of reasoning about Rust several-fold further.&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5">
<p>The index is built on top of the standard library <code>BTreeMap</code>. This required making <code>BTreeMap</code> work <a href="https://github.com/oxidecomputer/iddqd/blob/dd52024760267ce0a5bcd5b98a745a4b4468ebe5/crates/iddqd/src/support/btree_table.rs#L22-L69">with an external comparator</a> through some further unsafe trickery.&#160;<a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6">
<p>For a proposed version of an iterator which does let you borrow values from the iterator, see <a href="https://rust-lang.github.io/generic-associated-types-initiative/design_patterns/iterable.html"><code>LendingIterator</code></a>.&#160;<a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7">
<p>Is this second descent necessary? <a href="https://doc.rust-lang.org/std/collections/btree_map/struct.OccupiedEntry.html"><code>BTreeMap</code>&rsquo;s own <code>OccupiedEntry</code></a> stores a cursor into the map to avoid a second descent, and <code>iddqd</code> could be restructured to hold one. But there are other ways to trigger the same issue without using <code>Entry</code>, for example by using a chaotic <code>Ord</code> implementation that randomly returns <code>Equal</code> at times. I&rsquo;m using <code>Entry</code> here as the clearest way to demonstrate the bug.&#160;<a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:8">
<p>Restoring <code>Ord</code> is necessary here because, with <code>Equal</code>, the duplicate-detection step at the start of <code>insert_unique</code> would short-circuit and reject the insert. The point is that user code is free to misbehave in arbitrary ways, including changing its mind from one call to the next.&#160;<a href="#fnref:8" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:9">
<p>Rust binaries <a href="https://doc.rust-lang.org/rust-by-example/error/abort_unwind.html">can be configured</a> to either unwind (be <a href="https://doc.rust-lang.org/std/panic/fn.catch_unwind.html">recovered from</a>) or abort (tear down the process). Panic safety only matters with <code>panic = 'unwind'</code>. At Oxide, we ship our system software with <code>panic = 'abort'</code>. However, libraries cannot assume <code>abort</code> and <a href="https://sunshowers.io/posts/on-poisoning/#just-ship-with-panic--abort"><em>must</em> work with <code>unwind</code></a>.&#160;<a href="#fnref:9" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:10">
<p>Claude Opus 4.7 and GPT-5.5.&#160;<a href="#fnref:10" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/><category scheme="https://sunshowers.io/tags/unsafe" term="unsafe" label="unsafe"/></entry><entry><title type="html">In defense of lock poisoning in Rust</title><link href="https://sunshowers.io/posts/on-poisoning/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/socketaddrv6-not-roundtrip/?utm_source=atom_feed" rel="related" type="text/html" title="`SocketAddrV6` is not roundtrip serializable"/><link href="https://sunshowers.io/posts/cancelling-async-rust/?utm_source=atom_feed" rel="related" type="text/html" title="Cancelling async Rust"/><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="related" type="text/html" title="Why nextest is process-per-test"/><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="related" type="text/html" title="Open and closed universes"/><id>https://sunshowers.io/posts/on-poisoning/</id><published>2025-12-02T00:00:00+00:00</published><updated>2025-12-06T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>It&rsquo;s worth retaining one of multithreaded Rust&rsquo;s most valuable features.</blockquote><p>There&rsquo;s recently been some discussion about the benefits and downsides of lock (mutex) poisoning in Rust, spurred by <a href="https://github.com/rust-lang/rust/issues/149359">a recent proposal</a> to make the default mutex non-poisoned, i.e. <strong>silently unlock on panic</strong> (see also, <a href="https://news.ycombinator.com/item?id=46036883">recent discussion on Hacker News</a>). As a passionate defender of lock poisoning, I thought I&rsquo;d gather and write about my thoughts on this matter.</p>
<p>To summarize, I believe:</p>
<ul>
<li>Unexpected cancellations in critical sections cause real harm to system correctness.</li>
<li>Lock poisoning is an important part of ensuring the correctness of critical sections in Rust.</li>
<li>Poisoning applies more generally than mutexes, and providing an easy way to track that (via e.g. a <code>Poison&lt;T&gt;</code> wrapper) is valuable.</li>
<li>While there is conceptual elegance in separating out locking from poisoning on panic, the importance of lock poisoning overrides these concerns.</li>
</ul>
<h2 id="what-is-poisoning">What is poisoning?</h2>
<p>Rust, like most multithreaded languages, has <a href="https://doc.rust-lang.org/std/sync/struct.Mutex.html">mutexes</a>: a construct to ensure that a particular piece of data can only be accessed by one thread at a time. The way mutexes work in Rust is particularly well-considered:</p>
<ul>
<li>Rust uses a single-ownership model, and the notion of shared (<code>&amp;</code>) and exclusive (<code>&amp;mut</code>) references to some data. Most data structures are written such that mutations always require a <code>&amp;mut</code> reference to it.</li>
<li>In Rust, the data guarded by a mutex is owned by the mutex. (In many other languages, you have to track the mutex and data separately, and it&rsquo;s easy to get it wrong.)</li>
<li>When you lock a mutex, you start from a shared reference: a <code>&amp;Mutex&lt;T&gt;</code>. Once you have obtained the lock, you get back a <a href="https://doc.rust-lang.org/std/sync/struct.MutexGuard.html"><code>MutexGuard&lt;T&gt;</code></a>, which indicates that you now have exclusive access to the guarded data.</li>
<li>The <code>MutexGuard</code> can give you a <code>&amp;mut T</code>, so you have exclusive access to it.</li>
<li>When the <code>MutexGuard</code> is dropped, the lock is released. The period during which the lock is held is called the <em>critical section</em> (generally, not just in Rust).</li>
</ul>
<p>This is all quite reasonable! Let&rsquo;s look at an example that processes incoming messages for a set of tracked operations. Let&rsquo;s assume that multiple threads could be processing messages, so we have to guard the internal state with a mutex. (We&rsquo;ll discuss alternative approaches to this problem later.)</p>
<p>A simple implementation:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="p">{</span><span class="n">collections</span>::<span class="n">HashMap</span><span class="p">,</span><span class="w"> </span><span class="n">sync</span>::<span class="n">Mutex</span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">OperationId</span><span class="p">(</span><span class="cm">/* ... */</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">enum</span> <span class="nc">OperationState</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">InProgress</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* ... */</span><span class="w"> </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Completed</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* ... */</span><span class="w"> </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">OperationState</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Here, `process_message` consumes self and returns self. In practice this
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// is often because the state has some internal data that requires
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// processing by ownership.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">process_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">message</span>: <span class="nc">Message</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">match</span><span class="w"> </span><span class="bp">self</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* ... */</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">Operations</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">ops</span>: <span class="nc">Mutex</span><span class="o">&lt;</span><span class="n">HashMap</span><span class="o">&lt;</span><span class="n">OperationId</span><span class="p">,</span><span class="w"> </span><span class="n">OperationState</span><span class="o">&gt;&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Operations</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="sd">/// Process a message, updating the internal operation state appropriately.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">process</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">id</span>: <span class="kp">&amp;</span><span class="nc">OperationId</span><span class="p">,</span><span class="w"> </span><span class="n">message</span>: <span class="nc">Message</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Obtain a lock on the HashMap.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">lock</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">lock</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Once the lock has been acquired, it&#39;s guaranteed that no other
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// threads have any kind of access to the data. So a `&amp;mut` reference
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// can safely be handed to us.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// 
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// This step is shown for pedagogical reasons. Generally, `ops` is not
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// obtained explicitly. Instead, lock.remove and lock.insert are used
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// directly as `lock` dereferences to the underlying HashMap.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">ops</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">HashMap</span><span class="o">&lt;</span><span class="n">_</span><span class="p">,</span><span class="w"> </span><span class="n">_</span><span class="o">&gt;</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="o">*</span><span class="n">lock</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Retrieve the element from the map to process it.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">state</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ops</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// (return a not-found error here)
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">next_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">state</span><span class="p">.</span><span class="n">process_message</span><span class="p">(</span><span class="n">message</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">ops</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">id</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span><span class="w"> </span><span class="n">next_state</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// At this point, lock is dropped, and the mutex is available to other
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// threads.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This is a very typical use of mutexes: to guard one or more <em>invariants</em> or <em>properties</em> of some kind. These invariants are upheld while the mutex is unlocked. In this case, the invariant being guarded is that <code>Operations::ops</code> has complete and up-to-date tracking of all in-progress and completed operations.</p>
<p>Of equal importance is the fact that, while the mutex is held, the invariant is <em>temporarily violated</em>. In order to process the message, we have to remove the state from the map, create a new state, then put it back into the map. During this period, <code>Operations::ops</code> is missing this one operation, so it no longer tracks all operations. But this temporary violation is okay, because no other threads see this in-between state. Before the mutex is released, this code is responsible for putting the operation back into the map.</p>
<p>Is it always true that the operation is put back into the map? Unfortunately not always, in the presence of what I think of as <em>unexpected errors</em>. Many practitioners draw a separation between two different kinds of errors that a system can have. The terms <em>recoverable</em> and <em>unrecoverable</em> are sometimes used for them, but I tend to prefer the following terms (see also <a href="https://burntsushi.net/unwrap/#what-about-recoverable-vs-unrecoverable-errors">some discussion by Andrew Gallant</a>):</p>
<ul>
<li>An <em>expected error</em> is one that can occur in normal operation. For example, if a user specifies a directory to write a file to, and that directory is not writable, then that&rsquo;s in the realm of expectations (maybe the user mistyped the directory, for example).</li>
<li>An <em>unexpected error</em> is one that cannot occur in normal operation. Andrew presents the example of a fixed string literal that is processed as a regex. A fixed literal baked into the program really ought to be valid as a regex, so any issues are unexpected.</li>
</ul>
<p>Generally, in Rust, expected errors are handled via the <code>Result</code> type, and unexpected errors are handled by <a href="https://doc.rust-lang.org/book/ch09-01-unrecoverable-errors-with-panic.html">panicking</a>. Now, there isn&rsquo;t a firm requirement that things be this way.</p>
<ul>
<li>For example, some high-availability systems may choose to model unexpected errors via a <code>Result</code>-like type (see <a href="https://docs.rs/woah/">the <code>woah</code> crate</a> as an example).</li>
<li>Quick-and-dirty scripts may choose to handle both expected and unexpected errors as panics.</li>
<li>Panics can also be used for other purposes, e.g. to <a href="https://sunshowers.io/posts/cancelling-async-rust/#cancellations-in-synchronous-rust">cancel in-progress work in synchronous Rust</a>.</li>
</ul>
<p>But in typical production-grade Rust, expected errors are <code>Result</code>s while unexpected errors (and only unexpected errors) are panics. Lock poisoning is built around this assumption.</p>
<h3 id="what-if-a-panic-occurs">What if a panic occurs?</h3>
<p>Consider what happens if a panic occurs in <code>OperationState::process_message</code>. This depends in part on the build flags and surrounding code, so let&rsquo;s look closely at all the possibilities. In Rust, there are two ways to configure build flags on panic:</p>
<ul>
<li>The default is to <em>unwind</em>, or to walk up the stack and run cleanup code. With unwinding, panics can also be <em>caught</em> at a higher level: in the same thread with <a href="https://doc.rust-lang.org/std/panic/fn.catch_unwind.html"><code>catch_unwind</code></a>, or in another thread via <a href="https://doc.rust-lang.org/std/thread/struct.JoinHandle.html#method.join"><code>JoinHandle::join</code></a>.</li>
<li>The alternative is to <em>abort</em>, which causes the whole process to crash without performing any cleanup.</li>
</ul>
<p>Some real-world applications (such as most of what we ship at <a href="https://oxide.computer">Oxide</a>) abort on panic, but most of this post is actually moot for aborts. So in the rest of this post, we&rsquo;re going to focus on the default unwind behavior.</p>
<p>What do programs do on unwind?</p>
<ul>
<li>
<p>If a panic is invoked in the context of a <a href="https://doc.rust-lang.org/std/panic/fn.catch_unwind.html"><code>catch_unwind</code></a>, an <code>Err(E)</code> is returned, where <a href="https://doc.rust-lang.org/std/thread/type.Result.html">the value <code>E</code></a> is whatever message or other payload the panic occurred with.</p>
</li>
<li>
<p>If there&rsquo;s no <code>catch_unwind</code>, and the panic occurs on the main thread, then a message is printed out and the program exits with an error.</p>
<details>
    <summary>Click to expand example</summary>
<p>Consider this simple program:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="p">{</span><span class="n">thread</span><span class="p">,</span><span class="w"> </span><span class="n">time</span>::<span class="n">Duration</span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="fm">panic!</span><span class="p">(</span><span class="s">&#34;This is a panic message&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This program prints out:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">thread &#39;main&#39; (502586) panicked at src/main.rs:2:5:
</span></span></span><span class="line"><span class="cl"><span class="go">This is a panic message
</span></span></span><span class="line"><span class="cl"><span class="go">note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</span></span></span></code></pre></div><p>and the program exits with a non-success exit code.</p>
</details>
</li>
<li>
<p>If there&rsquo;s no <code>catch_unwind</code>, and the panic occurs on a different thread, then a message is printed out, and the panic message is returned as the result of <a href="https://doc.rust-lang.org/std/thread/struct.JoinHandle.html#method.join"><code>JoinHandle::join</code></a>.</p>
  <details>
    <summary>Click to expand example</summary>
<p>If you run this slightly more complex program:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">thread</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">join_handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="fm">panic!</span><span class="p">(</span><span class="s">&#34;This is a panic message&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">join_handle</span><span class="p">.</span><span class="n">join</span><span class="p">().</span><span class="n">expect</span><span class="p">(</span><span class="s">&#34;child thread succeeded&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Then it prints out:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">thread &#39;&lt;unnamed&gt;&#39; (517242) panicked at src/main.rs:5:9:
</span></span></span><span class="line"><span class="cl"><span class="go">This is a panic message
</span></span></span><span class="line"><span class="cl"><span class="go">note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="go">thread &#39;main&#39; (516882) panicked at src/main.rs:7:24:
</span></span></span><span class="line"><span class="cl"><span class="go">child thread succeeded: Any { .. }
</span></span></span></code></pre></div><p>and the program exits with a non-success exit code.</p>
<p>An interesting thing to note here is that there were <em>two</em> panics: one in the spawned child thread with the <code>panic!</code> message, and one in the main thread when the <code>expect</code> was called. The panic responsible for producing the non-success exit code was the one that occurred in the main thread, not the child thread.</p>
  </details>
</li>
<li>
<p>This raises the question: what if a non-main thread panics and the thread is <em>not</em> joined? With this program:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="p">{</span><span class="n">thread</span><span class="p">,</span><span class="w"> </span><span class="n">time</span>::<span class="n">Duration</span><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="fm">panic!</span><span class="p">(</span><span class="s">&#34;This is a panic message&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">thread</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">5</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>What gets printed out is:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">thread &#39;&lt;unnamed&gt;&#39; (543640) panicked at src/main.rs:5:9:
</span></span></span><span class="line"><span class="cl"><span class="go">This is a panic message
</span></span></span><span class="line"><span class="cl"><span class="go">note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</span></span></span></code></pre></div><p>And the program exits with a successful exit code! There&rsquo;s no indication that a panic occurred other than the message printed out, which can easily be missed.</p>
</li>
</ul>
<p>The upshot of all this is that panics in non-main threads are not magic. In order for the system to make decisions based on whether a panic occurred, it must process that, either with <code>catch_unwind</code> or via <code>JoinHandle::join</code>, and it&rsquo;s all too easy to just ignore panics.</p>
<p>Coming back to our <code>Operations</code> example above, what does that mean for our mutex&rsquo;s critical section?</p>
<ul>
<li><em>If</em> the Rust binary is configured to unwind on panic; and</li>
<li><em>if</em> a non-main thread panics in a critical section; and</li>
<li><em>if</em> there&rsquo;s no <code>catch_unwind</code> to catch the panic; and</li>
<li><em>if</em> the child thread is not explicitly joined on, or a join does happen but the error is ignored—</li>
<li><em>then</em>, the mutex invariant is <strong>permanently violated</strong>. The data guarded by the mutex is logically corrupted. The in-progress operation is lost!</li>
</ul>
<p>This might look like a lot of ifs, but they&rsquo;re more common than you might think: they&rsquo;re all either the default in Rust or a very common way to write code.</p>
<p>Rust&rsquo;s designers had the foresight to see this issue, and introduce <em>lock poisoning</em> as a detection mechanism for this failure mode. The way poisoning works is that:</p>
<ul>
<li>At the time a lock is released, there&rsquo;s <a href="https://github.com/rust-lang/rust/blob/1d60f9e070c1039b263e0f035c0f03dfcc610d0f/library/std/src/sync/poison.rs#L128-L132">a check</a> for whether the thread is currently panicking. If it is, the mutex is marked <em>poisoned</em>.</li>
<li>Then, the next time a lock is acquired, rather than a <code>MutexGuard</code>, a <a href="https://doc.rust-lang.org/std/sync/struct.PoisonError.html"><code>PoisonError</code></a> is returned.</li>
</ul>
<p>Almost all code immediately panics on seeing a <code>PoisonError</code> via <code>.lock().unwrap()</code>: this is often called <em>propagating panics</em>. But <code>PoisonError</code> can be handled more explicitly than that. Note that <code>PoisonError</code>, and poisoning more generally, is purely advisory: you can <a href="https://doc.rust-lang.org/std/sync/struct.PoisonError.html#method.into_inner">retrieve the data underneath</a>, and even <a href="https://doc.rust-lang.org/std/sync/struct.Mutex.html#method.clear_poison">clear the poison bit</a> in Rust 1.77 and above.</p>
<p>The fact is, though, that anything other than <code>.lock().unwrap()</code> is rare in practice. This is emphatically <em>not</em> a reason to remove poisoning, and in fact is a strong argument to <em>retain poisoning</em> while making the ergonomics better (<a href="/posts/on-poisoning/#typing-in-lockunwrap-is-annoying">see below</a>). What is important is <em>detection</em>, not <em>recovery</em>.</p>
<p>So, putting it all together: if a child thread panics in a critical section, then it is quite possible that the data is in an inconsistent or logically corrupt state. To indicate this, the mutex is marked poisoned. If the child thread is not waited on by the parent, this might be the <em>only indication</em> that a panic previously occurred in a critical section!</p>
<p>It is precisely this confluence of factors that makes lock poisoning such an important feature.</p>
<h2 id="unexpected-cancellations">Unexpected cancellations</h2>
<p>Is the problem of inconsistent mutex-guarded state limited to panic unwinding? I&rsquo;d argue that it is a property of <em>unexpected cancellations</em> more generally: you start executing a critical section thinking that it will be run to completion, but something causes that process to be interrupted.</p>
<p>In Rust, there are two sources of unexpected cancellations, with strong parallels between them:</p>
<ul>
<li>Panics, as discussed above.</li>
<li>In async Rust, <a href="https://sunshowers.io/posts/cancelling-async-rust">future cancellations</a> at an await point.</li>
</ul>
<p>As documented in Oxide <a href="https://rfd.shared.oxide.computer/rfd/397#_example_with_mutexes">RFD 397</a> and <a href="https://rfd.shared.oxide.computer/rfd/400#no_mutex">RFD 400</a>, unexpected future cancellations have resulted in so many mutex invariant violations that we now avoid Tokio mutexes <em>entirely</em><sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. My perspective here comes from much pain dealing with this issue in async Rust, and wanting very much for this footgun to not make its way to synchronous Rust.</p>
<p>See <a href="/posts/on-poisoning/#appendix-mutexes-and-future-cancellations">the appendix</a> for more details.</p>
<h2 id="do-panics-in-critical-sections-always-cause-invariant-violations">Do panics in critical sections always cause invariant violations?</h2>
<p>In other words, is poisoning often too conservative? My answer to this is that panics do not always cause invariant violations, but they&rsquo;re so common, and the downsides of corrupt state so unbounded, that it is still valuable to have lock poisoning as a strong heuristic.</p>
<p>Firstly, if all you&rsquo;re doing is reading data that just happens to be guarded by a mutex (maybe because some other function writes to that data), a panic in the critical section can&rsquo;t cause invariant violations. (But also, you may wish to use an <a href="https://doc.rust-lang.org/std/sync/struct.RwLock.html"><code>RwLock</code></a>.)</p>
<p>Secondly, some simple kinds of writes can also avoid causing invariant violations. For example if all you&rsquo;re doing is updating some counters<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[derive(Default)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">Counters</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">read_count</span>: <span class="kt">u64</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">write_count</span>: <span class="kt">u64</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">mutex</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Mutex</span>::<span class="n">new</span><span class="p">(</span><span class="n">Counters</span>::<span class="n">default</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// On read:
</span></span></span><span class="line"><span class="cl"><span class="o">*</span><span class="n">mutex</span><span class="p">.</span><span class="n">lock</span><span class="p">().</span><span class="n">unwrap</span><span class="p">().</span><span class="n">read_count</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// On write:
</span></span></span><span class="line"><span class="cl"><span class="o">*</span><span class="n">mutex</span><span class="p">.</span><span class="n">lock</span><span class="p">().</span><span class="n">unwrap</span><span class="p">().</span><span class="n">write_count</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Finally, it is sometimes possible to carefully architect code to be <a href="https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html">unwind safe</a>, such that if a panic occurs, either:</p>
<ul>
<li>internal invariants are not violated; or</li>
<li>the violation can easily be detected (effectively tracking the poison bit internally rather than in the <code>Mutex</code> wrapper).</li>
</ul>
<p>For example, the standard library&rsquo;s <code>HashMap</code> and <code>BTreeMap</code> are architected this way. In our <code>Operations</code> example, we could, rather than removing the operation from the map entirely, replace it with an <code>Invalid</code> sentinel state.</p>
<p>In these cases, it is true that a panic in a critical section is not harmful, and that the typical <code>.lock().unwrap()</code> approach will reduce system availability. But the important thing to keep in mind is that <strong>code changes over time</strong>. One of the things I like about Rust is how resilient it is to changes over time: by encoding properties like mutable access into the type system, Rust makes it that much harder for new team members (or even yourself six months from now) to screw up. However, <a href="https://sunshowers.io/posts/cancelling-async-rust/#systematic-solutions">like async cancel safety</a>, unwind safety is not encoded in Rust&rsquo;s type system<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>, so it&rsquo;s easy for code that&rsquo;s fine today to be wrong tomorrow.</p>
<p>The main downside to a <code>.lock().unwrap()</code> that misfires is reduced availability and denial of service. But the downsides to an undetected panic are unbounded, and can range from denial of service all the way to &ldquo;(part of) an HTTP request ending up <a href="https://github.com/rust-lang/rust/issues/134645#issuecomment-3584117188">sent to a party it should not have been sent to</a>,&rdquo; or in other words personal information leakage.</p>
<p>A downside that (while <a href="https://blog.cloudflare.com/18-november-2025-outage/">potentially serious</a>) is bounded, versus the kind of flaw that can kill an organization—I know which default I want.</p>
<p><strong>What about writing panic-free code?</strong> You can carefully write your critical sections to not have panics. But that is a property that&rsquo;s especially hard to maintain as code changes over time. Even something as simple as a <code>println!</code> can panic. Also, if the critical section can&rsquo;t panic, then it doesn&rsquo;t matter whether the mutex poisons or not.</p>
<h2 id="where-else-can-panics-cause-invariant-violations">Where else can panics cause invariant violations?</h2>
<p>A bit of history here: in Rust 1.0, panics could only be detected at thread boundaries via <a href="https://doc.rust-lang.org/std/thread/struct.JoinHandle.html#method.join"><code>JoinHandle::join</code></a>. This meant that back then, the only way for panics to cause invariant violations was for:</p>
<ul>
<li>shared data to be guarded by a mutex</li>
<li>a thread to panic in the middle of a critical section</li>
</ul>
<p>Since then, two Rust features were added:</p>
<ul>
<li><a href="https://doc.rust-lang.org/std/panic/fn.catch_unwind.html"><code>catch_unwind</code></a> in Rust 1.9.</li>
<li><a href="https://doc.rust-lang.org/std/thread/fn.scope.html">Scoped threads</a> in Rust 1.63.</li>
</ul>
<p>With both of these, you can operate on arbitrary data (i.e. not just data guarded by a mutex) and leave it in an inconsistent state. To see how, let&rsquo;s rewrite the <code>Operations</code> example above to not have a mutex inside of it, and to require exclusive access to make any modifications to it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[derive(Default)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">Operations</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">ops</span>: <span class="nc">HashMap</span><span class="o">&lt;</span><span class="n">OperationId</span><span class="p">,</span><span class="w"> </span><span class="n">OperationState</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Operations</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="sd">/// Process a message, updating the internal operation state appropriately.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="sd">/// 
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="sd">/// Note: this now requires &amp;mut self, not just &amp;self.
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">process</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">id</span>: <span class="kp">&amp;</span><span class="nc">OperationId</span><span class="p">,</span><span class="w"> </span><span class="n">message</span>: <span class="nc">Message</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// Retrieve the element from the map to process it.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">state</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// (return a not-found error here)
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">next_state</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">state</span><span class="p">.</span><span class="n">process_message</span><span class="p">(</span><span class="n">message</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="bp">self</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">id</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span><span class="w"> </span><span class="n">next_state</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Since there are no mutexes involved any more, this is no longer a critical section in the classical sense. But note that we still have the invariant that <code>ops</code> tracks all operations. This invariant <em>is</em> temporarily violated, with the idea that it&rsquo;ll be restored before the function returns. Since <code>&amp;mut</code> means nothing else has any kind of access (read or write) to this data, we know that this in-between state is not seen by anybody else.</p>
<p>But just like with mutexes, this breaks down with unwinding. With <code>catch_unwind</code>, you can do:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">panic</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">operations</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Operations</span>::<span class="n">default</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">panic</span>::<span class="n">catch_unwind</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">operations</span><span class="p">.</span><span class="n">process</span><span class="p">(</span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">message</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">});</span><span class="w">
</span></span></span></code></pre></div><p>And with scoped threads, you can do:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">thread</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">operations</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Operations</span>::<span class="n">default</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">thread</span>::<span class="n">scope</span><span class="p">(</span><span class="o">|</span><span class="n">s</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">join_handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">s</span><span class="p">.</span><span class="n">spawn</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">operations</span><span class="p">.</span><span class="n">process</span><span class="p">(</span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">message</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">});</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">});</span><span class="w">
</span></span></span></code></pre></div><p>If a panic occurs in <code>process_message</code>, <code>Operations</code> is logically corrupted. This failure mode has resulted in a proposal to have a <code>Poison&lt;T&gt;</code> wrapper that poisons on panicking. That <em>absolutely</em> makes sense and is worth pursuing.</p>
<h2 id="separating-mutexes-from-poisoning">Separating mutexes from poisoning?</h2>
<p>But, along with the <code>Poison&lt;T&gt;</code> wrapper, there are <a href="https://github.com/rust-lang/rust/issues/134645#issuecomment-3590942770">some suggestions</a> to go further and suggest that the current <code>std::sync::Mutex</code> type should be changed in the next Rust edition to, instead of poisoning on panic, <strong>silently unlock</strong>. (And also, as a followup, that the current <code>Mutex&lt;T&gt;</code> should instead become <code>Mutex&lt;Poison&lt;T&gt;&gt;</code>.)</p>
<p>(It&rsquo;s worth noting that there&rsquo;s another non-poisoning option: the mutex stays locked forever, as C programmers might expect. This option is somewhat appealing because it is safe by default, in a sense. But once a thread is stuck waiting on the mutex, there&rsquo;s <a href="https://sunshowers.io/posts/nextest-process-per-test/#appendix">no easy way to recover</a>. So it&rsquo;s not panics that propagate, it&rsquo;s stuck threads. This seems strictly worse than a poisoning mutex to me, so I&rsquo;ll assume the proposal means silent unlocking.)</p>
<p>I first want to give credit to this proposal: it is quite beautiful.</p>
<ul>
<li>It&rsquo;s more composable. The <code>Poison</code> wrapper can be used with arbitrary mutexes, so you can use it with mutexes such as <a href="https://docs.rs/parking_lot/"><code>parking_lot</code></a> that silently unlock on panic today. Single-threaded mutex equivalents like <a href="https://doc.rust-lang.org/std/cell/struct.RefCell.html"><code>RefCell</code></a> can also benefit from poisoning.</li>
<li>With Rust&rsquo;s philosophy of zero-cost abstractions, only users who need poisoning pay for it.</li>
<li>As observed above, not all mutexes need poisoning, and poisoning is useful without mutexes, so the two are seemingly independent of each other.</li>
</ul>
<p>While all of these are true, I keep coming back to how unbounded the downside of an undetected panic is, and how easy it is to get wedged in this state. Mutexes and poisoning have value separate from each other, but I don&rsquo;t think they are as independent as they seem at first. My understanding from writing Rust code over many years is that almost all uses of mutexes benefit from poisoning, and almost all instances of poisoning one needs to care about are with mutex-guarded data.</p>
<p><strong>Why are mutexes special?</strong> As we&rsquo;ve established above, poisoning can make sense in non-mutex contexts too. What makes mutexes a particularly valuable place to have it? The reason has to do with what I like to call the <a href="https://rfd.shared.oxide.computer/rfd/0400#_cancel_correctness"><em>cancellation blast radius</em></a>. Corrupted data only matters if it is shared mutable state: changed in a way that resulted in an invariant violation, and visible outside of where the failure occurred. If a typical <code>&amp;mut</code> reference panics in the middle of an operation, it&rsquo;s likely that the data it is referring to will also be torn down, in which case those invariant violations don&rsquo;t matter<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>. But the whole <em>point</em> of mutexes is that they&rsquo;re shared across threads! So data corruption is almost certain to be externally visible.</p>
<p>There are some use cases that would benefit from non-poisoning mutexes, like metrics and best-effort logging, but those cases shouldn&rsquo;t drive the <em>default</em>.</p>
<p>More specifically, I am worried that a common complaint about lock poisoning (<a href="/posts/on-poisoning/#typing-in-lockunwrap-is-annoying">see below</a>) is that it has too much friction. Having to use <code>Mutex&lt;Poison&lt;T&gt;&gt;</code> instead of <code>Mutex&lt;T&gt;</code> <em>adds even more friction</em>, so people are going to opt for non-poisoning mutexes more of the time. This is going to lead to grave mistakes in production.</p>
<p>This is a spot where zero-cost abstractions and safety by default are seemingly at odds with each other. I would like to see performance numbers to quantify this better, but if I may hazard a guess, the incremental cost of checking the poison flag (a single <a href="https://github.com/rust-lang/rust/blob/1d60f9e070c1039b263e0f035c0f03dfcc610d0f/library/std/src/sync/poison.rs#L141">atomic load</a> with relaxed ordering) is minimal compared to the cost of acquiring the lock in the first place.</p>
<h2 id="what-about-parking_lot-mutexes">What about <code>parking_lot</code> mutexes?</h2>
<p>I mentioned earlier that <code>parking_lot</code>&rsquo;s mutexes silently unlock on panic. A large chunk of the Rust ecosystem uses <code>parking_lot</code> today, often for performance reasons. Does that mean that code using <code>parking_lot</code> has these unbounded downsides?</p>
<p>The answer depends on a bunch of things, but in general (and especially in library code) that is indeed what I&rsquo;m suggesting. For instance, <a href="https://github.com/paritytech/parity-db/blob/a9c0e8517868ed513a50754940cd0ebc40a4a34d/src/db.rs#L679-L732">this critical section in <code>parity-db</code></a> is quite large. Reasoning about whether it&rsquo;s unwind-safe seems very difficult to me; this is exactly the kind of code that mutex poisoning does well to guard against.</p>
<p>In this case, the binary <a href="https://github.com/paritytech/parity-db/blob/a9c0e8517868ed513a50754940cd0ebc40a4a34d/Cargo.toml#L42-L43">is configured</a> to abort on panic, so it&rsquo;s fine. But reusable Rust libraries <em>cannot</em> require <code>panic = 'abort'</code>, and if this code were in a library on <code>crates.io</code>, it would be a real cause for concern.</p>
<h2 id="just-ship-with-panic--abort">Just ship with <code>panic = 'abort'</code>?</h2>
<p>A common response to this class of issue is to not bother with any of this unwinding stuff, and always abort on panic. This comes back to the idea of the cancellation blast radius described above: aborting the process on panic guarantees that in-memory state is torn down.</p>
<p>I have a lot of sympathy for this idea! This is what we do at Oxide. (Why am I writing this post if it doesn&rsquo;t affect my workplace? Well, first, I care about the health of Rust more generally. Second, libraries must work with unwinds. But most importantly, we <em>have</em> seen the pain of unexpected async cancellations at Oxide, so we <em>know</em> how bad it can be.)</p>
<p>But also, that works <em>fine</em> with the current approach: <code>.lock().unwrap()</code> always succeeds. Whether mutexes poison or not only matters with <code>panic = 'unwind'</code>.</p>
<p>This leads to what I think is driving a lot of discussion here:</p>
<h2 id="typing-in-lockunwrap-is-annoying">Typing in <code>.lock().unwrap()</code> is annoying</h2>
<p>I get this complaint. I really do. Having to write <code>.lock().unwrap()</code> everywhere sucks. It&rsquo;s extra characters in a language already filled with syntax noise. It can cause <code>rustfmt</code> to format your line of code across multiple lines.</p>
<p>These are all valid points. But there is a much better solution for them, one that doesn&rsquo;t give up the very important benefits of poisoning: in the next Rust edition, make <code>lock()</code> automatically panic if the mutex is poisoned! (And add a <code>lock_or_poison</code> method for the current behavior<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>).</p>
<p>It&rsquo;s worth comparing the different options here:</p>
<table>
	<thead>
			<tr>
					<th>Aspect</th>
					<th><code>lock().unwrap()</code></th>
					<th>Auto-panic</th>
					<th>Removing poison</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><strong>Syntax noise</strong></td>
					<td>Medium: <code>.unwrap()</code> everywhere</td>
					<td>Low: just <code>lock()</code></td>
					<td>Low by default, high with <code>Poison&lt;T&gt;</code></td>
			</tr>
			<tr>
					<td><strong>Safety by default</strong></td>
					<td>✅ Panics propagate</td>
					<td>✅ Panics propagate</td>
					<td>❌ Silent corruption possible</td>
			</tr>
			<tr>
					<td><strong>Opt-out available</strong></td>
					<td>✅ <code>lock().unwrap_or_else()</code></td>
					<td>✅ <code>lock_or_poison()</code></td>
					<td>❌ Must opt <em>in</em> via <code>Poison&lt;T&gt;</code></td>
			</tr>
			<tr>
					<td><strong>Works with <code>panic = 'abort'</code></strong></td>
					<td>✅</td>
					<td>✅</td>
					<td>✅</td>
			</tr>
			<tr>
					<td><strong>Ergonomics</strong></td>
					<td>Poor</td>
					<td>Good</td>
					<td>Good without poison, poor with</td>
			</tr>
			<tr>
					<td><strong>Backwards compatibility</strong></td>
					<td>Current behavior</td>
					<td>Requires new edition</td>
					<td>Requires new edition</td>
			</tr>
	</tbody>
</table>
<p>Based on this table, I believe the answer is clear: if a breaking change is going to happen, it&rsquo;s much better to make <code>lock</code> automatically panic than to make panics silently unlock.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Concurrent programming is very difficult. Rust makes it easier than most other languages, and lock poisoning is an important part of the story. Let&rsquo;s avoid introducing any regressions here.</p>
<p>Providing a <code>Poison&lt;T&gt;</code> wrapper makes a lot of sense. Making the default <code>std::sync::Mutex</code> silently unlock on panic would, however, be a mistake.</p>
<p>Should Rust&rsquo;s standard library even provide non-poisoning mutexes? That&rsquo;s a harder question. I&rsquo;m worried that their mere presence in the standard library will lower the barrier to people doing the wrong thing, particularly in libraries where <code>panic = 'abort'</code> cannot be assumed. But I think non-poisoning mutexes have some legitimate uses, so I don&rsquo;t object too strenuously if the tradeoffs are carefully documented.</p>
<p>Writing all this out was very helpful to me in getting my thoughts straight, and I hope it&rsquo;s helpful to you too.</p>
<p><em>Cover photo by <a href="https://wandering.shop/@aldeka/109301243634789356">Karen Rustad Tolva</a>, used with permission. Thanks to <a href="https://infosec.exchange/@munin">Fiona</a> and several of my colleagues at Oxide for reviewing drafts of this post. Any errors in it are my own.</em></p>
<p><em>Discuss on <a href="https://news.ycombinator.com/item?id=46125903">Hacker News</a>, <a href="https://lobste.rs/s/vkyrtf/defense_lock_poisoning_rust">Lobsters</a>, and <a href="https://www.reddit.com/r/rust/comments/1pck69k/in_defense_of_lock_poisoning_in_rust_by_sunshowers/">/r/rust</a>.</em></p>
<p><em>Updated 2025-12-06: Added a section to</em> <a href="/posts/on-poisoning/#separating-mutexes-from-poisoning">Separating mutexes from poisoning</a> <em>on what makes poisoning particularly valuable for mutexes, more than in other contexts.</em></p>
<hr>
<h2 id="appendix-mutexes-and-future-cancellations">Appendix: Mutexes and future cancellations</h2>
<p>Unlike panics, the standard library&rsquo;s mutex does not poison on future cancellations. (I believe it&rsquo;s not possible to poison on future cancellations with the RAII pattern.)</p>
<ul>
<li>There is a saving grace here: most people use the <a href="https://tokio.rs/">Tokio executor</a>, which typically requires <a href="https://docs.rs/tokio/latest/tokio/task/fn.spawn.html">spawned tasks</a> to be <a href="https://doc.rust-lang.org/nightly/core/marker/trait.Send.html"><code>Send</code></a>: transferable between OS threads while suspended at an await point.</li>
<li>The standard library&rsquo;s <code>MutexGuard</code> is not <a href="https://doc.rust-lang.org/std/sync/struct.MutexGuard.html#impl-Send-for-MutexGuard%3C%27_,&#43;T%3E">transferable between OS threads</a>, which prevents await points in a critical section.</li>
</ul>
<p>But wait! Tokio provides its own <a href="https://docs.rs/tokio/1/tokio/sync/struct.Mutex.html"><code>Mutex</code> type</a>, which <em>is</em> sendable to another thread. This means that it <em>is</em> possible to put an await point in a critical section, and so the issue of unexpected cancellations within a critical section rears its ugly head with Tokio mutexes.</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>Tokio mutexes also do not poison if the critical section panics. At Oxide we abort on panic so this matters less as a reason to avoid them, but is yet another asterisk that pushes me away from them.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>A more typical way to manage simple counters is with <a href="https://doc.rust-lang.org/std/sync/atomic/struct.AtomicU64.html">an atomic</a>, but if the data is more complex than just an integer, a mutex may be necessary.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>Rust does have some <a href="https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html">rudimentary support for unwind safety</a> in the type system, but it is more of a suggestion that most users <a href="https://doc.rust-lang.org/std/panic/struct.AssertUnwindSafe.html">ignore</a>, and because of that there&rsquo;s <a href="https://github.com/rust-lang/rfcs/pull/3260">a proposal</a> to remove it from Rust.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>The exception is <code>Drop</code> implementations, which may have to be written robustly to guard against temporary invariant violations.&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5">
<p>The more obvious <code>try_lock</code> is <a href="https://doc.rust-lang.org/std/sync/struct.Mutex.html#method.try_lock">already taken</a>.&#160;<a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/></entry><entry><title type="html">`SocketAddrV6` is not roundtrip serializable</title><link href="https://sunshowers.io/posts/socketaddrv6-not-roundtrip/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/cancelling-async-rust/?utm_source=atom_feed" rel="related" type="text/html" title="Cancelling async Rust"/><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="related" type="text/html" title="Why nextest is process-per-test"/><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="related" type="text/html" title="Open and closed universes"/><id>https://sunshowers.io/posts/socketaddrv6-not-roundtrip/</id><published>2025-11-04T00:00:00+00:00</published><updated>2025-11-05T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>An investigation into some lesser-known IPv6 features.</blockquote><p>A few weeks ago at <a href="https://oxide.computer/">Oxide</a>, we encountered a bug where a particular, somewhat large, data structure was erroring on serialization to JSON via <a href="https://serde.rs/"><code>serde</code></a>. The problem was that JSON only supports map keys that are strings<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>, and the data structure had an infrequently-populated map with keys that were more complex than that<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p>
<p>We fixed the bug, but a concern still remained: what if some other map that was empty most of the time had a complex key in it? The easiest way to guard against this is by generating random instances of the data structure and attempting to serialize them, checking that this operation doesn&rsquo;t panic. The most straightforward way to do this is with <em>property-based testing</em>, where you define:</p>
<ul>
<li>a way to <em>generate</em> random instances of a particular type, and</li>
<li>given a failing input, a way to <em>shrink</em> it down to a minimal failing value.</li>
</ul>
<p>Modern property-based testing frameworks like <a href="https://proptest-rs.github.io/proptest/intro.html"><code>proptest</code></a>, which we use at Oxide, combine these two algorithms into a single <em>strategy</em>, through a technique known as <em>integrated shrinking</em>. (For a more detailed overview, see my <a href="https://sunshowers.io/posts/monads-through-pbt/">monad tutorial</a>, where I talk about the undesirable performance characteristics of monadic composition when it comes to integrated shrinking.)</p>
<p>The <code>proptest</code> library has a notion of a canonical strategy for a type, expressed via <a href="https://docs.rs/proptest/latest/proptest/arbitrary/trait.Arbitrary.html">the <code>Arbitrary</code> trait</a>. The easiest way to define <code>Arbitrary</code> instances for large, complex types is to use a <a href="https://docs.rs/test-strategy/">derive macro</a>. Annotate your type with the macro:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">test_strategy</span>::<span class="n">Arbitrary</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="cp">#[derive(Arbitrary)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">MyType</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">id</span>: <span class="nb">String</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">data</span>: <span class="nc">BTreeMap</span><span class="o">&lt;</span><span class="nb">String</span><span class="p">,</span><span class="w"> </span><span class="n">MyInnerType</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="cp">#[derive(Arbitrary)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">MyInnerType</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">value</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>As long as all the fields have <code>Arbitrary</code> defined for them—and the <code>proptest</code> library defines the trait for most types in the standard library—your type has a working random generator and shrinker associated with it. It&rsquo;s pretty neat!</p>
<p>I put together an <code>Arbitrary</code> implementation for our very complex type, then wrote a property-based test to ensure that it serializes properly:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">test_strategy</span>::<span class="n">proptest</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// The outermost struct is called PlanningReport.
</span></span></span><span class="line"><span class="cl"><span class="cp">#[proptest]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">planning_report_json_serialize</span><span class="p">(</span><span class="n">planning_report</span>: <span class="nc">PlanningReport</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">serde_json</span>::<span class="n">to_string</span><span class="p">(</span><span class="o">&amp;</span><span class="n">planning_report</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And, running it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> cargo nextest run -p nexus-types planning_report
</span></span><span class="line"><span class="cl"><span class="go">        PASS [   4.879s] nexus-types deployment::planning_report::tests::planning_report_json_serialize
</span></span></span><span class="line"><span class="cl"><span class="go">────────────
</span></span></span><span class="line"><span class="cl"><span class="go">     Summary [   4.880s] 1 test run: 1 passed, 24 skipped
</span></span></span></code></pre></div><p>The test passed!</p>
<p>But while we&rsquo;re here, surely we should also be able to <em>deserialize</em> a <code>PlanningReport</code>, and then ensure that we get the same value back, right? We&rsquo;ve already done the hard part, so let&rsquo;s go ahead and add this test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[proptest]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">planning_report_json_roundtrip</span><span class="p">(</span><span class="n">planning_report</span>: <span class="nc">PlanningReport</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">json</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">serde_json</span>::<span class="n">to_string</span><span class="p">(</span><span class="o">&amp;</span><span class="n">planning_report</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">deserialized</span>: <span class="nc">PlanningReport</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">serde_json</span>::<span class="n">from_str</span><span class="p">(</span><span class="o">&amp;</span><span class="n">json</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="fm">prop_assert_eq!</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">planning_report</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">deserialized</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="s">&#34;input and output are equal&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And…</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> cargo nextest run -p nexus-types planning_report
</span></span><span class="line"><span class="cl"><span class="go">        FAIL [   3.688s] nexus-types deployment::planning_report::tests::planning_report_json_roundtrip
</span></span></span><span class="line"><span class="cl"><span class="go">  stderr ───
</span></span></span><span class="line"><span class="cl"><span class="go">  minimal failing input: input = _PlanningReportJsonRoundtripArgs {
</span></span></span><span class="line"><span class="cl"><span class="go">      [... many fields omitted for brevity]
</span></span></span><span class="line"><span class="cl"><span class="go">      mgs_updates: PlanningMgsUpdatesStepReport {
</span></span></span><span class="line"><span class="cl"><span class="go">          blocked_mgs_updates: [],
</span></span></span><span class="line"><span class="cl"><span class="go">          pending_mgs_updates: PendingMgsUpdates {
</span></span></span><span class="line"><span class="cl"><span class="go">              by_baseboard: {
</span></span></span><span class="line"><span class="cl"><span class="go">                  BaseboardId {
</span></span></span><span class="line"><span class="cl"><span class="go">                      part_number: &#34;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="go">                      serial_number: &#34;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="go">                  }: PendingMgsUpdate {
</span></span></span><span class="line"><span class="cl"><span class="go">                      sp_type: Sled,
</span></span></span><span class="line"><span class="cl"><span class="go">                      slot_id: 0,
</span></span></span><span class="line"><span class="cl"><span class="go">                      details: HostPhase1(
</span></span></span><span class="line"><span class="cl"><span class="go">                          PendingMgsUpdateHostPhase1Details {
</span></span></span><span class="line"><span class="cl"><span class="go">                              expected_active_phase_1_slot: A,
</span></span></span><span class="line"><span class="cl"><span class="go">                              expected_boot_disk: A,
</span></span></span><span class="line"><span class="cl"><span class="go">                              sled_agent_address: [::ffff:0.0.0.0]:0,
</span></span></span><span class="line"><span class="cl"><span class="go">                          },
</span></span></span><span class="line"><span class="cl"><span class="go">                      ),
</span></span></span><span class="line"><span class="cl"><span class="go">                  },
</span></span></span><span class="line"><span class="cl"><span class="go">              },
</span></span></span><span class="line"><span class="cl"><span class="go">          },
</span></span></span><span class="line"><span class="cl"><span class="go">      },
</span></span></span><span class="line"><span class="cl"><span class="go">      [...]
</span></span></span><span class="line"><span class="cl"><span class="go">  }
</span></span></span></code></pre></div><p>The roundtrip test failed!</p>
<p>Why in the world did the test fail? My first idea was to try and do a textual diff of the <code>Debug</code> outputs of the two data structures. In this case, I tried out the <a href="https://docs.rs/pretty_assertions"><code>pretty_assertions</code></a> library, with something like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">planning_report_json_roundtrip</span><span class="p">(</span><span class="n">planning_report</span>: <span class="nc">PlanningReport</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">pretty_assertions</span>::<span class="fm">assert_eq!</span><span class="p">(</span><span class="n">planning_report</span><span class="p">,</span><span class="w"> </span><span class="n">deserialized</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And the output I got was:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> cargo nextest run -p nexus-types planning_report
</span></span><span class="line"><span class="cl"><span class="go">[...]
</span></span></span><span class="line"><span class="cl"><span class="go">Test failed: assertion failed: `(left == right)`
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="go">Diff &lt; left / right &gt; :
</span></span></span><span class="line"><span class="cl"><span class="go"> PlanningReport {
</span></span></span><span class="line"><span class="cl"><span class="go">     [...]
</span></span></span><span class="line"><span class="cl"><span class="go">     mgs_updates: PlanningMgsUpdatesStepReport {
</span></span></span><span class="line"><span class="cl"><span class="go">         blocked_mgs_updates: [],
</span></span></span><span class="line"><span class="cl"><span class="go">         pending_mgs_updates: PendingMgsUpdates {
</span></span></span><span class="line"><span class="cl"><span class="go">             by_baseboard: {
</span></span></span><span class="line"><span class="cl"><span class="go">                 BaseboardId {
</span></span></span><span class="line"><span class="cl"><span class="go">                     part_number: &#34;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="go">                     serial_number: &#34;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="go">                 }: PendingMgsUpdate {
</span></span></span><span class="line"><span class="cl"><span class="go">                     baseboard_id: BaseboardId {
</span></span></span><span class="line"><span class="cl"><span class="go">                         part_number: &#34;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="go">                         serial_number: &#34;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="go">                     },
</span></span></span><span class="line"><span class="cl"><span class="go">                     sp_type: Sled,
</span></span></span><span class="line"><span class="cl"><span class="go">                     slot_id: 0,
</span></span></span><span class="line"><span class="cl"><span class="go">                     details: HostPhase1(
</span></span></span><span class="line"><span class="cl"><span class="go">                         PendingMgsUpdateHostPhase1Details {
</span></span></span><span class="line"><span class="cl"><span class="go">                             expected_active_phase_1_slot: A,
</span></span></span><span class="line"><span class="cl"><span class="go">                             expected_boot_disk: A,
</span></span></span><span class="line"><span class="cl"><span class="go">                             sled_agent_address: [::ffff:0.0.0.0]:0,
</span></span></span><span class="line"><span class="cl"><span class="go">                         },
</span></span></span><span class="line"><span class="cl"><span class="go">                     ),
</span></span></span><span class="line"><span class="cl"><span class="go">                 },
</span></span></span><span class="line"><span class="cl"><span class="go">             },
</span></span></span><span class="line"><span class="cl"><span class="go">         },
</span></span></span><span class="line"><span class="cl"><span class="go">     },
</span></span></span><span class="line"><span class="cl"><span class="go">     [...]
</span></span></span><span class="line"><span class="cl"><span class="go"> }
</span></span></span></code></pre></div><p>There&rsquo;s nothing in the output! No <code>&lt;</code> or <code>&gt;</code> as would typically be printed. It&rsquo;s as if there wasn&rsquo;t a difference at all, and yet the assertion failing indicated the before and after values just weren&rsquo;t the same.</p>
<h2 id="what-is-going-on">What is going on?</h2>
<p>We have one clue to go by: the integrated shrinking algorithm in <code>proptest</code> tries to shrink maps down to empty ones. But it looks like the <code>pending_mgs_updates</code> map is <em>non-empty</em>. This means that something in either the <code>BaseboardId</code> key or the <code>PendingMgsUpdate</code> value was suspicious.</p>
<p>A <code>PendingMgsUpdate</code> is defined as:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// These types had many more fields -- most have been omitted for brevity.
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">PendingMgsUpdate</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">baseboard_id</span>: <span class="nc">Arc</span><span class="o">&lt;</span><span class="n">BaseboardId</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">sp_type</span>: <span class="nc">SpType</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">slot_id</span>: <span class="kt">u16</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">details</span>: <span class="nc">PendingMgsUpdateDetails</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">enum</span> <span class="nc">PendingMgsUpdateDetails</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">HostPhase1</span><span class="p">(</span><span class="n">PendingMgsUpdateHostPhase1Details</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">PendingMgsUpdateHostPhase1Details</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">expected_active_phase_1_slot</span>: <span class="nc">M2Slot</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">expected_boot_disk</span>: <span class="nc">M2Slot</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">sled_agent_address</span>: <span class="nc">SocketAddrV6</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Most of these types were pretty simple. The only one that looked even remotely suspicious was the <a href="https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html"><code>SocketAddrV6</code></a>, which ostensibly represents an <a href="https://doc.rust-lang.org/std/net/struct.Ipv6Addr.html">IPv6 address</a> plus a port number.</p>
<p>What&rsquo;s going on with the <code>SocketAddrV6</code>? Does the <code>Arbitrary</code> implementation for it do something weird? Well, let&rsquo;s <a href="https://docs.rs/proptest/1.9.0/src/proptest/arbitrary/_std/net.rs.html#51-54">look at it</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="fm">arbitrary!</span><span class="p">(</span><span class="n">SocketAddrV6</span><span class="p">,</span><span class="w"> </span><span class="n">SMapped</span><span class="o">&lt;</span><span class="p">(</span><span class="n">Ipv6Addr</span><span class="p">,</span><span class="w"> </span><span class="kt">u16</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">),</span><span class="w"> </span><span class="bp">Self</span><span class="o">&gt;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">static_map</span><span class="p">(</span><span class="n">any</span>::<span class="o">&lt;</span><span class="p">(</span><span class="n">Ipv6Addr</span><span class="p">,</span><span class="w"> </span><span class="kt">u16</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="o">&gt;</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">|</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">,</span><span class="w"> </span><span class="n">d</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="bp">Self</span>::<span class="n">new</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">,</span><span class="w"> </span><span class="n">d</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>Like a lot of abstracted-out library code it looks a bit strange, but at its core it seems to be simple enough:</p>
<ul>
<li>generate four values: an <code>Ipv6Addr</code>, a <code>u16</code>, a <code>u32</code>, and another <code>u32</code></li>
<li>then pass them in to <code>SocketAddrV6::new</code>.</li>
</ul>
<p>The <code>Ipv6Addr</code> is self-explanatory, and the <code>u16</code> is probably the port number. But what are these last two values? Let&rsquo;s look at <a href="https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.new">the <code>SocketAddrV6::new</code> constructor</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">ip</span>: <span class="nc">Ipv6Addr</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">port</span>: <span class="kt">u16</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">flowinfo</span>: <span class="kt">u32</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">scope_id</span>: <span class="kt">u32</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">SocketAddrV6</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">SocketAddrV6</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">ip</span><span class="p">,</span><span class="w"> </span><span class="n">port</span><span class="p">,</span><span class="w"> </span><span class="n">flowinfo</span><span class="p">,</span><span class="w"> </span><span class="n">scope_id</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>What in the world are these two <code>flowinfo</code> and <code>scope_id</code> values? They look <em>mighty</em> suspicious.</p>
<p>A thing that caught my eye was the <a href="https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#textual-representation">&ldquo;Textual representation&rdquo; section</a> of the <code>SocketAddrV6</code>, which defined the representation as:</p>
<ul>
<li>A left square bracket (<code>[</code>)</li>
<li>The textual representation of an IPv6 address</li>
<li><em>Optionally</em>, a percent sign (<code>%</code>) followed by the scope identifier encoded as a decimal integer</li>
<li>A right square bracket (<code>]</code>)</li>
<li>A colon (<code>:</code>)</li>
<li>The port, encoded as a decimal integer.</li>
</ul>
<p>Note what&rsquo;s missing from this representation: the <code>flowinfo</code> field!</p>
<p>We finally have a theory for what&rsquo;s going on:</p>
<ul>
<li><code>proptest</code> generated a <code>SocketAddrV6</code> with a non-zero <code>flowinfo</code> field.</li>
<li>When we went to serialize this field as JSON, we used the textual representation, which dropped the <code>flowinfo</code> field.</li>
<li>When we deserialized it, the <code>flowinfo</code> field was set to zero.</li>
<li>As a result, the before and after values were no longer equal.</li>
</ul>
<p>Why did this not show up in the textual diff of the <code>Debug</code> values? For most types in Rust, the <code>Debug</code> representation breaks out all the fields and their values. But for <code>SocketAddrV6</code>, the <code>Debug</code> implementation (quite reasonably) <a href="https://doc.rust-lang.org/1.91.0/src/core/net/socket_addr.rs.html#698-702">forwards to the <code>Display</code> implementation</a>. So the <code>flowinfo</code> field is completely hidden, and the only way to look at it is through the <a href="https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.flowinfo"><code>flowinfo</code> method</a>. Whoops.</p>
<p>How can we test this theory? The easiest way is to generate random values of <code>SocketAddrV6</code> where <code>flowinfo</code> is always set to zero, and see if that passes our roundtrip tests. The <code>proptest</code> ecosystem has pretty good support for generating and using this kind of non-canonical strategy. Let&rsquo;s try it out:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// This defines a strategy where flowinfo is always 0.
</span></span></span><span class="line"><span class="cl"><span class="k">fn</span> <span class="nf">socket_addr_v6_without_flowinfo</span><span class="p">()</span><span class="w"> </span>-&gt; <span class="nc">impl</span><span class="w"> </span><span class="n">Strategy</span><span class="o">&lt;</span><span class="n">Value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SocketAddrV6</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">any</span>::<span class="o">&lt;</span><span class="p">(</span><span class="n">Ipv6Addr</span><span class="p">,</span><span class="w"> </span><span class="kt">u16</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="o">&gt;</span><span class="p">().</span><span class="n">prop_map</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="o">|</span><span class="p">(</span><span class="n">addr</span><span class="p">,</span><span class="w"> </span><span class="n">port</span><span class="p">,</span><span class="w"> </span><span class="n">scope_id</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="n">SocketAddrV6</span>::<span class="n">new</span><span class="p">(</span><span class="n">addr</span><span class="p">,</span><span class="w"> </span><span class="n">port</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">scope_id</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Then, we can use this function like so.
</span></span></span><span class="line"><span class="cl"><span class="cp">#[derive(Arbitrary)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">PendingMgsUpdateHostPhase1Details</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">expected_active_phase_1_slot</span>: <span class="nc">M2Slot</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">expected_boot_disk</span>: <span class="nc">M2Slot</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="cp">#[strategy(socket_addr_v6_without_flowinfo())]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">sled_agent_address</span>: <span class="nc">SocketAddrV6</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Pretty straightforward, and similar to how <code>serde</code> lets you provide custom implementations <a href="https://serde.rs/field-attrs.html#with">through <code>#[serde(with = ...)]</code></a>.</p>
<p>Let&rsquo;s test it out again:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="o">%</span><span class="w"> </span><span class="n">cargo</span><span class="w"> </span><span class="n">nextest</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="n">nexus</span><span class="o">-</span><span class="n">types</span><span class="w"> </span><span class="n">planning_report</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="no">PASS</span><span class="w"> </span><span class="p">[</span><span class="w">   </span><span class="mf">4.828</span><span class="n">s</span><span class="p">]</span><span class="w"> </span><span class="n">nexus</span><span class="o">-</span><span class="n">types</span><span class="w"> </span><span class="n">deployment</span>::<span class="n">planning_report</span>::<span class="n">tests</span>::<span class="n">planning_report_json_roundtrip</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="err">────────────</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">     </span><span class="n">Summary</span><span class="w"> </span><span class="p">[</span><span class="w">   </span><span class="mf">4.829</span><span class="n">s</span><span class="p">]</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">test</span><span class="w"> </span><span class="n">run</span>: <span class="mi">1</span><span class="w"> </span><span class="n">passed</span><span class="p">,</span><span class="w"> </span><span class="mi">24</span><span class="w"> </span><span class="n">skipped</span><span class="w">
</span></span></span></code></pre></div><p>All right, looks like our theory is confirmed! We can now merrily be on our way… right?</p>
<p>This little adventure left us with more questions than answers, though:</p>
<ul>
<li>What does this <code>flowinfo</code> field mean?</li>
<li>A <code>SocketAddrV4</code> is just <a href="https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV4.html#method.new">an <code>Ipv4Addr</code> plus a port</a>; why is a <code>SocketAddrV6</code> different?</li>
<li>Why is the <code>flowinfo</code> not part of the textual representation? <code>Ipv4Addr</code>, <code>Ipv6Addr</code>, and <code>SocketAddrV4</code> are all roundtrip serializable. Why is <code>SocketAddrV6</code> not?</li>
<li>Also: what is the <code>scope_id</code> field?</li>
</ul>
<h2 id="but-what-is-flowinfo-anyway">But what is <code>flowinfo</code>, anyway?</h2>
<p>The best place to start looking is in the IETF Request for Comments (RFCs)<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> that specify IPv6. The <a href="https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.flowinfo">Rust documentation for <code>flowinfo</code></a> helpfully links to <a href="https://datatracker.ietf.org/doc/html/rfc2460#section-6">RFC 2460, section 6</a> and <a href="https://datatracker.ietf.org/doc/html/rfc2460#section-7">section 7</a>.</p>
<p>The <code>flowinfo</code> field is actually a <em>combination</em> of two fields that are part of every IPv6 packet:</p>
<ul>
<li>a 20-bit Flow Label, and</li>
<li>an 8-bit Traffic Class<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>.</li>
</ul>
<p>Section 6 of the RFC says:</p>
<blockquote>
<p>Flow Labels</p>
<p>The 20-bit Flow Label field in the IPv6 header may be used by a
source to label sequences of packets for which it requests special
handling by the IPv6 routers, such as non-default quality of service
or &ldquo;real-time&rdquo; service.  This aspect of IPv6 is, at the time of
writing, still experimental and subject to change as the requirements
for flow support in the Internet become clearer. […]</p>
</blockquote>
<p>And section 7:</p>
<blockquote>
<p>Traffic Classes</p>
<p>The 8-bit Traffic Class field in the IPv6 header is available for use
by originating nodes and/or forwarding routers to identify and
distinguish between different classes or priorities of IPv6 packets.
At the point in time at which this specification is being written,
there are a number of experiments underway in the use of the IPv4
Type of Service and/or Precedence bits to provide various forms of
&ldquo;differentiated service&rdquo; for IP packets […].</p>
</blockquote>
<h3 id="traffic-classes">Traffic Classes</h3>
<p>Let&rsquo;s look at the Traffic Class field first. This field is similar to <a href="https://en.wikipedia.org/wiki/Differentiated_services">IPv4&rsquo;s differentiated services code point (DSCP)</a>, and is meant to provide quality of service (QoS) over the network. (For example, prioritizing low-latency gaming and video conferencing packets over bulk downloads.)</p>
<p>The DSCP field in IPv4 is not part of a <code>SocketAddrV4</code>, but the Traffic Class—through the <code>flowinfo</code> field—is part of a <code>SocketAddrV6</code>. Why is that the case? Rust&rsquo;s definition of <code>SocketAddrV6</code> mirrors the <code>sockaddr_in6</code> defined by <a href="https://datatracker.ietf.org/doc/html/rfc2553#section-3.3">RFC 2553, section 3.3</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="k">struct</span> <span class="n">sockaddr_in6</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kt">sa_family_t</span>     <span class="n">sin6_family</span><span class="p">;</span>    <span class="cm">/* AF_INET6 */</span>
</span></span><span class="line"><span class="cl">    <span class="kt">in_port_t</span>       <span class="n">sin6_port</span><span class="p">;</span>      <span class="cm">/* transport layer port # */</span>
</span></span><span class="line"><span class="cl">    <span class="kt">uint32_t</span>        <span class="n">sin6_flowinfo</span><span class="p">;</span>  <span class="cm">/* IPv6 traffic class &amp; flow info */</span>
</span></span><span class="line"><span class="cl">    <span class="k">struct</span> <span class="n">in6_addr</span> <span class="n">sin6_addr</span><span class="p">;</span>      <span class="cm">/* IPv6 address */</span>
</span></span><span class="line"><span class="cl">    <span class="kt">uint32_t</span>        <span class="n">sin6_scope_id</span><span class="p">;</span>  <span class="cm">/* set of interfaces for a scope */</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span>
</span></span></code></pre></div><p>Similarly, Rust&rsquo;s <code>SocketAddrV4</code> mirrors the <code>sockaddr_in</code> struct. There isn&rsquo;t a similar RFC for <code>sockaddr_in</code>; the de facto standard is <a href="https://en.wikipedia.org/wiki/Berkeley_sockets">Berkeley sockets</a>, designed in 1983. The <a href="https://man7.org/linux/man-pages/man3/sockaddr.3type.html">Linux man page for <code>sockaddr_in</code></a> defines it as:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="k">struct</span> <span class="n">sockaddr_in</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kt">sa_family_t</span>     <span class="n">sin_family</span><span class="p">;</span>     <span class="cm">/* AF_INET */</span>
</span></span><span class="line"><span class="cl">    <span class="kt">in_port_t</span>       <span class="n">sin_port</span><span class="p">;</span>       <span class="cm">/* Port number */</span>
</span></span><span class="line"><span class="cl">    <span class="k">struct</span> <span class="n">in_addr</span>  <span class="n">sin_addr</span><span class="p">;</span>       <span class="cm">/* IPv4 address */</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span>
</span></span></code></pre></div><p>So <code>sin6_flowinfo</code>, which includes the Traffic Class, is part of <code>sockaddr_in6</code>, but the very similar DSCP field is not part of <code>sockaddr_in</code>. Why? I&rsquo;m not entirely sure about this, but here&rsquo;s an attempt to reconstruct a history:</p>
<ul>
<li>QoS was not originally part of the 1980s Berkeley sockets specification.</li>
<li>DSCP came about much later (<a href="https://www.rfc-editor.org/info/rfc2474">RFC 2474</a>, 1998).</li>
<li>Because C structs do not provide encapsulation, the <code>sockaddr_in</code> definition was set in stone and couldn&rsquo;t be changed.</li>
<li>So instead, the DSCP field is set as an <em>option</em> on the socket, via <a href="https://pubs.opengroup.org/onlinepubs/9799919799/functions/setsockopt.html"><code>setsockopt</code></a>.</li>
<li>By the time IPv6 came around, it was pretty clear that QoS was important, so the Traffic Class was baked into the <code>sockaddr_in6</code> struct.</li>
</ul>
<p>(Even if <code>sockaddr_in</code> could be extended to have this field, would it be a good idea to do so? Put a pin in this for now.)</p>
<h3 id="flow-labels">Flow Labels</h3>
<p>RFC 2460 says that the Flow Label is &ldquo;experimental and subject to change&rdquo;. The RFC was written back in 1998, over a quarter-century ago—has anyone found a use for it since then?</p>
<p><a href="https://www.rfc-editor.org/rfc/rfc6437">RFC 6437</a>, published in 2011, attempts to specify semantics for IPv6 Flow Labels. Section 2 of the RFC says:</p>
<blockquote>
<p>The 20-bit Flow Label field in the IPv6 header [RFC2460] is used by a
node to label packets of a flow. […] Packet classifiers can
use the triplet of Flow Label, Source Address, and Destination
Address fields to identify the flow to which a particular packet
belongs.</p>
</blockquote>
<p>The RFC says that Flow Labels can potentially be used by routers for load balancing, where they can use the triplet <em>source address, destination address, flow label</em> to figure out that a series of packets are all associated with each other. But this is an internal implementation detail generated by the source program, and not something IPv6 users copy/pasting an address generally have to think about. So it makes sense that it isn&rsquo;t part of the textual representation.</p>
<p><a href="https://www.rfc-editor.org/rfc/rfc6294">RFC 6294</a> surveys Flow Label use cases, and some of the ones mentioned are:</p>
<ul>
<li>as a pseudo-random value that can be used as part of a hash key for load balancing, or</li>
<li>as extra QoS bits on top of the 8 bits provided by the Traffic Class field.</li>
</ul>
<p>But <a href="https://networkengineering.stackexchange.com/a/79944">this Stack Exchange answer</a> by Andrei Korshikov says:</p>
<blockquote>
<p>Nowadays […] there [are] no clear advantages of additional 20-bit QoS field over existent Traffic Class (Differentiated Class of Service) field. So &ldquo;Flow Label&rdquo; is still waiting for its meaningful usage.</p>
</blockquote>
<p>In my view, putting <code>flowinfo</code> in <code>sockaddr_in6</code> was an understandable choice given the optimism around QoS in 1998, but it was a bit of a mistake in hindsight. The Flow Label field never found widespread adoption, and the Traffic Class field is more of an application-level concern. In general, I think there should be a separation between types that are losslessly serializable and types that are not, and <code>sockaddr_in6</code> violates this expectation. Making the Traffic Class (QoS) a socket option, like in IPv4, avoids these serialization issues.</p>
<h2 id="what-is-scope_id">What is <code>scope_id</code>?</h2>
<p>What about the other additional field, <code>scope_id</code>? What does it mean, and why does it <em>not</em> have to be zeroed out?</p>
<p>The documentation for a <code>SocketAddrV6</code> says that in its textual representation, the scope identifier is included after the IPv6 address and a <code>%</code> character, within square brackets. So, for example, the following code sample:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SocketAddrV6</span>::<span class="n">new</span><span class="p">(</span><span class="s">&#34;::1&#34;</span><span class="p">.</span><span class="n">parse</span><span class="p">().</span><span class="n">unwrap</span><span class="p">(),</span><span class="w"> </span><span class="mi">80</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="mi">42</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;</span><span class="si">{}</span><span class="s">&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">addr</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>prints out <code>[::1%42]:80</code>. What does this field mean?</p>
<p>The reason <code>scope_id</code> exists has to do with <em>link-local addressing</em>. Imagine you connect two computers directly to each other via, say, an Ethernet cable. There isn&rsquo;t a central server telling the computers which addresses to use, or anything similar—in this situation, how can the two computers talk to each other?</p>
<p>To address this issue, OS vendors came up with the idea to just assign random addresses on each end of the link. The behavior is defined in <a href="https://www.rfc-editor.org/rfc/rfc3927#section-2.1">RFC 3927, section 2.1</a>:</p>
<blockquote>
<p>When a host wishes to configure an IPv4 Link-Local address, it
selects an address using a pseudo-random number generator with a
uniform distribution in the range from 169.254.1.0 to 169.254.254.255
inclusive.</p>
</blockquote>
<p>(You might have seen these 169.254 addresses on your home computers if your router is down. Those are link-local addresses.)</p>
<p>Sounds simple enough, right? But there is a pretty big problem with this approach: what if a computer has more than one interface on which a link-local address has been established? When a program tries to send some data over the network, the computer has to know which interface to send the data out on. But with multiple link-local interfaces, the outbound one becomes ambiguous. This is described in <a href="https://www.rfc-editor.org/rfc/rfc3927#section-6.3">section 6.3</a> of the RFC:</p>
<blockquote>
<p>Address Ambiguity</p>
<p>Application software run on a multi-homed host that supports IPv4
Link-Local address configuration on more than one interface may fail.</p>
<p>This is because application software assumes that an IPv4 address is
unambiguous, that it can refer to only one host.  IPv4 Link-Local
addresses are unique only on a single link.  A host attached to
multiple links can easily encounter a situation where the same
address is present on more than one interface, or first on one
interface, later on another; in any case associated with more than
one host. […]</p>
</blockquote>
<p>The IPv6 protocol designers took this lesson to heart. Every time an IPv6-capable computer connects to a network, it establishes a link-local address starting with <code>fe80::</code>. (You should be able to see this address via <code>ip addr</code> on Linux, or your OS&rsquo;s equivalent.) But if you&rsquo;re connected to multiple networks, <em>all of them</em> will have addresses <a href="https://labs.ripe.net/author/philip_homburg/whats-the-deal-with-ipv6-link-local-addresses/">beginning with <code>fe80::</code></a>. Now if an application wants to establish a connection to a computer in this <code>fe80::</code> range, how can it tell the OS which interface to use?</p>
<p>That&rsquo;s exactly where <code>scope_id</code> comes in: it allows the <code>SocketAddrV6</code> to specify which network interface to use. Each interface has an index associated with it, which you can see on Linux with <code>ip addr</code>. When I run that command, I see:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> ip addr
</span></span><span class="line"><span class="cl"><span class="go">1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
</span></span></span><span class="line"><span class="cl"><span class="go">2: enp4s0f0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 9000 qdisc mq state UP group default qlen 1000
</span></span></span><span class="line"><span class="cl"><span class="go">    [...]
</span></span></span><span class="line"><span class="cl"><span class="go">    inet6 fe80::11fe:b754:2233:afb9/64 scope link noprefixroute
</span></span></span><span class="line"><span class="cl"><span class="go">       valid_lft forever preferred_lft forever
</span></span></span><span class="line"><span class="cl"><span class="go">3: enp4s0f1: &lt;NO-CARRIER,BROADCAST,MULTICAST,UP&gt; mtu 1500 qdisc mq state DOWN group default qlen 1000
</span></span></span><span class="line"><span class="cl"><span class="go">4: wlp13s0: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000
</span></span></span></code></pre></div><p>The <code>1</code>, <code>2</code>, <code>3</code> and <code>4</code> listed here are all the indexes that can be used as the scope ID. Let&rsquo;s try pinging our address:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> ping6 fe80::11fe:b754:2233:afb9
</span></span><span class="line"><span class="cl"><span class="go">ping6: Warning: IPv6 link-local address on ICMP datagram socket may require ifname or scope-id =&gt; use: address%&lt;ifname|scope-id&gt;
</span></span></span><span class="line"><span class="cl"><span class="go">PING fe80::11fe:b754:2233:afb9 (fe80::11fe:b754:2233:afb9) 56 data bytes
</span></span></span></code></pre></div><p>Aha! The warning tells us that for a link-local address, the scope ID needs to be specified. Let&rsquo;s try that using the <code>%</code> syntax:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> ping6 fe80::11fe:b754:2233:afb9%2
</span></span><span class="line"><span class="cl"><span class="go">PING fe80::11fe:b754:2233:afb9%2 (fe80::11fe:b754:2233:afb9%enp4s0f0) 56 data bytes
</span></span></span><span class="line"><span class="cl"><span class="go">64 bytes from fe80::11fe:b754:2233:afb9%enp4s0f0: icmp_seq=1 ttl=64 time=0.050 ms
</span></span></span><span class="line"><span class="cl"><span class="go">64 bytes from fe80::11fe:b754:2233:afb9%enp4s0f0: icmp_seq=2 ttl=64 time=0.052 ms
</span></span></span></code></pre></div><p>Success! What if we try a different scope ID?</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">%</span> ping6 fe80::11fe:b754:2233:afb9%3
</span></span><span class="line"><span class="cl"><span class="go">PING fe80::11fe:b754:2233:afb9%3 (fe80::11fe:b754:2233:afb9%enp4s0f1) 56 data bytes
</span></span></span><span class="line"><span class="cl"><span class="go">^C
</span></span></span><span class="line"><span class="cl"><span class="go">--- fe80::11fe:b754:2233:afb9%3 ping statistics ---
</span></span></span><span class="line"><span class="cl"><span class="go">3 packets transmitted, 0 received, 100% packet loss, time 2080ms
</span></span></span></code></pre></div><p>This makes sense: the address is only valid for scope ID 2 (the <code>enp4s0f0</code> interface). When we told <code>ping6</code> to use a different scope, 3, the address was no longer reachable. This neatly solves the 169.254 problem with IPv4 addresses.</p>
<p>Since scope IDs can help disambiguate the interface on which a connection ought to be made, it <em>does</em> make sense to include this field in <code>SocketAddrV6</code>, as well as in its textual representation.</p>
<div class="framed">
<h3 id="scope-id-portability">Scope ID portability</h3>
<p>The keen-eyed among you may have noticed that the <code>ping6</code> commands above printed out an alternate representation: <code>fe80::11fe:b754:2233:afb9%enp4s0f0</code>. The <code>enp4s0f0</code> at the end is the network interface that corresponds to the numeric scope ID. Many programs can handle this representation, but Rust&rsquo;s <code>SocketAddrV6</code> can&rsquo;t.</p>
<p>Another thing you might have noticed is that the scope ID only makes sense on a particular computer. A scope ID such as <code>2</code> means different things on different computers. So the scope ID is roundtrip serializable, but not portable across machines.</p>
</div>
<h2 id="conclusion">Conclusion</h2>
<p>In this post we started off by looking at a somewhat strange inconsistency and ended up deep in the IPv6 specification. In our case, the <code>SocketAddrV6</code> instances were always for internal services talking to each other without any QoS considerations, so <code>flowinfo</code> was always zero. Given that knowledge, we were okay adjusting the property-based tests to always generate instances where <code>flowinfo</code> was set to zero. (<a href="https://github.com/oxidecomputer/omicron/pull/9085">Here&rsquo;s the PR as landed</a>.)</p>
<p>Still, it raises questions: Should we wrap <code>SocketAddrV6</code> in a <a href="https://sunshowers.io/posts/types/#liminal-code">newtype</a> that enforces this constraint? Should <code>serde</code> provide a non-standard alternate serializer that also includes the <code>flowinfo</code> field? Should <code>Debug</code> not forward to <code>Display</code> when <code>Display</code> hides fields? Should Rust have had separate types from the start? (Probably too late now.) And should Berkeley sockets not have included <code>flowinfo</code> at all, given that it makes the type impossible to represent as text without loss?</p>
<p>The lesson it really drives home for me is how important the <a href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment">principle of least surprise</a> can be. Both <code>Ipv4Addr</code> and <code>Ipv6Addr</code> have lossless textual representations, and <code>SocketAddrV4</code> does as well. By analogy it would seem like <code>SocketAddrV6</code> would, too, and yet it does not!</p>
<p>IPv6 learned so much from IPv4&rsquo;s mistakes, and yet its designers couldn&rsquo;t help but make some mistakes of their own. This makes sense: the designers could only see the problems they were solving then, just as we can only see those we&rsquo;re solving now—and just as we encounter problems with their solutions, future generations will encounter problems with ours.</p>
<hr>
<p><em>Thanks to <a href="https://infosec.exchange/@munin">Fiona</a>, and several of my colleagues at Oxide, for reviewing drafts of this post.</em></p>
<p><em>Discuss on <a href="https://news.ycombinator.com/item?id=45813719">Hacker News</a> and <a href="https://lobste.rs/s/vq2xdc/socketaddrv6_is_not_roundtrip">Lobsters</a>.</em></p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p><em>Update 2025-11-05:</em> For map keys, the <code>serde_json</code> serializer and deserializer supports strings, as well as numbers which are serialized in their decimal representation as strings. (Originally, this said that JSON itself supports string or integer map keys, which was incorrect. Thanks to <a href="https://lobste.rs/s/vq2xdc/socketaddrv6_is_not_roundtrip#c_hr5g5q">Halkcyon on Lobsters</a> for the correction.)&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>This is why our Rust map crate where keys can borrow from values, <a href="https://docs.rs/iddqd/latest/iddqd/"><code>iddqd</code></a>, serializes its maps as lists or sequences.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>The Requests for Discussion we use at Oxide are <a href="https://rfd.shared.oxide.computer/rfd/0001">inspired</a> by RFCs, though we use a slightly different term (RFD) to convey the fact that our documents are less set in stone than IETF RFCs are.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>The two fields sum up to 28 bits, and the <code>flowinfo</code> field is a <code>u32</code>, so there&rsquo;s four bits remaining. I couldn&rsquo;t find documentation for these four bits anywhere—they appear to be unused padding in the <code>u32</code>. If you know about these bits, please let me know!&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/><category scheme="https://sunshowers.io/tags/ipv6" term="ipv6" label="ipv6"/></entry><entry><title type="html">Cancelling async Rust</title><link href="https://sunshowers.io/posts/cancelling-async-rust/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="related" type="text/html" title="Why nextest is process-per-test"/><link href="https://sunshowers.io/posts/beyond-ctrl-c-signals/?utm_source=atom_feed" rel="related" type="text/html" title="Beyond Ctrl-C: The dark corners of Unix signal handling"/><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="related" type="text/html" title="Open and closed universes"/><id>https://sunshowers.io/posts/cancelling-async-rust/</id><published>2025-10-03T00:00:00+00:00</published><updated>2025-10-24T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>Correctness in the face of cancellations: a written version of my talk at RustConf 2025.</blockquote><div class="sectioned-post">
<p><em>This is an edited, written version of my RustConf 2025 talk about cancellations in async Rust. Like <a href="/posts/beyond-ctrl-c-signals/">the written version of my RustConf 2023 talk</a>, I&rsquo;ve tried to retain the feel of a talk while making it readable as a standalone blog entry. Some links:</em></p>
<ul>
<li><em><a href="https://www.youtube.com/watch?v=zrv5Cy1R7r4">Video of the talk</a> on YouTube.</em></li>
<li><em><a href="https://docs.google.com/presentation/u/1/d/e/2PACX-1vTMc4EdHRf6ulz-xaAhZFGZwxJ7jPQgYWczT6pEIvwfXILV4ZEgMdLuoRh70bgh9SP7mxblEnyXuZD0/pub?start=false&amp;loop=false&amp;delayms=60000">Slides</a> on Google Slides.</em></li>
<li><em><a href="https://github.com/sunshowers/cancelling-async-rust">Repository with links and notes</a> on GitHub.</em></li>
<li><em><a href="https://lwn.net/Articles/1036924/">Coverage on Linux Weekly News</a>.</em></li>
</ul>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
			<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/zrv5Cy1R7r4?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"></iframe>
		</div>

<h2 id="introduction">Introduction</h2>
<p>Let’s start with a simple example – you decide to read from a channel in a loop and gather a bunch of messages:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">match</span><span class="w"> </span><span class="n">rx</span><span class="p">.</span><span class="n">recv</span><span class="p">().</span><span class="k">await</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">process</span><span class="p">(</span><span class="n">msg</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>All good, nothing wrong with this, but you realize sometimes the channel is empty for long periods of time, so you add a timeout and print a message:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">    </span><span class="k">match</span><span class="w"> </span><span class="n">timeout</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span><span class="w"> </span><span class="n">rx</span><span class="p">.</span><span class="n">recv</span><span class="p">()).</span><span class="k">await</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Ok</span><span class="p">(</span><span class="n">msg</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">process</span><span class="p">(</span><span class="n">msg</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="p">,</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">        </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;no messages for 5 seconds&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>There&rsquo;s nothing wrong with this code—it behaves as expected.</p>
<p>Now you realize you need to write a bunch of messages out to a channel in a loop:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">msg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">next_message</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">match</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">msg</span><span class="p">).</span><span class="k">await</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;sent successfully&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>But sometimes the channel gets too full and blocks, so you add a timeout and print a message:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">msg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">next_message</span><span class="p">();</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">    </span><span class="k">match</span><span class="w"> </span><span class="n">timeout</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">msg</span><span class="p">)).</span><span class="k">await</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Ok</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;sent successfully&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="p">,</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">        </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;no space for 5 seconds&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>It turns out that <em>this</em> code is often incorrect, because not all messages make their way to the channel.</p>
<p>Hi, I’m Rain, and this post is about cancelling async Rust. This post is split into three parts:</p>
<ol>
<li><a href="/posts/cancelling-async-rust/#1-what-is-cancellation"><em>What is cancellation?</em></a> It&rsquo;s an extremely powerful part of async Rust but also one that is very hard to reason thoroughly about.</li>
<li><a href="/posts/cancelling-async-rust/#2-analyzing-cancellations"><em>Analyzing cancellations:</em></a> Going deep into their mechanics and providing some helpful ways to think about them.</li>
<li><a href="/posts/cancelling-async-rust/#3-what-can-be-done"><em>What can be done?</em></a> Solutions, including practical guidance, and real bugs we’ve found and fixed in production codebases.</li>
</ol>
<p>Before we begin, I want to lay my cards on the table – I really love async Rust!</p>

  <figure class="floatright" >
    
    <img src="/images/beyond-ctrl-c.jpg"  alt="Me speaking at RustConf 2023. Beyond Ctrl-C: The dark corners of Unix signal handling."   />
    
    
      <figcaption class="center" >Me speaking at RustConf 2023.</figcaption>
    
  </figure>


<ul>
<li>
<p>I gave <a href="/posts/beyond-ctrl-c-signals/">a talk</a> at RustConf a couple years ago talking about how async Rust is a great fit for signal handling in complex applications.</p>
</li>
<li>
<p>I’m also the author of <a href="https://nexte.st/">cargo-nextest</a>, a next-generation test runner for Rust, where async Rust is the best way I know of to express some really complex algorithms that I wouldn’t know how to express otherwise. I wrote <a href="/posts/nextest-and-tokio/">a blog post</a> about this a few years ago.</p>
</li>
</ul>
<p>Now, I work at <a href="https://oxide.computer/">Oxide Computer Company</a>, where we make <em>cloud-in-a-box computers</em>. We make vertically integrated systems where you provide power and networking on one end, and the software you want to run on the other end, and we take care of everything in between.</p>
<p>Of course, we use Rust everywhere, and in particular we use async Rust extensively for our higher-level software, such as <a href="https://github.com/oxidecomputer/crucible">storage</a>, <a href="https://github.com/oxidecomputer/maghemite">networking</a> and the <a href="https://github.com/oxidecomputer/omicron/">customer-facing management API</a>. But along the way we’ve encountered a number of issues around async cancellation, and a lot of this post is about what we learned along the way.</p>
<h2 id="1-what-is-cancellation">1. What is cancellation?</h2>
<p>What does cancellation mean? Logically, a cancellation is exactly what it sounds like: you start some work, and then change your mind and decide to stop doing that work.</p>
<p>As you might imagine this is a useful thing to do:</p>
<ul>
<li>You may have started a large download or a long network request</li>
<li>Maybe you’ve started reading a file, similar to the <code>head</code> command.</li>
</ul>
<p>But then you change your mind: you want to cancel it rather than continue it to completion.</p>
<h3 id="cancellations-in-synchronous-rust">Cancellations in synchronous Rust</h3>
<p>Before we talk about async Rust, it’s worth thinking about how you’d do cancellations in synchronous Rust.</p>
<p>One option is to have some kind of <em>flag</em> you periodically check, maybe stored in an atomic:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">while</span><span class="w"> </span><span class="o">!</span><span class="n">should_cancel</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">Ordering</span>::<span class="n">Relaxed</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">expensive_operation</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><ul>
<li>The code that wishes to perform the cancellation can set that flag.</li>
<li>Then, the code which checks that flag can exit early.</li>
</ul>
<p>This approach is fine for smaller bits of code but doesn’t really scale well to large chunks of code since you’d have to sprinkle these checks everywhere.</p>
<p>A related option, if you’re working with a framework as part of your work, is to <em>panic with a special payload</em> of some kind.</p>
<ul>
<li>If that feels strange to you, you’re not alone! But the <a href="https://github.com/salsa-rs/salsa">Salsa</a> framework for incremental computation, used by—among other things—<a href="https://github.com/rust-lang/rust-analyzer">rust-analyzer</a>, uses this approach.</li>
<li>Something I learned recently was that this only works on build targets which have a notion of <a href="https://doc.rust-lang.org/std/panic/fn.catch_unwind.html">panic unwinding</a>, or being able to bubble up the panic. Not all platforms support this, and in particular, <a href="https://webassembly.org/">Wasm</a> doesn&rsquo;t. This means that Salsa cancellations don’t work if you build rust-analyzer for Wasm.</li>
</ul>
<p>A third option is to <em>kill the whole process</em>. This is a <em>very heavyweight</em> approach, but an effective one in case you spawn processes to do your work.</p>
<p>Rather than kill the whole process, can you kill a single thread?</p>
<ul>
<li>While <a href="https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminatethread">some OSes have APIs</a> to perform this action, they tend to warn very strongly against it. That’s because in general, most code is <a href="/posts/nextest-process-per-test/#appendix">just not ready</a> for a thread disappearing from underneath.</li>
<li>In particular, thread killing is not permitted by safe Rust, since it can cause serious corruption. For example, Rust mutexes would likely stay locked forever.</li>
</ul>
<p>All of these options are suboptimal or of limited use in some way. In general, the way I think about it is that there isn’t a <em>universal protocol</em> for cancellation in synchronous Rust.</p>
<p>In contrast, there <em>is</em> such a protocol in async Rust, and in fact cancellations are extraordinarily easy to perform in async Rust.</p>
<p>Why is that so? To understand that, let’s look at what a future is.</p>
<h3 id="what-is-a-future">What is a future?</h3>
<p>Here’s a simple example of a future:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// This creates a state machine.
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">request</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">process</span><span class="p">(</span><span class="n">data</span><span class="p">).</span><span class="k">await</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">};</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Nothing executes yet. `future` is just a struct in memory.
</span></span></span></code></pre></div><p>In this future, you first perform a network request which returns some data, and then you process it.</p>
<p>The Rust compiler looks at this future and generates a state machine, which is just a struct or enum in memory:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// The compiler generates something like:
</span></span></span><span class="line"><span class="cl"><span class="k">enum</span> <span class="nc">MyFuture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Start</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">WaitingForNetwork</span><span class="p">(</span><span class="n">NetworkFuture</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">WaitingForProcess</span><span class="p">(</span><span class="n">ProcessFuture</span><span class="p">,</span><span class="w"> </span><span class="n">Data</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Done</span><span class="p">(</span><span class="nb">Result</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// It&#39;s just data, no running code!
</span></span></span></code></pre></div><p>If you&rsquo;ve written async Rust before the <code>async</code> and <code>await</code> keywords, you’ve probably written code like it by hand. It’s basically just an enum describing all the possible states the future can be in.</p>
<p>The compiler also generates an implementation of <a href="https://doc.rust-lang.org/std/future/trait.Future.html">the <code>Future</code> trait</a> for this future:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Future</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyFuture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">poll</span><span class="p">(</span><span class="cm">/* ... */</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Poll</span><span class="o">&lt;</span><span class="bp">Self</span>::<span class="n">Output</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">match</span><span class="w"> </span><span class="bp">self</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">Start</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* ... */</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="n">WaitingForNetwork</span><span class="p">(</span><span class="n">fut</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* ... */</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c1">// etc
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>and when you call <code>.await</code> on the future, it gets translated down to this underlying <code>poll</code> function. It is only when <code>await</code> or this <code>poll</code> function is called that something actually happens.</p>
<p>Note that this is diametrically opposed to how async works in other languages like Go, JavaScript, or C#. In those languages, when you create a future to await on, it starts doing its thing, immediately, in the background:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="c1">// JavaScript: starts running immediately
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">promise</span> <span class="o">=</span> <span class="nx">fetch</span><span class="p">(</span><span class="s1">&#39;/api/data&#39;</span><span class="p">);</span>
</span></span></code></pre></div><p>That’s regardless of whether you await it or not.</p>
<p>In Rust, this <code>get</code> call does nothing until you actually call <code>.await</code> on it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Rust: just data, does nothing!
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">reqwest</span>::<span class="n">get</span><span class="p">(</span><span class="s">&#34;/api/data&#34;</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>I know I sound a bit like a broken record here, but if you can take away one thing from this post, it would be that futures are passive, and completely inert until awaited or polled.</p>
<h3 id="the-universal-protocol">The universal protocol</h3>
<p>So what does the universal protocol to cancel futures look like?  It is simply to <em>drop</em> the future, or to not await it, or poll it any more. Since a future is just a state machine, you can throw it away at any time the poll function isn’t actively being called.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">some_async_work</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nb">drop</span><span class="p">(</span><span class="n">future</span><span class="p">);</span><span class="w"> </span><span class="c1">// cancelled
</span></span></span></code></pre></div><p>The upshot of all this is that any Rust future can be cancelled at any await point.</p>
<p>Given how hard cancellation tends to be in synchronous environments, the ability to easily cancel futures in async Rust is extraordinarily powerful—in many ways its greatest strength!</p>
<p>But there is a flip side, which is that cancelling futures is far, far too easy. This is for two reasons.</p>
<ol>
<li>
<p>First, it’s just way too easy to quietly drop a future. As we’re going to see, there are all kinds of code patterns that lead to silently dropping futures.</p>
</li>
<li>
<p>Now this wouldn’t be so bad, if not for the second reason: that cancellation of parent futures propagates down to child futures.</p>
<p>Because of Rust’s <a href="https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html">single ownership model</a>, child futures are owned by parent ones. If a parent future is dropped or cancelled, the same happens to the child.</p>
<p>To figure out whether a child future’s cancellation can cause issues, you have to look at its parent, and grandparent, and so on. Reasoning about cancellation becomes a very complicated <em>non-local operation</em>.</p>
</li>
</ol>
<h2 id="2-analyzing-cancellations">2. Analyzing cancellations</h2>
<p>I’m going to cover some examples in a bit, but before we do that I want to talk about a couple terms, some of which you might have seen references to already.</p>
<h3 id="cancel-safety-and-cancel-correctness">Cancel safety and cancel correctness</h3>
<p>The first term is <strong>cancel safety</strong>. You might have seen mentions of this in the Tokio documentation. Cancel safety, as generally defined, means the property of a future that can be cancelled (i.e. dropped) without any side effects.</p>
<p>For example, a <a href="https://docs.rs/tokio/latest/tokio/time/fn.sleep.html">Tokio sleep future</a> is cancel safe: you can just stop waiting on the sleep and it’s completely fine.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">time</span>::<span class="n">sleep</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nb">drop</span><span class="p">(</span><span class="n">future</span><span class="p">);</span><span class="w"> </span><span class="c1">// this has no side effects
</span></span></span></code></pre></div><p>An example of a future that is not cancel safe is <a href="https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Sender.html#method.send">Tokio’s MPSC send</a>, which sends a message over a channel:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="cm">/* ... */</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">message</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nb">drop</span><span class="p">(</span><span class="n">future</span><span class="p">);</span><span class="w"> </span><span class="c1">// message is lost!
</span></span></span></code></pre></div><p>If this future is dropped, the message is lost forever.</p>
<p>The important thing is that cancel safety is a <em>local property</em> of an <em>individual future</em>.</p>
<p>But cancel safety is not all that one needs to care about. What actually matters is the context the cancellation happens in, or in other words whether the cancellation actually causes some kind of larger property in the system to be violated.</p>
<ul>
<li>For example, if you drop a future which sends a message, but for whatever reason you don’t care about the message any more, it’s not really a bug!</li>
</ul>
<p>To capture this I tend to use a different term called <strong>cancel correctness</strong>, which I define as a <em>global property</em> of <em>system correctness</em> in the face of cancellations. (This isn’t a standard term, but it’s a framing I’ve found really helpful in understanding cancellations.)</p>
<p>When is cancel correctness violated? It requires three things:</p>
<ol>
<li>
<p><em>The system has a cancel-unsafe future somewhere within it.</em> As we’ll see, many APIs that are cancel-unsafe can be reworked to be cancel-safe. If there aren’t any cancel-unsafe futures in the system, then the system is cancel correct.</p>
</li>
<li>
<p><em>A cancel-unsafe future is actually cancelled.</em> This may sound a bit trivial, but if cancel-unsafe futures are always run to completion, then the system can’t have cancel correctness bugs.</p>
</li>
<li>
<p><em>Cancelling the future violates some property of a system.</em> This could be data loss as with <code>Sender::send</code>, some kind of invariant violation, or some kind of cleanup that must be performed but isn&rsquo;t.</p>
</li>
</ol>
<p>So a lot of making Rust async robust is about trying to tackle one of these three things.</p>
<p>I want to zoom in for a second on invariant violations and talk about an example of a Tokio API that is very prone to cancel correctness issues: <a href="https://docs.rs/tokio/latest/tokio/sync/struct.Mutex.html">Tokio mutexes</a>.</p>
<h3 id="the-pain-of-tokio-mutexes">The pain of Tokio mutexes</h3>
<p>The way Tokio mutexes work is: you create a mutex, you lock it which gives you mutable access to the data underneath, and then you unlock it by releasing the mutex.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">guard</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mutex</span><span class="p">.</span><span class="n">lock</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// Access guard.data, protected by the mutex...
</span></span></span><span class="line"><span class="cl"><span class="nb">drop</span><span class="p">(</span><span class="n">guard</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>If you look at the <a href="https://docs.rs/tokio/latest/tokio/sync/struct.Mutex.html#method.lock"><code>lock</code> function&rsquo;s documentation</a>, in the “cancel safety” section it says:</p>
<blockquote>
<p>This method uses a queue to fairly distribute locks in the order they were requested. Cancelling a call to lock makes you lose your place in the queue.</p>
</blockquote>
<p>Okay, so not totally cancel safe, but the only kind of unsafety is fairness, which doesn’t sound too bad.</p>
<p>But the problems lie in what you actually do with the mutex. In practice, most uses of mutexes are in order to <em>temporarily violate invariants</em> that are otherwise upheld when a lock isn’t held.</p>
<p>I’ll use a real world example of a cancel correctness bug that we found at my job at Oxide: we had code to manage a bunch of data sent over by our computers, which we call sleds. The shared state was guarded by a mutex, and a typical operation was:</p>
<ol>
<li>Obtain a lock on the mutex.</li>
<li>Obtain the sled-specific data by value, moving it to an invalid <code>None</code> state.</li>
<li>Perform an action.</li>
<li>Set the sled-specific data back to the next valid state.</li>
</ol>
<p>Here&rsquo;s a rough sketch of what that looks like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">guard</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mutex</span><span class="p">.</span><span class="n">lock</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// guard.data is Option&lt;T&gt;: Some to begin with
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">guard</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">take</span><span class="p">();</span><span class="w"> </span><span class="c1">// guard.data is now None
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">new_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">process_data</span><span class="p">(</span><span class="n">data</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">guard</span><span class="p">.</span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">new_data</span><span class="p">);</span><span class="w"> </span><span class="c1">// guard.data is Some again
</span></span></span></code></pre></div><p>This is all well and good, but the problem is that the action being performed actually had an await point within it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">guard</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mutex</span><span class="p">.</span><span class="n">lock</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// guard.data is Option&lt;T&gt;: Some to begin with
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">guard</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">take</span><span class="p">();</span><span class="w"> </span><span class="c1">// guard.data is now None
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// DANGER: cancellation here leaves data in None state!
</span></span></span><span class="line hl"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">new_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">process_data</span><span class="p">(</span><span class="n">data</span><span class="p">).</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">guard</span><span class="p">.</span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">new_data</span><span class="p">);</span><span class="w"> </span><span class="c1">// guard.data is Some again
</span></span></span></code></pre></div><p>If the code that operated on the mutex got cancelled at that await point, then the data would be stuck in the invalid <code>None</code> state. Not great!</p>
<p>And keep in mind the non-local reasoning aspect: when doing this analysis, you need to look at the whole chain of callers.</p>
<h3 id="cancellation-patterns">Cancellation patterns</h3>
<p>Now that we’ve talked about some of the bad things that can happen during cancellations, it’s worth asking what kinds of code patterns lead to futures being cancelled.</p>
<p>The most straightforward example, and maybe a bit of a silly one, is that you create a future but simply forget to call <code>.await</code> on it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="n">some_async_work</span><span class="p">();</span><span class="w"> </span><span class="c1">// missing .await
</span></span></span></code></pre></div><p>Now Rust actually warns you if you don&rsquo;t call <code>.await</code> on the future:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">warning: unused implementer of `Future` that must be used
</span></span><span class="line"><span class="cl">   |
</span></span><span class="line"><span class="cl">11 |     some_async_work();
</span></span><span class="line"><span class="cl">   |     ^^^^^^^^^^^^^^^^^
</span></span><span class="line"><span class="cl">   |
</span></span><span class="line"><span class="cl">   = note: futures do nothing unless you `.await` or poll them
</span></span></code></pre></div><p>But a code pattern I’ve sometimes made mistakes with is that the future returns a <code>Result</code>, and you want to ignore the result so you assign it to an underscore like so:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">some_async_work</span><span class="p">();</span><span class="w"> </span><span class="c1">// future returns Result
</span></span></span></code></pre></div><p>If I forget to call <code>.await</code> on the future, Rust doesn’t warn me about it at all, and then I’m left scratching my head about why this code didn’t run. I know this sounds really silly and basic, but I’ve made this mistake a bunch of times.</p>
<p><em>(After my talk, it was pointed out to me that Clippy 1.67 and above have a <a href="https://rust-lang.github.io/rust-clippy/master/index.html#let_underscore_future"><code>let_underscore_future</code></a> warn-by-default lint for this. Hooray!)</em></p>
<p>Another example of futures being cancelled is <code>try</code> operations, such as Tokio’s <a href="https://docs.rs/tokio/latest/tokio/macro.try_join.html"><code>try_join</code> macro</a>. For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">do_stuff_async</span><span class="p">()</span><span class="w"> </span>-&gt; <span class="nb">Result</span><span class="o">&lt;</span><span class="p">(),</span><span class="w"> </span><span class="o">&amp;</span><span class="nb">&#39;static</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// async work
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">more_async_work</span><span class="p">()</span><span class="w"> </span>-&gt; <span class="nb">Result</span><span class="o">&lt;</span><span class="p">(),</span><span class="w"> </span><span class="o">&amp;</span><span class="nb">&#39;static</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// more here
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="fm">try_join!</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">do_stuff_async</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">more_async_work</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// ...
</span></span></span></code></pre></div><p>If you call <code>try_join</code> with a bunch of futures, and all of them succeed, it’s all good. But if one of them fails, the rest simply get cancelled.</p>
<p>In fact, at Oxide we had a pretty bad bug around this: we had code to stop a bunch of services, all expressed as futures. We used <code>try_join</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="fm">try_join!</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">stop_service_a</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">stop_service_b</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">stop_service_c</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>If one of these operations failed for whatever reason, we would stop running the code to wait for the other services to exit. Oops!</p>
<p>But perhaps the most well-known source of cancellations is Tokio&rsquo;s <a href="https://docs.rs/tokio/latest/tokio/macro.select.html"><code>select</code> macro</a>. Select is this incredibly beautiful operation. It is called with a set of futures, and it drives all of them forward concurrently:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">result1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">future1</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">handle_result1</span><span class="p">(</span><span class="n">result1</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">result2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">future2</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">handle_result2</span><span class="p">(</span><span class="n">result2</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Each future has a code block associated with it (above, <code>handle_result1</code> and <code>handle_result2</code>). If one of the futures completes, the corresponding code block is called. But also, all of the other futures are always cancelled!</p>
<p>For a variety of reasons, select statements in general, and <em>select loops</em> in particular, are particularly prone to cancel correctness issues. So a lot of the documentation about cancel safety talks about select loops. But I want to emphasize here that select is not the <em>only</em> source of cancellations, just a particularly <em>notable</em> one.</p>
<h2 id="3-what-can-be-done">3. What can be done?</h2>
<p>So, now that we’ve looked at all of these issues with cancellations, what can be done about it?</p>
<p>First, I want to break the bad news to you – there is <em>no general, fully reliable solution for this</em> in Rust today. But in our experience there are a few patterns that have been successful at reducing the likelihood of cancellation bugs.</p>
<p>Going back to our definition of cancel correctness, there are three prongs all of which come together to produce a bug:</p>
<ul>
<li>A cancel-unsafe future exists</li>
<li>This cancel-unsafe future is cancelled</li>
<li>The cancellation violates a system property</li>
</ul>
<p>Most solutions we’ve come up with try and tackle one of these prongs.</p>
<h3 id="making-futures-cancel-safe">Making futures cancel-safe</h3>
<p>Let’s look at the first prong: the system has a cancel-unsafe future somewhere in it. Can we use code patterns to make futures be cancel-safe? It turns out we can! I’ll give you two examples here.</p>
<p>The first is MPSC sends. Let’s come back to the example from earlier where we would lose messages entirely:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">msg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">next_message</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">match</span><span class="w"> </span><span class="n">timeout</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">msg</span><span class="p">)).</span><span class="k">await</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Ok</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;sent successfully&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;no space for 5 seconds&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Can we find a way to make this cancel safe?</p>
<p>In this case, yes, and we do so by breaking up the operation into two parts:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">    </span><span class="k">match</span><span class="w"> </span><span class="n">timeout</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="n">reserve</span><span class="p">()).</span><span class="k">await</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Ok</span><span class="p">(</span><span class="n">permit</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">            </span><span class="n">permit</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">next_message</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;sent successfully&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">))</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nb">Err</span><span class="p">(</span><span class="n">_</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">&#34;no space for 5 seconds&#34;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><ul>
<li>The first component is the operation to <a href="https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Sender.html#method.reserve">reserve a permit</a> or slot in the channel. This is an initial async operation that’s cancel-safe.</li>
<li>The second is to generate, then <a href="https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Permit.html#method.send">send the message</a>, which is an operation that becomes infallible.</li>
</ul>
<p>(I want to put an asterisk here that reserve is not entirely cancel-safe, since Tokio’s MPSC follows a first-in-first-out pattern and dropping the future means losing your place in line. Keep this in mind for now.)</p>
<p><em>Update 2025-10-24: The code sample now calls <code>next_message</code> after a permit has been reserved. Thanks to <a href="https://lobste.rs/s/d4batn/cancelling_async_rust#c_9pm7e4">quad on Lobsters</a> for the correction.</em></p>
<hr>
<p>The second is with <a href="https://docs.rs/tokio/latest/tokio/io/trait.AsyncWrite.html">Tokio’s <code>AsyncWrite</code></a>.</p>
<p>If you’ve written synchronous Rust you’re probably familiar with <a href="https://doc.rust-lang.org/std/io/trait.Write.html#method.write_all">the <code>write_all</code> method</a>, which writes an entire buffer out:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Write</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">buffer</span>: <span class="kp">&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="cm">/* ... */</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">writer</span><span class="p">.</span><span class="n">write_all</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>In synchronous Rust, this is a great API. But within async Rust, the <code>write_all</code> pattern is absolutely not cancel safe! If the future is dropped before completion, you have no idea how much of this buffer was written out.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">io</span>::<span class="n">AsyncWriteExt</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">buffer</span>: <span class="kp">&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="cm">/* ... */</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">writer</span><span class="p">.</span><span class="n">write_all</span><span class="p">(</span><span class="n">buffer</span><span class="p">).</span><span class="k">await</span><span class="o">?</span><span class="p">;</span><span class="w"> </span><span class="c1">// Not cancel-safe!
</span></span></span></code></pre></div><p>But there’s an alternative API that is cancel-safe, called <a href="https://docs.rs/tokio/latest/tokio/io/trait.AsyncWriteExt.html#method.write_all_buf"><code>write_all_buf</code></a>. This API is carefully designed to enable the reporting of partial progress, and it doesn’t just accept a <em>buffer</em>, but rather something that looks like a cursor on top of it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">io</span>::<span class="n">AsyncWriteExt</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">buffer</span>: <span class="nc">io</span>::<span class="n">Cursor</span><span class="o">&lt;&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">]</span><span class="o">&gt;</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="cm">/* ... */</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">writer</span><span class="p">.</span><span class="n">write_all_buf</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">buffer</span><span class="p">).</span><span class="k">await</span><span class="o">?</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>When part of the buffer is written out, the cursor is advanced by that number of bytes. So if you call <code>write_all_buf</code> in a loop, you’ll be resuming from this partial progress, which works great.</p>
<h3 id="not-cancelling-futures">Not cancelling futures</h3>
<p>Going back to the three prongs: the second prong is about actually cancelling futures. What code patterns can be used to not cancel futures? Here are a couple of examples.</p>
<p>The first one is, in a place like a select loop, resume futures rather than cancelling them each time. You’d typically achieve this by <a href="https://doc.rust-lang.org/std/pin/">pinning a future</a>, and then polling a mutable reference to that future. For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line hl"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span>::<span class="n">pin</span><span class="p">(</span><span class="n">channel</span><span class="p">.</span><span class="n">reserve</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">tokio</span>::<span class="fm">select!</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line hl"><span class="cl"><span class="w">        </span><span class="n">permit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">future</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="w"> </span><span class="n">permit</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">_</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">other_condition</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">continue</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Coming back to our example of MPSC sends, the one asterisk with <code>reserve</code> is that cancelling it makes you lose your place in line. Instead, if you pin the <code>reserve</code> future and poll a mutable reference to it, you don’t lose your place in line.</p>
<p>(Does the difference here matter? It depends, but you can now have this strategy available to you.)</p>
<hr>
<p>The second example is to use tasks. I mentioned earlier that futures are Rust are diametrically opposed to similar notions in languages like JavaScript. Well, there’s an alternative in async Rust that&rsquo;s much closer to the JavaScript idea, and that’s <a href="https://docs.rs/tokio/latest/tokio/task/">tasks</a>.</p>
<ul>
<li>Unlike futures which are driven by the caller, tasks are driven by the runtime (such as Tokio).</li>
<li>With Tokio, dropping a handle to a task does not cause it to be cancelled, which means they’re a good place to run cancel-unsafe code.</li>
</ul>
<p>A fun example is that at Oxide, we have an HTTP server called <a href="https://docs.rs/dropshot">Dropshot</a>. Previously, whenever an HTTP request came in, we’d use a future for it, and drop the future if the TCP connection was closed.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// Before: Future cancelled on TCP close
</span></span></span><span class="line"><span class="cl"><span class="n">handle_request</span><span class="p">(</span><span class="n">req</span><span class="p">).</span><span class="k">await</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>This was really bad because future cancellations could happen due to the behavior of not just the parent future, but of a process that was running across a network! This is a rather extreme form of non-local reasoning.</p>
<p>We addressed this by spinning up a task for each HTTP request, and by running the code to completion even if the connection is closed:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// After: Task runs to completion
</span></span></span><span class="line"><span class="cl"><span class="n">tokio</span>::<span class="n">spawn</span><span class="p">(</span><span class="n">handle_request</span><span class="p">(</span><span class="n">req</span><span class="p">));</span><span class="w">
</span></span></span></code></pre></div><h3 id="systematic-solutions">Systematic solutions?</h3>
<p>The last thing I want to say is that <strong>this sucks</strong>!</p>
<p>The promise of Rust is that you don’t need to do this kind of non-local reasoning—that you can understand important behavior by looking at code directly around the behavior, then use the type system to scale that up to global correctness. Almost everything in Rust, from <code>&amp;</code> and <code>&amp;mut</code> to <code>unsafe</code>, is geared towards making that possible. However, future cancellations fly directly in the face of that, and I think they’re probably the <strong>least Rusty part of Rust</strong>. This is all really unfortunate.</p>
<p>Can we come up with something more systematic than this kind of ad-hoc reasoning?</p>
<p>There doesn’t exist anything in safe Rust today, but there are a few different ideas people have come up with. I wanted to give a nod to those ideas:</p>
<ul>
<li><em>Async drop</em> would let you run async code when a future is cancelled. This would handle some, though not all, of the cases we discussed today.</li>
<li>There’s also a couple different proposals for what are called <em>linear types</em>, where you could force some code to be run on drop, or mark a particular future as non-cancellable (once it’s been created it must be driven to completion).</li>
</ul>
<p>All of these options have really significant implementation challenges, though. This <a href="https://without.boats/blog/asynchronous-clean-up/">blog post from boats</a> covers some of these solutions, and the implementation challenges with them.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this post, we:</p>
<ul>
<li>Saw that futures are passive</li>
<li>Introduced cancel safety and cancel correctness as concepts</li>
<li>Examined some bugs that can occur with cancellation</li>
<li>Looked at some recommendations you can use to mitigate the downsides of cancellation</li>
</ul>
<p>Some of the recommendations are:</p>
<ul>
<li>Avoid Tokio mutexes</li>
<li>Rewrite APIs to make futures cancel-safe</li>
<li>Find ways to ensure that cancel-unsafe futures are driven to completion</li>
</ul>
<p>There&rsquo;s a very deep well of complexity here, a lot more than I can cover in one blog post:</p>
<ul>
<li>Why are futures passive, anyway?</li>
<li>Cooperative cancellation: cancellation tokens</li>
<li>Actor model as an alternative to Tokio mutexes</li>
<li>Task aborts</li>
<li>Structured concurrency</li>
<li>Relationship to panic safety and mutex poisoning</li>
</ul>
<p>If you’re curious about any of these, check out <a href="https://github.com/sunshowers/cancelling-async-rust">this link</a> where I’ve put together a collection of documents and blog posts about these concepts. In particular, I&rsquo;d recommend reading these two Oxide RFDs:</p>
<ul>
<li><a href="https://rfd.shared.oxide.computer/rfd/397">RFD 397 Challenges with async/await in the control plane</a> by David Pacheco</li>
<li><a href="https://rfd.shared.oxide.computer/rfd/400">RFD 400 Dealing with cancel safety in async Rust</a> by myself</li>
</ul>
<p><em>Thank you for reading this post to the end! And thanks to many of my coworkers at Oxide for reviewing the talk and the RFDs linked above, and for suggestions and constructive feedback.</em></p>
</div>
]]></content><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/><category scheme="https://sunshowers.io/tags/talks" term="talks" label="talks"/></entry><entry><title type="html">Why nextest is process-per-test</title><link href="https://sunshowers.io/posts/nextest-process-per-test/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/nextest-and-tokio/?utm_source=atom_feed" rel="related" type="text/html" title="How (and why) nextest uses tokio"/><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="related" type="text/html" title="Open and closed universes"/><id>https://sunshowers.io/posts/nextest-process-per-test/</id><published>2025-01-07T00:00:00+00:00</published><updated>2025-01-07T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>Why nextest runs each test in its own process: a game-theoretic view.</blockquote><p><em>I&rsquo;m often asked why the Rust test runner I maintain, <a href="https://nexte.st/">cargo-nextest</a>, runs every test in a separate process. Here&rsquo;s my best attempt at explaining the rationale behind it.</em></p>
<p><em>This document is cross-posted from the canonical copy at <a href="https://nexte.st/docs/design/why-process-per-test/">the nextest site</a>. Benchmarks <a href="https://nexte.st/docs/benchmarks/">are available</a> there as well, though a large part of this document talks about non-benchmarkable reliability improvements.</em></p>
<h1 id="why-process-per-test">Why process-per-test?</h1>
<p>A key factor distinguishing nextest from <code>cargo test</code> is that nextest runs <strong>each test in a separate process</strong>. With nextest, the default execution model is now, and will always be, process-per-test. This document explains why.</p>
<p>Some terms used in this document:</p>
<ul>
<li><em>Process-per-test:</em> Each test lives in its own process.</li>
<li><em>Shared-process:</em> Several tests share the same process.</li>
</ul>
<h2 id="the-ecosystem-coordination-challenge">The ecosystem coordination challenge</h2>
<p>The Rust ecosystem is now staggeringly large. There are millions of Rust developers and billions of lines of Rust code in production today. A key challenge with driving adoption of any new technology, like a new test runner, is <strong>coordination</strong>: everyone needs to agree on how to run and manage them.</p>
<ul>
<li>There are vast numbers of existing tests, representing millions of developer-hours of effort, that must continue to work</li>
<li>These tests integrate with all sorts of different libraries, including many not written in Rust, or with other unusual requirements</li>
<li>Test runners need to be able to run tests reliably, and rely on certain properties—for example, that killing one test will have no effect on others</li>
</ul>
<h2 id="schelling-points-in-game-theory">Schelling points in game theory</h2>

  <figure class="floatright" >
    
    <img src="/images/schelling.jpg"  alt="Photo of Nobel laureate Thomas Schelling, who first described focal (or Schelling) points."   />
    
    
      <figcaption class="center" >Thomas Schelling won the 2005 Nobel Prize in Economics for his work on focal points. <a href="https://www.bostonglobe.com/metro/obituaries/2016/12/17/thomas-schelling-nobel-winning-economist-who-influenced-nuclear-policy/1iMPQdz8NQFB75HwAPwWdM/story.html"><em>Boston Globe</em></a></figcaption>
    
  </figure>


<p><a href="https://en.wikipedia.org/wiki/Game_theory">Game theory</a> offers a powerful framework to study these kinds of coordination problems. Consider this classic problem: if two people need to meet in London on a particular day but, for whatever reason, can&rsquo;t coordinate the time and place, they&rsquo;re likely to choose noon at <a href="https://en.wikipedia.org/wiki/Big_Ben">Big Ben</a>. This becomes what game theorists call a <a href="https://en.wikipedia.org/wiki/Focal_point_%28game_theory%29"><em>focal point</em></a>, also known as a <em>Schelling point</em>—a natural default that everyone can assume without discussion.</p>
<p>The process-per-test model is powerful because <strong>it serves as the Schelling point</strong>, the Big Ben, of how to run tests. Just like it&rsquo;s commonly agreed that the Great Clock of Westminster is the most well-known landmark in London, software developers have all agreed that processes form natural fault boundaries. The process-per-test model forms a universal protocol that everyone in the ecosystem can rely on without explicit coordination.</p>
<h2 id="the-benefits-of-separate-processes">The benefits of separate processes</h2>
<p>This game-theoretic lens provides fresh insight into the benefits of process-per-test:</p>
<ul>
<li>
<p><strong>Test authors and libraries don&rsquo;t need to coordinate global state usage.</strong> There are a surprising number of real-world use cases that <em>require</em> per-test isolation—particularly integration tests against global state:</p>
<ul>
<li>Tests against a graphical API like <a href="https://www.khronos.org/egl">EGL</a>, which only allows one thread in a process to create a GPU context. <a href="https://github.com/gfx-rs/wgpu"><code>wgpu</code></a> uses nextest for this reason</li>
<li>Tests that <a href="https://nexte.st/docs/configuration/extra-args">must be run on the main thread</a> of their corresponding process, like tests against certain macOS frameworks</li>
<li>Tests that require altering environment variables or other global context</li>
</ul>
<p>Even if particular tests don&rsquo;t <em>require</em> per-test isolation, they may often <em>benefit</em> from it. For example, singletons become separated per-test. (You may argue that singletons aren&rsquo;t desirable, or should not be used for tests—but instead, think of per-test isolation as a principled, zero-coordination solution to this class of issues.)</p>
</li>
<li>
<p><strong>Internal memory handling doesn&rsquo;t need to be coordinated across tests.</strong> Memory corruption in one test doesn&rsquo;t cause others to behave erratically. One test segfaulting does not take down a bunch of other tests.</p>
</li>
<li>
<p><strong>Test runners don&rsquo;t need a custom protocol.</strong> With process-per-test, test runners can rely on universal OS primitives, like the ability to kill processes. For more discussion, see <a href="/posts/nextest-process-per-test/#the-costs-of-coordination"><em>The costs of coordination</em></a> below.</p>
</li>
</ul>
<p>Process-per-test is not free, though. It&rsquo;s worth acknowledging some costs:</p>
<ul>
<li>
<p>Tests might want to share in-memory state. They might have an in-memory semaphore for rate-limiting, a shared in-memory immutable cache that is primed on first access, or an in-memory server to which tests communicate via message passing. Process-per-test makes these valuable patterns <strong>harder to achieve</strong>: semaphores must <a href="https://nexte.st/docs/configuration/test-groups">be managed by</a> the test runner, in-memory state must be stored on disk, and shared servers must live out-of-process. Adapting existing tests to process-per-test may require a significant engineering effort.</p>
</li>
<li>
<p>Another downside that&rsquo;s noticeable in some situations is process creation <strong>performance</strong>. Creating processes is very fast on Linux and most other Unix-like systems. It is quite slow on Windows, though, and it can be slow on macOS <a href="https://nexte.st/docs/installation/macos#gatekeeper">if anti-malware protections are interfering</a>.</p>
<p>How much this is an issue in practice depends on how long individual tests take to run. For many projects on Windows, nextest is overall faster than <code>cargo test</code> anyway because it <a href="https://nexte.st/docs/design/how-it-works">handles long-pole tests</a> better.</p>
</li>
</ul>
<h2 id="the-costs-of-coordination">The costs of coordination</h2>
<p>Process-per-test enables rich lifecycle management with zero coordination via universal OS primitives. What kinds of coordination would enable shared-process tests?</p>
<p>Here&rsquo;s a partial list of operations that nextest performs using OS primitives:</p>
<ol>
<li><strong>Start a test.</strong> In particular, start a test at a specific moment, and not before then; and also, do not start a test if it is filtered out or the run is cancelled. With a process-per-test model, this is natural: start the test process.</li>
<li><strong>Know when a test is done, as it&rsquo;s done.</strong> With process-per-test, wait for the process to exit.</li>
<li><strong>Measure test times.</strong> Not just after a test is done, but while it is running. With process-per-test, use wall-clock time.</li>
<li><strong>Terminate tests that timed out.</strong> In particular, one test timing out should not cause others to be killed. With process-per-test, this is <a href="https://nexte.st/docs/design/architecture/signal-handling">solved</a> via signals on Unix and job objects on Windows.</li>
<li><strong>Retry failed tests.</strong> This is an extension to point 1: retrying tests, and marking tests as flaky if they succeeded later.</li>
<li><strong>Cancel tests on an input.</strong> On a test failure or receiving a signal, the test runner may decide to leave tests running but no longer schedule new ones, or to cancel existing tests.</li>
<li><strong>Gather test output.</strong> With process-per-test, nextest can read standard output and standard error for the process.</li>
</ol>
<p>Compare this to <code>cargo test</code>&rsquo;s shared-process model. <code>cargo test</code> works by embedding a small runner into each test binary which manages rate-limiting and filtering.</p>
<ul>
<li>
<p>However, as of early 2025, each runner works in isolation: there is no global orchestrator that can schedule tests across multiple binaries.</p>
</li>
<li>
<p>There is also no way to configure timeouts for tests, or to cancel some (but not all) running tests. For more, see <a href="/posts/nextest-process-per-test/#appendix">the appendix</a>.</p>
</li>
<li>
<p>If a test takes down the process, all of the other tests in the binary are effectively cancelled as well. In this case, getting a full accounting of all tests and what happened to them is extremely difficult. <a href="https://nexte.st/docs/machine-readable/junit">JUnit reports</a> wouldn&rsquo;t be nearly as valuable if they were missing many tests!</p>
</li>
</ul>
<p>Effective and robust management of tests in a shared-process model requires coordination between three entities:</p>
<ul>
<li><strong>The test runner:</strong> There must be an overall orchestrator, and for robustness it must live in a separate process from any of the test binaries.</li>
<li><strong>A component within the test binary:</strong> These requirements preclude any kind of one-shot model where the list of tests is provided upfront and run to completion. Instead, there must be an in-memory component that&rsquo;s dynamic and event-driven. There needs to be a rich, bidirectional communication protocol between the test runner and this component. (Simpler protocols like <a href="https://docs.rs/jobserver">the GNU make jobserver</a> can do rate-limiting, but not cancellation, retries, or time measurement.)</li>
<li><strong>The test itself:</strong> For cancellation in particular, the test must cooperate. For more on why, see <a href="/posts/nextest-process-per-test/#appendix">the appendix</a>.</li>
</ul>
<p>This represents a <em>lot</em> of extra work! Not just the technical kind (though that is certainly quite involved), but also the coordination kind: building support for such a protocol and getting developers to adopt it. And it would still not solve every problem. (For example, segfaults would still take down the whole process.)</p>
<h2 id="conclusion">Conclusion</h2>

  <figure class="floatright" >
    
    <img src="/images/big-ben.jpg"  alt="Photo of the clock on the Great Clock of Westminster, also known as Big Ben."   />
    
    
      <figcaption class="center" >Big Ben, the most recognizable landmark in London. <a href="https://unsplash.com/photos/big-ben-london-MdJq0zFUwrw"><em>Unsplash / Henry Be</em></a></figcaption>
    
  </figure>


<p>There are many technical benefits to the process-per-test model, but the biggest benefit is in (the lack of) coordination: the process-per-test model acts as a Schelling point that all participants can agree on by default. This is the key reason that nextest commits to this model being the default in perpetuity.</p>
<p>Nevertheless, we&rsquo;re excited to see developments in this space. We&rsquo;ll consider opt-ins for newer patterns that can deliver feature-rich and reliable test running at scale like nextest does today.</p>
<hr>
<h2 id="appendix">Appendix: thread cancellation is hard</h2>
<p>Many nextest users rely on its ability to cancel running tests due to timeouts. This section talks about why terminating threads is much more difficult than terminating processes.</p>
<ul>
<li>With process-per-test, nextest sends a <code>SIGTERM</code> signal to the test process, and if it doesn&rsquo;t exit within 10 seconds, a <code>SIGKILL</code>.</li>
<li>With a shared-process model, assuming each test is in a separate thread, a timeout would require the thread to be killed rather than the process.</li>
</ul>
<p>Unlike process termination, though, thread termination is inherently hazardous. The Windows <a href="https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminatethread"><code>TerminateThread</code></a> function explicitly warns against ever using it:</p>
<blockquote>
<p>TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination.</p>
</blockquote>
<p>With <code>TerminateThread</code> and the POSIX equivalent, <a href="https://man7.org/linux/man-pages/man3/pthread_cancel.3.html"><code>pthread_cancel</code></a>, the behavior of synchronization primitives like <code>std::sync::Mutex</code> is undefined. Certainly, Rust mutexes will not be marked <a href="https://doc.rust-lang.org/beta/std/sync/struct.Mutex.html#poisoning">poisoned</a>. It&rsquo;s possible (even likely) that mutexes are held forever and never released. Thread-killing operations would be devastating to test reliability.</p>
<p>Practically speaking, the only reliable way to cancel synchronous Rust code is with some kind of cooperation from the test itself. This could mean the test checking a flag every so often, or by the test regularly calling into a library that panics with a special payload on cancellation.</p>
<p>With async Rust, cancellation is <a href="https://docs.rs/tokio/latest/tokio/task/struct.JoinHandle.html#method.abort">somewhat easier</a>, because yield points are cancellation points. However, in an eerie similarity with <code>pthread_cancel</code>, <a href="https://docs.rs/tokio/latest/tokio/sync/struct.Mutex.html">Tokio mutexes</a> are <a href="https://docs.rs/cancel-safe-futures/0.1/cancel_safe_futures/sync/struct.RobustMutex.html"><em>also</em> not marked poisoned</a> on a task or future cancellation<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<p>It can be illuminating to contrast process termination with thread termination:</p>
<ul>
<li>Threads do not generally expect other threads to just disappear without warning, but processes are expected to sometimes exit abnormally.</li>
<li>Synchronization across processes is possible via functions like <a href="http://linux.die.net/man/2/flock"><code>flock</code></a>, but is often advisory and generally uncommon. Code that uses these tools is generally prepared to encounter protected data in an invalid state.</li>
<li>Cross-process mutexes are quite rare (message-passing is much more common), and shared-memory code is usually written with great care.</li>
</ul>
<p>These examples make clear how Schelling points manifest and perpetuate: we&rsquo;ve all generally agreed that other processes might behave in strange ways, but we assume that other threads within our process are going to behave reliably.</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>Based on our experiences with cancellation in other Rust projects, we strongly recommend Tokio mutexes be treated as a feature of last resort.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/nextest" term="nextest" label="nextest"/><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/></entry><entry><title type="html">Open and closed universes</title><link href="https://sunshowers.io/posts/open-closed-universes/?utm_source=atom_feed" rel="alternate" type="text/html"/><link href="https://sunshowers.io/posts/types/?utm_source=atom_feed" rel="related" type="text/html" title="The social consequences of type systems"/><id>https://sunshowers.io/posts/open-closed-universes/</id><published>2021-08-02T00:00:00+00:00</published><updated>2021-08-13T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>How to write types that represent choices.</blockquote>
  <figure class="center" >
    
    <img src="/images/open-closed-walnuts.jpg"  alt="Depiction of walnuts, some cracked open and others not."   />
    
    
  </figure>


<p>Type systems are tools for modeling some aspect of reality. Some types need to represent one of several different choices.</p>
<ul>
<li>Sometimes, all the choices may be known in advance and will likely never change&mdash;this is often called a <em>closed universe</em> of values.</li>
<li>Other times, the set of options will change over time, and the type needs to represent an <em>open universe</em>.</li>
</ul>
<p>How should these different kinds of choices be modeled?</p>
<div class="framed">
<h1 id="aside-this-matters-more-for-libraries-than-for-applications">Aside: this matters more for libraries than for applications</h1>
<p>If you&rsquo;re writing an end-user application, none of this matters. You have full control over all aspects of your code, so use whatever is most convenient. However, if you&rsquo;re writing a library that other developers will use, you may care about API stability and forward compatibility.</p>
</div>
<h1 id="closed-universes-as-enums">Closed universes as enums</h1>
<p>A simple example is the <code>Option</code> type in Rust, or its analogs like <code>Maybe</code> in Haskell. <code>Option</code> is <a href="https://doc.rust-lang.org/std/option/enum.Option.html">defined as</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">enum</span> <span class="nb">Option</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nb">None</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nb">Some</span><span class="p">(</span><span class="n">T</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Part of the definition of <code>Option</code> is that it can only be either <code>Some</code> or <code>None</code>. This is never going to change, so <code>Option</code> values form a closed universe. A simple enum is appropriate for such cases.</p>
<p>The main advantage of this approach is that it minimizes the burden on consumers: they know that an <code>Option</code> can only be <code>Some</code> or <code>None</code>, and only have to care about such cases.</p>
<p>The main disadvantage is that any sort of new addition would constitute API breakage. You&rsquo;re expanding a universe you formerly assumed to be closed, and everyone who depends on you needs to be made aware of this.</p>
<h1 id="semi-open-and-fully-open-universes">Semi-open and fully open universes</h1>
<p>As a library author, you may run into two kinds of open universes:</p>
<ol>
<li>Open with respect to the library itself: new choices can be defined by library authors in the future, but not by downstream consumers. I&rsquo;m going to call these <em>semi-open universes</em>.</li>
<li>Open with respect to users of the library: new choices can be defined by both library authors and consumers. These might be described as <em>fully open universes</em>.</li>
</ol>
<h2 id="non-exhaustive-enums">Non-exhaustive enums</h2>
<p>If you&rsquo;re trying to model semi-open universes, consider using a non-exhaustive enum.</p>
<p>For example, you may have an error type in a library that bubbles up other errors. One way to write this is:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">enum</span> <span class="nc">Error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Io</span><span class="p">(</span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Json</span><span class="p">(</span><span class="n">serde_json</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The problem with this approach is that errors typically do not form a closed universe. A future version may depend on some other library, for example <code>toml</code>, and you&rsquo;d have to add a new option to the <code>Error</code> type:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">enum</span> <span class="nc">Error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Io</span><span class="p">(</span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Json</span><span class="p">(</span><span class="n">serde_json</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Toml</span><span class="p">(</span><span class="n">toml</span>::<span class="n">de</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Anyone matching on the <code>Error</code> type would have to handle this new, additional case&hellip; unless they&rsquo;ve specified a <a href="https://doc.rust-lang.org/book/ch18-03-pattern-syntax.html#ignoring-values-in-a-pattern">wildcard pattern</a>.</p>
<p>Some programming languages allow library developers to <em>force</em> consumers to specify a wildcard pattern. In Rust, you can annotate a type with <code>#[non_exhaustive]</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[non_exhaustive]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">enum</span> <span class="nc">Error</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Io</span><span class="p">(</span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Json</span><span class="p">(</span><span class="n">serde_json</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Toml</span><span class="p">(</span><span class="n">toml</span>::<span class="n">de</span>::<span class="n">Error</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Anyone matching against the error type would have to add an additional wildcard pattern:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">match</span><span class="w"> </span><span class="n">err</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Io</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Json</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Toml</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// The match statement will fail to compile without this last wildcard pattern
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">_</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This ensures that the library developer can continue to add more options over time without causing breakages. The main downside is that all consumers need to handle the additional wildcard pattern and do something sensible with it<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<h2 id="many-strings">Many strings</h2>
<p>The non-exhaustive enum approach is best suited for semi-open universes. What about situations where consumers should have the ability to add new choices?</p>
<p>The best way is often to use a string, or a newtype wrapper around it. For example, one may wish to represent a choice between computer architectures such as <code>i386</code>, <code>amd64</code>, and <code>aarch64</code>. More architectures may be added in the future, too, but often the library just needs to pass the architecture down to some other tool without doing much processing on it. One way to represent this is<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">borrow</span>::<span class="n">Cow</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Architecture</span><span class="p">(</span><span class="n">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">name</span>: <span class="nc">impl</span><span class="w"> </span><span class="nb">Into</span><span class="o">&lt;</span><span class="n">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;&gt;</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Architecture</span><span class="p">(</span><span class="n">name</span><span class="p">.</span><span class="n">into</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Well-known strings can be represented as constants:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new_const</span><span class="p">(</span><span class="n">name</span>: <span class="kp">&amp;</span><span class="nb">&#39;static</span> <span class="kt">str</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Architecture</span><span class="p">(</span><span class="n">Cow</span>::<span class="n">Borrowed</span><span class="p">(</span><span class="n">name</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">I386</span>: <span class="nc">Self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Architecture</span>::<span class="n">new_const</span><span class="p">(</span><span class="s">&#34;i386&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="no">AMD64</span>: <span class="nc">Self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Architecture</span>::<span class="n">new_const</span><span class="p">(</span><span class="s">&#34;amd64&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="no">AARCH64</span>: <span class="nc">Self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Architecture</span>::<span class="n">new_const</span><span class="p">(</span><span class="s">&#34;aarch64&#34;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This allows known choices to not be <a href="http://wiki.c2.com/?StringlyTyped">stringly-typed</a>, and currently-unknown ones to be used without having to change the library.</p>
<p>The format doesn&rsquo;t have to be restricted to strings. For example, an <code>Architecture</code> may wish to also carry its <a href="https://en.wiktionary.org/wiki/bitness">bitness</a>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">name</span>: <span class="nc">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">bitness</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">name</span>: <span class="nc">impl</span><span class="w"> </span><span class="nb">Into</span><span class="o">&lt;</span><span class="n">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;&gt;</span><span class="p">,</span><span class="w"> </span><span class="n">bitness</span>: <span class="kt">usize</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span>: <span class="nc">name</span><span class="p">.</span><span class="n">into</span><span class="p">(),</span><span class="w"> </span><span class="n">bitness</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new_const</span><span class="p">(</span><span class="n">name</span>: <span class="kp">&amp;</span><span class="nb">&#39;static</span> <span class="kt">str</span><span class="p">,</span><span class="w"> </span><span class="n">bitness</span>: <span class="kt">usize</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">name</span>: <span class="nc">Cow</span>::<span class="n">Borrowed</span><span class="p">(</span><span class="n">name</span><span class="p">),</span><span class="w"> </span><span class="n">bitness</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">I386</span>: <span class="nc">Self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Architecture</span>::<span class="n">new_const</span><span class="p">(</span><span class="s">&#34;i386&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">32</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="no">AMD64</span>: <span class="nc">Self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Architecture</span>::<span class="n">new_const</span><span class="p">(</span><span class="s">&#34;amd64&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">64</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="no">AARCH64</span>: <span class="nc">Self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Architecture</span>::<span class="n">new_const</span><span class="p">(</span><span class="s">&#34;aarch64&#34;</span><span class="p">,</span><span class="w"> </span><span class="mi">64</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>In general, this works best for choices that are:</p>
<ul>
<li>used as input types to a library that doesn&rsquo;t do much processing on them, and</li>
<li>the data associated with each choice doesn&rsquo;t vary by choice.</li>
</ul>
<div class="framed">
<h2 id="a-pattern-to-avoid-unknown-or-other-variants">A pattern to avoid: <code>Unknown</code> or <code>Other</code> variants</h2>
<p>One pattern I&rsquo;ve seen is to use another variant that represents an unknown value. For example:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">enum</span> <span class="nc">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">I386</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Amd64</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Aarch64</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">Unknown</span><span class="p">(</span><span class="n">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The problem with this approach is that it becomes very hard to define equality and comparisons on these types. Are <code>Architecture::I386</code> and <code>Architecture::Unknown(&quot;i386&quot;)</code> the same, or are they different? In practice, this becomes tricky very quickly.</p>
<p>In general, this approach is best avoided.</p>
</div>
<h2 id="traits">Traits</h2>
<p>The final, most general approach is to use traits, typeclasses or other interface mechanisms. For example, architectures can be represented as a trait in Rust:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">name</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">I386</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">I386</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">name</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Cow</span>::<span class="n">Borrowed</span><span class="p">(</span><span class="s">&#34;i386&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Amd64</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">Amd64</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">name</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Cow</span>::<span class="n">Borrowed</span><span class="p">(</span><span class="s">&#34;amd64&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Aarch64</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">Aarch64</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">name</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Cow</span>::<span class="n">Borrowed</span><span class="p">(</span><span class="s">&#34;aarch64&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>Then, other library methods could use:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="c1">// This could take the architecture as a type parameter as well.
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">process_architecture</span><span class="p">(</span><span class="n">arch</span>: <span class="kp">&amp;</span><span class="nc">dyn</span><span class="w"> </span><span class="n">Architecture</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">arch</span><span class="p">.</span><span class="n">name</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>And custom architectures can be defined as:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">MyArchitecture</span><span class="p">(</span><span class="nb">String</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyArchitecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">name</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">Cow</span>::<span class="n">Owned</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="mf">0.</span><span class="n">clone</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>This also works for output types such as returned errors.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">read_u64</span><span class="p">()</span><span class="w"> </span>-&gt; <span class="nb">Result</span><span class="o">&lt;</span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="nb">Box</span><span class="o">&lt;</span><span class="k">dyn</span><span class="w"> </span><span class="n">Error</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">&#39;static</span><span class="o">&gt;&gt;</span><span class="w">
</span></span></span></code></pre></div><p>You can try and downcast the error into a more specific type:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">match</span><span class="w"> </span><span class="n">read_u64</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nb">Ok</span><span class="p">(</span><span class="n">value</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nb">Err</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c1">// The function might have returned an IO error.
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">match</span><span class="w"> </span><span class="n">err</span><span class="p">.</span><span class="n">downcast</span>::<span class="o">&lt;</span><span class="n">std</span>::<span class="n">io</span>::<span class="n">Error</span><span class="o">&gt;</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nb">Ok</span><span class="p">(</span><span class="n">io_error</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nb">Err</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="c1">// Could be a TOML error.
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="k">match</span><span class="w"> </span><span class="n">err</span><span class="p">.</span><span class="n">downcast</span>::<span class="o">&lt;</span><span class="n">toml</span>::<span class="n">de</span>::<span class="n">Error</span><span class="o">&gt;</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="nb">Ok</span><span class="p">(</span><span class="n">toml_error</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="c1">// ...,
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="nb">Err</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                        </span><span class="c1">// Some other sort of error. Handle appropriately.
</span></span></span><span class="line"><span class="cl"><span class="w">                    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>but this turns a formerly compile-time check into a runtime one. A common bug that happens with this approach is version skew: for example, let&rsquo;s say the library had originally been using <code>toml 0.4</code>. In a minor release, it upgrades to <code>toml 0.5</code> and return errors from the new version, while the consumer continues to downcast to the old <code>toml 0.4</code> version. Such cases will fail silently<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>.</p>
<h3 id="traits-vs-many-strings">Traits vs many strings</h3>
<p>Traits are similar to the many-strings approach because they both model fully open universes. However, there are some tradeoffs between the two.</p>
<p>One benefit of traits is that they offer greater implementation flexibility: trait methods allow for custom behavior on
user-defined types.</p>
<p>A downside is that traits make each choice in an open universe a <em>type</em>, not a <em>value</em>, so operations like equality become harder to define. It isn&rsquo;t impossible, though: the trait could require a <code>key</code> method which returns a many-strings type.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="cp">#[derive(Eq, PartialEq)]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">struct</span> <span class="nc">ArchitectureKey</span><span class="p">(</span><span class="k">pub</span><span class="w"> </span><span class="n">Cow</span><span class="o">&lt;</span><span class="nb">&#39;static</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="o">&gt;</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Architecture</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">fn</span> <span class="nf">key</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">ArchitectureKey</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The contents of <code>ArchitectureKey</code> can then be used for equality checks.</p>
<p>Another downside is ergonomics: traits and interfaces are more awkward than values. A sufficiently complex trait may even require some sort of code generation to use effectively. Rust has <a href="https://doc.rust-lang.org/book/ch19-06-macros.html#declarative-macros-with-macro_rules-for-general-metaprogramming">declarative</a> and <a href="https://blog.rust-lang.org/2018/12/21/Procedural-Macros-in-Rust-2018.html">procedural</a> macros to help, which work okay but add even more complexity on top.</p>
<p>Whether strings are sufficient or full-blown traits are required is a case-by-case decision. In general, the many-strings approach is much simpler and is powerful enough for many kinds of choices; traits provide greater power and are always worth keeping in mind, though.</p>
<h3 id="sealed-traits">Sealed traits</h3>
<p>A trait can also model a semi-open universe if you <a href="https://rust-lang.github.io/api-guidelines/future-proofing.html#sealed-traits-protect-against-downstream-implementations-c-sealed"><em>seal</em> it</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">MyTrait</span>: <span class="nc">private</span>::<span class="n">Sealed</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">MyType</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">impl</span><span class="w"> </span><span class="n">MyTrait</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">mod</span> <span class="nn">private</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Sealed</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">impl</span><span class="w"> </span><span class="n">Sealed</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="k">super</span>::<span class="n">MyType</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>The library itself may add new implementations of the trait, but downstream consumers cannot because they do not have access to <code>Sealed</code><sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>.</p>
<h1 id="conclusion">Conclusion</h1>

  <figure class="floatright" >
    
    <img src="/images/infinity-and-beyond.jpg"  alt="Stylized representation of Buzz Lightyear from Pixar&#39;s Toy Story, with the text &#34;To infinity... and beyond!&#34; on it. Print by Evan Troutt, courtesy Gallery1988."   />
    
    
      <figcaption class="center" >Print by Evan Troutt, courtesy <a href="https://nineteeneightyeight.com/products/evan-troutt-to-infinity-and-beyond-print?variant=21023513247803">Gallery1988</a>.</figcaption>
    
  </figure>


<p>Type systems are often used to represent one of several choices. The choices can either be fixed (closed universe), or might continue to change over time (open universe). These are different situations that usually need to be handled in different ways.</p>
<p>Hope this helps!</p>
<hr>
<p>The idea of writing this post arose from several conversations with members of the Rust community. Thanks to <a href="https://twitter.com/munin">Munin</a>, <a href="https://twitter.com/yaahc_">Jane Lusby</a>, <a href="https://twitter.com/ManishEarth">Manish Goregaokar</a>, <a href="https://twitter.com/mgattozzi">Michael Gattozzi</a> and <a href="https://twitter.com/slurpsmadrips">Izzy Muerte</a> for reviewing drafts of this post.</p>
<hr>
<p><em>Updated 2021-08-04: edited some text, and corrected the footnote about <code>Cow</code> to indicate that it is required.</em></p>
<p><em>Updated 2021-08-13: added a note about more complex versions of the many-strings pattern, expanded on the difference between it and traits, and expanded on what it would mean to unify sealed traits and non-exhaustive enums. Thanks to <a href="https://twitter.com/ManishEarth">Manish</a> for the inspiration.</em></p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>I wish that Rust had a way to lint, but not fail to compile, on missing patterns in a non-exhaustive match.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>This implementation uses the <a href="https://doc.rust-lang.org/std/borrow/enum.Cow.html"><code>Cow</code></a> type, which with the <code>'static</code> lifetime stands for either a borrowed string that lasts the duration of the entire program&mdash;typically one embedded in the binary&mdash;or an owned <code>String</code> created at runtime. If this were just a <code>String</code>, then the <code>const</code> implementation below wouldn&rsquo;t be possible because <code>const fn</code>s can&rsquo;t allocate memory.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>With static types, this sort of version skew error is usually caught at compile time. With dynamic casting it can only be caught at runtime.</p>
<p>You might argue that if the error type comes from a public dependency, an incompatible upgrade to the dependency constitutes a semantic version breakage. However, this is a bit of a boundary case; the library authors might disagree with you and say that their stability guarantees don&rsquo;t extend to downcasting. Semantic versioning is primarily a low-bandwidth way for library developers and consumers to communicate about relative burdens in the event of a library upgrade.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>Given that both sealed traits and non-exhaustive enums model the same kinds of semi-open universes in fairly similar fashions, a direction languages can evolve in is to unify the two. Scala has already done <a href="https://contributors.scala-lang.org/t/proposal-for-enumerations-in-scala/4020">something similar</a>.</p>
<p>Rust could gain several features that bring non-exhaustive enums and sealed traits incrementally closer to each other:</p>
<ul>
<li>Allow <code>match</code> statements to match sealed traits if a wildcard match is specified.</li>
<li>Make enum variants their own types rather than just <a href="https://wiki.haskell.org/Constructor#Data_constructor">data constructors</a>.</li>
<li>Automatically convert the return values of functions like <code>fn foo() -&gt; impl Trait</code> into enums if necessary. (The trait doesn&rsquo;t need to be sealed in this case, because the enum would be anonymous and matching on its variants wouldn&rsquo;t be possible.)</li>
<li>Inherent methods on an enum would need to be &ldquo;lifted up&rdquo; to become trait methods. As of Rust 1.54, there are several kinds of methods that can be specified on enums but not on traits. Features like <a href="https://github.com/rust-lang/rfcs/blob/master/text/2071-impl-trait-existential-types.md">existential types</a> help narrow that gap.</li>
</ul>
&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/types" term="types" label="types"/><category scheme="https://sunshowers.io/tags/rust" term="rust" label="rust"/></entry></feed>