<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us"><generator uri="https://gohugo.io/" version="0.170.0">Hugo</generator><title type="html">Illumos on sunshowers</title><subtitle type="html">thoughts and ideas</subtitle><link href="https://sunshowers.io/tags/illumos/" rel="alternate" type="text/html" title="html"/><link href="https://sunshowers.io/tags/illumos/index.xml" rel="alternate" type="application/rss+xml" title="rss"/><link href="https://sunshowers.io/tags/illumos/atom.xml" rel="self" type="application/atom+xml" title="atom"/><updated>2026-11-15T23:03:09+00:00</updated><rights>© Rain 2020-present. Licensed under CC BY-NC-SA 4.0 unless marked otherwise.</rights><author><name>Rain</name></author><id>https://sunshowers.io/tags/illumos/</id><entry><title type="html">Debugging a rustc segfault on illumos</title><link href="https://sunshowers.io/posts/rustc-segfault-illumos/?utm_source=atom_feed" rel="alternate" type="text/html"/><id>https://sunshowers.io/posts/rustc-segfault-illumos/</id><published>2024-08-04T00:00:00+00:00</published><updated>2024-08-05T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>Using <code>mdb</code> and <code>pmap</code> to find the cause of a crash.</blockquote><div class="sectioned-post">
<p>At <a href="https://oxide.computer/">Oxide</a>, we use <a href="https://github.com/oxidecomputer/helios/">Helios</a> as
the base OS for the cloud computers we sell. Helios is a distribution of
<a href="https://illumos.org/">illumos</a>, a Unix-based operating system descended from Solaris.</p>
<p>As someone who learned illumos on the job, I&rsquo;ve been really impressed by the powerful debugging
tools it provides. I had a chance to use some of them recently to track down a <a href="https://en.wikipedia.org/wiki/Segmentation_fault">segmentation
fault</a> in the Rust compiler, with the help of
several of my colleagues. I learned a lot from the process, and I thought I&rsquo;d write about it!</p>
<p>I&rsquo;m writing this post for an audience of curious technologists who aren&rsquo;t necessarily familiar with
systems work. If you&rsquo;re an experienced systems developer, parts of it are likely familiar to
you&mdash;feel free to skip over them.</p>
<h2 id="the-crash">The crash</h2>
<p>A couple of weeks ago, I wanted to make a change to the Rust standard library on illumos. I logged
into my illumos box and cloned the <a href="https://github.com/rust-lang/rust">Rust repository</a> (revision
<code>2d5a628</code>). Following the <a href="https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html">setup
instructions</a>, I configured the <code>rustc</code> build system with the <code>library</code> build profile.</p>
<p>When I went to run <code>./x.py check</code>, I saw an error with the following output:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> ./x.py check
</span></span><span class="line"><span class="cl"><span class="go">Checking stage0 cranelift (x86_64-unknown-illumos)
</span></span></span><span class="line"><span class="cl"><span class="go">    Checking cranelift-codegen v0.109.0
</span></span></span><span class="line"><span class="cl"><span class="go">rustc exited with signal: 11 (SIGSEGV) (core dumped)
</span></span></span><span class="line"><span class="cl"><span class="go">error: could not compile `cranelift-codegen` (lib)
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="go">Caused by:
</span></span></span><span class="line"><span class="cl"><span class="go">  process didn&#39;t exit successfully: ...
</span></span></span><span class="line"><span class="cl"><span class="go">Build completed unsuccessfully in 0:00:03
</span></span></span></code></pre></div><p>Quite concerning! Like any good technologist I tried running the command again. But the segfault
seemed to be completely deterministic: the program would crash while compiling <code>cranelift-codegen</code>
every time.</p>
<p>Coincidentally, we had our <abbr title="Every 2 weeks">fortnightly</abbr> &ldquo;Rust @ Oxide&rdquo; virtual meetup at around that time. There wasn&rsquo;t much to discuss there, so we turned that meeting into a debugging session. (I love how my coworkers get excited about debugging strange issues.)</p>
<h2 id="background-the-bootstrap-process">Background: the bootstrap process</h2>

  <figure class="floatright" >
    
    <img src="/images/bootstrap-stages.png"  alt="A flowchart to indicate stages of compilation. For a full description, see the link for &#34;a series of stages&#34;."   />
    
    
      <figcaption class="center" >Rust compiler build stages.</figcaption>
    
  </figure>


<p>Like the compilers for many other languages, the Rust compiler is written in the language it is
intending to compile (in this case, Rust). In other words, the Rust compiler is
<a href="https://en.wikipedia.org/wiki/Self-hosting_%28compilers%29#Compilers"><em>self-hosting</em></a>.</p>
<p>Any self-hosting compiler needs to answer the question: how in the world do you compile the compiler
if you don&rsquo;t already have a working compiler? This is known as the <a href="https://en.wikipedia.org/wiki/Bootstrapping_%28compilers%29"><em>bootstrapping
problem</em></a>. There are several ways to
address the problem, but the two most common are:</p>
<ol>
<li>
<p><strong>Use the previous version of the compiler.</strong> In other words, use version N-1 of the compiler to
compile version N. For example, use Rust 1.75 to compile Rust 1.76.</p>
<details>
<summary>From where do you begin, though?</summary>
<p>The earliest versions of Rust were written in Ocaml. So if
you&rsquo;re spinning up Rust on a brand new platform and have an Ocaml compiler available, you can
actually start <a href="https://github.com/rust-lang/rust/tree/ef75860a0a72f79f97216f8aaa5b388d98da6480/src/boot">from
there</a>
and effectively create your own lineage of compilers.</p>
<p>There are also implementations of Rust in other languages, like <a href="https://github.com/thepowersgang/mrustc"><code>mrustc</code></a> in C++, which can be used to build some (typically pretty old) version of the compiler. Interestingly, these other implementations don&rsquo;t need to be perfect&mdash;for example, since they&rsquo;re only used to compile code that&rsquo;s known to be valid, they don&rsquo;t need to handle errors well. That&rsquo;s a large chunk of the complexity of a real compiler.</p>
</details>
</li>
<li>
<p><strong>Cross-compile from another platform.</strong> As a shortcut, if you have a way to cross-compile code
from another platform, you can use that to set up the initial compiler. This is the most common
method for setting up Rust on a new platform. (But note that method 1 must be used on at least
one platform.)</p>
</li>
</ol>
<p>While bootstrapping from the previous version of Rust, the toolchain follows <a href="https://rustc-dev-guide.rust-lang.org/building/bootstrapping/what-bootstrapping-does.html#stages-of-bootstrapping">a series of
stages</a>, ranging from <em>stage 0</em> to <em>stage 2</em>.</p>
<p>In our case, since we&rsquo;re working with the standard library we&rsquo;re only concerned with <em>stage 0</em>: the
standard library compiled with the previous version of <code>rustc</code>. That is the build process that crashed.</p>
<h2 id="orienting-ourselves">Orienting ourselves</h2>
<p>The first thing to find is the version of <code>rustc</code> that&rsquo;s crashing. There are a few ways to find the compiler, but a simple <code>find</code> command works well:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> find . -name rustc
</span></span><span class="line"><span class="cl"><span class="go">./compiler/rustc
</span></span></span><span class="line"><span class="cl"><span class="go">./src/doc/rustc
</span></span></span><span class="line"><span class="cl"><span class="go">./build/x86_64-unknown-illumos/stage0/bin/rustc
</span></span></span></code></pre></div><p>This command finds <code>rustc</code> at <code>./build/x86_64-unknown-illumos/stage0/bin/rustc</code>. Let&rsquo;s ask it for its version:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> ./build/x86_64-unknown-illumos/stage0/bin/rustc -Vv
</span></span><span class="line"><span class="cl"><span class="go">rustc 1.80.0-beta.1 (75ac3b633 2024-06-10)
</span></span></span><span class="line"><span class="cl"><span class="go">binary: rustc
</span></span></span><span class="line"><span class="cl"><span class="go">commit-hash: 75ac3b6331873133c4f7a10f2252afd6f3906c6a
</span></span></span><span class="line"><span class="cl"><span class="go">commit-date: 2024-06-10
</span></span></span><span class="line"><span class="cl"><span class="go">host: x86_64-unknown-illumos
</span></span></span><span class="line"><span class="cl"><span class="go">release: 1.80.0-beta.1
</span></span></span><span class="line"><span class="cl"><span class="go">LLVM version: 18.1.7
</span></span></span></code></pre></div><p>Can the bug be reproduced independently of the Rust toolchain? The toolchain does all sorts of
non-standard things, so it&rsquo;s worth checking. The output says <code>cranelift-codegen v0.109.0</code>, so let&rsquo;s try building that separately. Again, there are a few ways to do this, but the easiest is to make a simple Cargo project that depends on the crate.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">package</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">name</span> <span class="p">=</span> <span class="s2">&#34;cranelift-codegen-test&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nx">version</span> <span class="p">=</span> <span class="s2">&#34;0.1.0&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nx">edition</span> <span class="p">=</span> <span class="s2">&#34;2021&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">dependencies</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">cranelift-codegen</span> <span class="p">=</span> <span class="s2">&#34;=0.109.0&#34;</span>
</span></span></code></pre></div><p>And then run <code>cargo build</code>. I didn&rsquo;t have rustc 1.80.0 beta 1 on the machine, so I tried with the 1.80.0 release:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> cargo +1.80.0 build
</span></span><span class="line"><span class="cl"><span class="go">   Compiling cranelift-codegen v0.109.0
</span></span></span><span class="line"><span class="cl"><span class="go">error: could not compile `cranelift-codegen` (lib)
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="go">Caused by:
</span></span></span><span class="line"><span class="cl"><span class="go">  process didn&#39;t exit successfully: `/home/rain/.rustup/toolchains/1.80.0-x86_64-unknown-illumos/bin/rustc ...` (signal: 11, SIGSEGV: invalid memory reference)
</span></span></span></code></pre></div><p>Yep, it crashes in the same spot.</p>
<p>This is a minimal-enough example, so let&rsquo;s work with this.</p>
<h2 id="finding-the-core-file">Finding the core file</h2>

  <figure class="floatright" >
    
    <img src="/images/dumpster-fire.jpg"  alt="A cute cartoon depiction of a dumpster fire with a smiley face."   />
    
    
      <figcaption class="center" >Not this kind of dump! (<a href="https://www.pinterest.com/pin/dumpster-fire-2021-by-bywayanyone--227502218670597019/">Pinterest</a>)</figcaption>
    
  </figure>


<p>When a program crashes, systems are typically configured to generate a <a href="https://en.wikipedia.org/wiki/Core_dump">core dump</a>, also known as a core file. The first step while debugging any crash is to ensure that core dumps are generated, and then to find one to examine it.</p>
<p>On illumos, many of the system-level administration tools are called <code>&lt;something&gt;adm</code>. The tool for managing core files is called <code>coreadm</code>. Let&rsquo;s run that:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> coreadm
</span></span><span class="line"><span class="cl"><span class="go">     global core file pattern:
</span></span></span><span class="line"><span class="cl"><span class="go">     global core file content: default
</span></span></span><span class="line"><span class="cl"><span class="go">       init core file pattern: core
</span></span></span><span class="line"><span class="cl"><span class="go">       init core file content: default
</span></span></span><span class="line"><span class="cl"><span class="go">            global core dumps: disabled
</span></span></span><span class="line"><span class="cl"><span class="go">       per-process core dumps: enabled
</span></span></span><span class="line"><span class="cl"><span class="go">      global setid core dumps: disabled
</span></span></span><span class="line"><span class="cl"><span class="go"> per-process setid core dumps: disabled
</span></span></span><span class="line"><span class="cl"><span class="go">     global core dump logging: disabled
</span></span></span></code></pre></div><p>This suggests that core &ldquo;per-process core dumps&rdquo; are enabled. The lack of a pattern indicates that the defaults are used. Generally, on Unix systems the default is to generate a file named <code>core</code> in the current directory of the crashing process.</p>
<p>A simple <code>ls</code> in our little test project doesn&rsquo;t show a <code>core</code> file, which means that it might be elsewhere. Let&rsquo;s just do a global <code>find</code> for it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> find / -name core -type f
</span></span></code></pre></div><p>This showed a few files on my system, including: <code>~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cranelift-codegen-0.109.0/core</code>. Bingo! That looks like a hit. (Why is it in the registry? Because when compiling a crate, Cargo sets the current working directory of the child <code>rustc</code> process to the crate&rsquo;s directory.)</p>
<p>The next step is to move the file into another directory<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. After doing that, let&rsquo;s start examining it.</p>
<h2 id="examining-the-core-file-registers-and-call-stack">Examining the core file: registers and call stack</h2>
<p>The best way to examine a core file on illumos is with the <a href="https://illumos.org/books/mdb/preface.html">Modular Debugger, <code>mdb</code></a>. <code>mdb</code> is a powerful tool that can be used to inspect the state of both live and dead processes, as well as the kernel itself.</p>
<p>Using <code>mdb</code> with the core file is simple: just run <code>mdb core</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> mdb core
</span></span><span class="line"><span class="cl"><span class="go">Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
</span></span></span><span class="line"><span class="cl"><span class="gp">&gt;</span>
</span></span></code></pre></div><p>The first step is to enable symbol demangling<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>. The command to do that in <code>mdb</code> is <code>$G</code>, so let&rsquo;s run that:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> <span class="nv">$G</span>
</span></span><span class="line"><span class="cl"><span class="go">C++ symbol demangling enabled
</span></span></span></code></pre></div><p>(The output says &ldquo;C++&rdquo;, but illumos&rsquo;s demangler can handle Rust symbols, too.)</p>
<p>Let&rsquo;s look at the <a href="https://en.wikipedia.org/wiki/Processor_register">CPU registers</a> now. A register stores a small amount of data that the CPU can access very quickly. Core files typically have the contents of registers at the time of the crash, which can be very useful for debugging.</p>
<p>In <code>mdb</code>, the command to print out registers is <code>$r</code> or <code>::regs</code>. Here&rsquo;s the output:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> <span class="nv">$r</span>
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rax</span> <span class="o">=</span> 0x000000000fb0d460       %r8  <span class="o">=</span> 0x0000000001000000
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rbx</span> <span class="o">=</span> 0x0000000000000000       %r9  <span class="o">=</span> 0x0000000000000000
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rcx</span> <span class="o">=</span> 0x0000000000000000       %r10 <span class="o">=</span> 0x0000000000000010
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rdx</span> <span class="o">=</span> 0x0000000000000001       %r11 <span class="o">=</span> 0x0000000000000286
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rsi</span> <span class="o">=</span> 0x000000000fb0d3d0       %r12 <span class="o">=</span> 0x0000000000000d96
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rdi</span> <span class="o">=</span> 0xfffffc7fed8e5f30       %r13 <span class="o">=</span> 0x0000000000000000
</span></span><span class="line"><span class="cl"><span class="go">                                %r14 = 0x000000000fb0d3d0
</span></span></span><span class="line"><span class="cl"><span class="go">                                %r15 = 0xfffffc7fed8e6200
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">cs</span> <span class="o">=</span> 0x0053    %fs <span class="o">=</span> 0x0000    %gs <span class="o">=</span> 0x0000
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">ds</span> <span class="o">=</span> 0x004b    %es <span class="o">=</span> 0x004b    %ss <span class="o">=</span> 0x004b
</span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rip</span> <span class="o">=</span> 0xfffffc7fd1adc4bb librustc_driver-86178b5e8d46877c.so<span class="sb">`</span>&lt;rustc_parse::parser::Parser&gt;::parse_path_segment+0x7b
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rbp</span> <span class="o">=</span> 0xfffffc7fed8e6140
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rsp</span> <span class="o">=</span> 0xfffffc7fed8e5c20
</span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">rflags</span> <span class="o">=</span> 0x00010216
</span></span><span class="line"><span class="cl"><span class="go">  id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0
</span></span></span><span class="line"><span class="cl"><span class="go">  status=&lt;of,df,IF,tf,sf,zf,AF,PF,cf&gt;
</span></span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">gsbase</span> <span class="o">=</span> 0x0000000000000000
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">fsbase</span> <span class="o">=</span> 0xfffffc7fee830a80
</span></span><span class="line"><span class="cl"><span class="gp">%</span><span class="nv">trapno</span> <span class="o">=</span> 0xe
</span></span><span class="line"><span class="cl"><span class="go">   %err = 0x6
</span></span></span></code></pre></div><p>All right, there&rsquo;s a lot going on here. A full accounting of the registers on x86-64 is beyond the scope of this post, but if you&rsquo;re interested <a href="https://math.hws.edu/eck/cs220/f22/registers.html">here&rsquo;s a quick summary</a>. The most important registers here are <code>%rip</code>, <code>%rsp</code>, and <code>%rbp</code>. All three of these are 64-bit addresses.</p>

  <figure class="floatright" >
    
    <img src="/images/call-stack.png"  alt="A visual depiction of a call stack, showing three inactive frames plus an active frame."   />
    
    
      <figcaption class="center" >A visual depiction of a call stack.</figcaption>
    
  </figure>


<ul>
<li>
<p><code>%rip</code> is the <strong>instruction pointer</strong>, also known as the <strong>program counter</strong>. <code>%rip</code> is a special register that points to the next instruction to be executed. The CPU uses to keep track of where it is in the program.</p>
</li>
<li>
<p><code>%rsp</code> is the <strong>stack pointer</strong>. The call stack is a region of memory that is used to store function call information and local variables. The stack pointer points to the head of the stack.</p>
<p>Note that on most architectures including x86-64, the stack grows down in memory: when a function is called, a new <em>stack frame</em> is set up and the stack pointer is decremented by however much space the function needs.</p>
</li>
<li>
<p><code>%rbp</code> is the <strong>base pointer</strong>, more commonly known as the <strong>frame pointer</strong>. It points to the base of the current stack frame<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>.</p>
</li>
</ul>
<p>We can also look at the call stack via the <code>$C</code> command. The stack turns out to be enormous (<a href="https://gist.github.com/sunshowers/5edb7207c1e1234b0400bc6517f45b29">full output</a>):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> <span class="nv">$C</span> ! wc -l
</span></span><span class="line"><span class="cl"><span class="go">1493
</span></span></span><span class="line"><span class="cl"><span class="gp">&gt;</span> <span class="nv">$C</span>
</span></span><span class="line"><span class="cl"><span class="go">fffffc7fed8e6140 librustc_driver-86178b5e8d46877c.so`&lt;rustc_parse::parser::Parser&gt;::parse_path_segment+0x7b()
</span></span></span><span class="line"><span class="cl"><span class="go">&lt;... snipped ...&gt;
</span></span></span></code></pre></div><p>(The <code>!</code> is used to send the output to a shell command, in this case one that counts the number of lines.)</p>
<p>It looks like the crash is in the <code>rustc</code> parser. (Notably, the crash is while compiling a crate called <code>cranelift-codegen</code>, which suggests automatic code generation. Generated code often tends to stress the parser in ways that manually written code does not.)</p>
<p>Based on the call stack, it looks like the <code>rustc</code> parser is recursive in nature. A quick Google
search
<a href="https://users.rust-lang.org/t/what-type-of-parser-does-the-rust-compiler-use/71430">confirms</a> that
the <code>rustc</code> parser is a &ldquo;simple hand-written recursive descent parser&rdquo;. This isn&rsquo;t surprising, since
most production parsers are written this way. (For example, <a href="https://docs.rs/syn"><code>syn</code></a> is also a
recursive descent parser.)</p>
<p>Turning our attention to the instruction pointer <code>fffffc7fd1adc4bb</code>, we can use the <code>::dis</code> command to disassemble the function at that address. (<a href="https://gist.github.com/sunshowers/959c649926fe4a9d8bc53d967f895cdf#file-gistfile0-txt-L11">Full output</a>; the <code>-a</code> flag ensures that addresses are not converted to very long function names.)</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> fffffc7fd1adc4bb::dis -a
</span></span><span class="line"><span class="cl"><span class="go">&lt;... snipped ...&gt;
</span></span></span><span class="line"><span class="cl"><span class="go">fffffc7fd1adc4b6                movl   $0x1,%edx
</span></span></span><span class="line"><span class="cl"><span class="go">fffffc7fd1adc4bb                call   +0x1caf0 &lt;librustc_driver-86178b5e8d46877c.so`&lt;rustc_parse::parser::Parser&gt;::parse_ident_common&gt;
</span></span></span><span class="line"><span class="cl"><span class="go">fffffc7fd1adc4c0                cmpl   $0x0,0xfffffffffffffdf0(%rbp)
</span></span></span><span class="line"><span class="cl"><span class="go">&lt;... snipped ...&gt;
</span></span></span></code></pre></div><p>So it looks like the crash is happening in a <code>call</code> instruction to another function,
<code>parse_ident_common</code>.</p>
<p>(Keep in mind that this information could be completely unreliable! The stack might be corrupted, the registers might be wrong, and so on. But it&rsquo;s what we have for now.)</p>
<h2 id="examining-the-address-space">Examining the address space</h2>
<p>On <a href="https://en.wikipedia.org/wiki/Virtual_memory">virtual memory systems</a>, which includes all modern
desktop and server systems, each process gets the illusion that it has a very large amount of memory
all to itself. This is called the address space of a process. The instructions, the call stack, and
the heap all get their own regions of addresses in that space, called <em>memory mappings</em>. The 64-bit
addresses that we saw earlier are all part of the address space.</p>
<p><code>mdb</code> has a command called <code>whatis</code> to look up which part of memory an address is at. Let&rsquo;s look at the stack pointer first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> fffffc7fed8e5c20::whatis
</span></span><span class="line"><span class="cl"><span class="go">fffffc7fed8e5c20 is in [ unknown ] [fffffc7fed8e5000,fffffc7fed8e6000)
</span></span></span></code></pre></div><p>This tells us that the address is in the range <code>0xfffffc7fed8e5000</code> to <code>0xfffffc7fed8e6000</code>. This is
a small 4 KiB range.</p>
<p>What about the frame pointer?</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> fffffc7fed8e6140::whatis
</span></span><span class="line"><span class="cl"><span class="go">fffffc7fed8e6140 is in [ unknown ] [fffffc7fed8e6000,fffffc7fed9e7000)
</span></span></span></code></pre></div><p>This appears to be in a different range.</p>
<p>In this case, the ending address is <code>fffffc7fed9e7000</code> (note the <code>9e</code>, not the <code>8e</code>!). This address
is <strong><code>0x101000</code></strong> bytes away from the starting address. That is equal to 1028 <abbr title="Kibibyte, or 1024 bytes">KiB</abbr>, or 1 <abbr title="Mebibyte, or 1024 kibibytes">MiB</abbr> + 4 KiB page<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>.</p>
<p>Something else that&rsquo;s relevant here is what permissions each range of addresses has. Like files on Unix, a block of virtual memory can have <em>read</em>, <em>write</em>, or <em>execute</em> permissions. (In this case, <em>execute</em> means that it is valid for the instruction pointer to point here<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>.)</p>
<p>On illumos, a tool called <code>pmap</code> can show these spaces. <code>pmap</code> works on both live processes and core files. Running <code>pmap core</code> shows the permissions for the addresses we&rsquo;re interested in (<a href="https://gist.github.com/sunshowers/03fdbd76162a838d9b11b3c9beba6a81#file-gistfile0-txt-L27-L28">full output</a>):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> pmap core
</span></span><span class="line"><span class="cl"><span class="go">&lt;... snipped ..&gt;
</span></span></span><span class="line"><span class="cl"><span class="go">FFFFFC7FED8E5000          4K -----    [ anon ]
</span></span></span><span class="line"><span class="cl"><span class="go">FFFFFC7FED8E6000       1028K rw---    [ anon ]
</span></span></span><span class="line"><span class="cl"><span class="go">&lt;... snipped ..&gt;
</span></span></span></code></pre></div><p>The 1028 KiB range is read-write, and the 4 KiB range above that doesn&rsquo;t have any permissions whatsoever.</p>
<p><strong>This would explain the segfault</strong>. A segfault is an attempt to operate on a part of memory that the program doesn&rsquo;t have permissions for. Attempting to read from or write to memory which has no permissions is an example of that.</p>
<h2 id="formulating-a-theory">Formulating a theory</h2>
<p>At this point, we have enough information to come up with a theory:</p>
<ul>
<li>The thread had a call stack of 1028 KiB available to it, starting at <code>fffffc7fed8e6000</code>.</li>
<li>The call stack pointer was at <code>fffffc7fed8e6140</code> (only <code>0x140</code> = 320 bytes away), and it tried to create a frame of size <code>0x520</code> (1312) bytes, at <code>fffffc7fed8e5c20</code>.</li>
<li>This caused the call stack to be <em>exhausted</em>: the thread ran out of space<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup>.</li>
<li>When the thread ran out of space, it indexed into a 4 KiB section known as a <em>guard page</em>. The thread did not have any permissions to operate on the page, and was in fact designed to cause a segfault if accessed in any way.</li>
<li>The program then (correctly) segfaulted.</li>
</ul>
<p>But there are also other bits of evidence that this theory doesn&rsquo;t explain, or even cuts against. (This is what makes post-mortem debugging exciting! There are often contradictory-seeming pieces of information that need to be explained.)</p>
<ol>
<li>
<p><strong>The memory is marked <code>anon</code> or <code>unknown</code>.</strong> That&rsquo;s not how call stacks are supposed to be marked! In the <code>pmap</code> output, there&rsquo;s a line which says:</p>
<pre tabindex="0"><code>FFFFFC7FED7B1000        316K rw---    [ stack tid=3 ]
</code></pre><p>So you&rsquo;d expect call stacks to be marked with <code>[ stack tid=&lt;something&gt; ]</code>, not <code>[ anon ]</code>.</p>
</li>
<li>
<p><strong>Why is the size of the allocation 1028 KiB?</strong> You&rsquo;d generally expect stack sizes to be a round power of two.</p>
</li>
<li>
<p><strong>Isn&rsquo;t 1028 KiB kind of small?</strong> The thread is a non-main thread, and <a href="https://doc.rust-lang.org/std/thread/#stack-size">the default stack size for Rust threads is 2 MiB</a>. Why is our thread ~1 MiB and not 2 MiB?</p>
<details>
<summary>How are call stack sizes determined?</summary>
<p>On Unix platforms, for the main thread, the call stack size is determined by <code>ulimit -s</code> (in KiB). On my illumos machine, this printed <code>10240</code>, indicating a 10 MiB call stack.</p>
<p>For child threads, the call stack size is determined by whatever created them. For Rust, the default is 2 MiB.</p>
</details>
</li>
<li>
<p><strong>Why doesn&rsquo;t this crash happen on other platforms?</strong> If this is a crash in the <code>rustc</code> parser, one would ordinarily expect it to arise everywhere. Yet it doesn&rsquo;t seem to occur on Linux, macOS, or Windows. What&rsquo;s special about illumos?</p>
</li>
<li>
<p><strong>Setting <code>RUST_MIN_STACK</code> doesn&rsquo;t help.</strong> Rust-created thread stack sizes can be configured via <a href="https://doc.rust-lang.org/std/thread/#stack-size">the <code>RUST_MIN_STACK</code> environment variable</a>. If we try to use that:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> <span class="nv">RUST_MIN_STACK</span><span class="o">=</span><span class="k">$((</span><span class="m">4</span> <span class="o">*</span> <span class="m">1024</span> <span class="o">*</span> <span class="m">1024</span><span class="k">))</span> cargo build
</span></span></code></pre></div><p>It turns out that <code>rustc</code> crashes at exactly the same spot. That&rsquo;s really strange!</p>
<p>It is possible that the stack size was overridden at thread creation time. The documentation for <code>RUST_MIN_STACK</code> says: &ldquo;Note that setting <code>Builder::stack_size</code> will override this.&rdquo; But that seems unlikely.</p>
</li>
</ol>
<h2 id="a-closer-look-at-the-call-stack">A closer look at the call stack</h2>
<p>Looking towards the bottom of the call stack, there&rsquo;s <a href="https://gist.github.com/sunshowers/5edb7207c1e1234b0400bc6517f45b29#file-gistfile0-txt-L1478-L1483">something really strange</a>:</p>
<pre tabindex="0"><code>fffffc7fed9e5f80 librustc_driver-86178b5e8d46877c.so`rustc_query_system::query::plumbing::try_execute_query...
fffffc7fed9e5fd0 librustc_driver-86178b5e8d46877c.so`stacker::grow::&lt;rustc_middle::query::erase::Erased&lt;[u8; 16]&gt;, ...&gt;
fffffc7fed9e5ff0 librustc_driver-86178b5e8d46877c.so`psm::on_stack::with_on_stack...
fffffc7fed7e4960 librustc_driver-86178b5e8d46877c.so`rust_psm_on_stack+9()
fffffc7fed7e4a20 librustc_driver-86178b5e8d46877c.so`stacker::_grow+0x13e()
fffffc7fed7e4ad0 librustc_driver-86178b5e8d46877c.so`rustc_query_impl::query_impl::resolver_for_lowering_raw::get_query_non_incr...
</code></pre><p>Notice the jump in addresses from <code>fffffc7fed7e4960</code> to <code>fffffc7fed9e5ff0</code>? Normally, stack addresses are decremented as new functions are called: the number goes down. In this case the stack address is <em>incremented</em>. The number went up. Strange.</p>
<p>Also notice that this coincides with the use of a function called <code>stacker::_grow</code>. Now that&rsquo;s a real lead!</p>
<p><strong>What part of memory is <code>fffffc7fed7e4960</code> in?</strong> <code>mdb</code> says:</p>
<pre tabindex="0"><code>&gt; fffffc7fed7e4960::whatis
fffffc7fed7e4960 is in [ stack tid=3 ]
</code></pre><p>So <em>this</em> address is part of the stack for thread 3. <a href="https://gist.github.com/sunshowers/03fdbd76162a838d9b11b3c9beba6a81#file-gistfile0-txt-L26"><code>pmap</code> agrees</a>:</p>
<pre tabindex="0"><code>FFFFFC7FED7B1000        316K rw---    [ stack tid=3 ]
</code></pre><p><strong>What is <code>stacker</code>?</strong> Time for some googling! Per <a href="https://docs.rs/stacker">the documentation</a>, <code>stacker</code> is:</p>
<blockquote>
<p>A library to help grow the stack when it runs out of space.</p>
<p>This is an implementation of manually instrumented segmented stacks where
points in a program’s control flow are annotated with “maybe grow the stack
here”. Each point of annotation indicates how far away from the end of the
stack it’s allowed to be, plus the amount of stack to allocate if it does
reach the end.</p>
</blockquote>
<p>Because the <code>rustc</code> parser is recursive, it is susceptible to call stack exhaustion. The use of <code>stacker</code> is supposed to prevent, or at least mitigate, that.</p>
<p><strong>How does <code>stacker</code> work?</strong> The library has <a href="https://docs.rs/stacker/0.1.15/stacker/fn.maybe_grow.html">a pretty simple API</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">maybe_grow</span><span class="o">&lt;</span><span class="n">R</span><span class="p">,</span><span class="w"> </span><span class="n">F</span>: <span class="nb">FnOnce</span><span class="p">()</span><span class="w"> </span>-&gt; <span class="nc">R</span><span class="o">&gt;</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">red_zone</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">stack_size</span>: <span class="kt">usize</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">callback</span>: <span class="nc">F</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">R</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div>
  <figure class="floatright" >
    
    <img src="/images/rust-stack.jpg"  alt="An image from YouTube showing Rust the video game, titled &#34;EZ Triple Floor Stack&#34;."   />
    
    
      <figcaption class="center" >Er, wrong Rust.</figcaption>
    
  </figure>


<p>The developer is expected to intersperse calls to <code>maybe_grow</code> within their recursive function. If less than <code>red_zone</code> bytes of stack space remain, <code>stacker</code> will allocate a new segment of <code>stack_size</code> bytes, and run <code>callback</code> with the stack pointer pointing to the new segment.</p>
<p><strong>How does rustc use <code>stacker</code>?</strong> The code is in <a href="https://github.com/rust-lang/rust/blob/dba8e2d2c2890a8b9e88cbf4855ac5711337946c/compiler/rustc_data_structures/src/stack.rs#L17">this file</a>. The code requests an additional 1 MiB stack with a red zone of 100 KiB.</p>
<p><strong>Why did <code>stacker</code> create a new stack segment?</strong> In our case, the call is at the very bottom of the stack, when plenty of space should be available, so ordinarily <code>stacker</code> should not need to allocate a new segment. Why did it do so here?</p>
<p>The answer is <a href="https://github.com/rust-lang/stacker/blob/5df2309ccf7b1671909386c2670c7342a4d44142/src/lib.rs#L412-L457">in <code>stacker</code>&rsquo;s source code</a>. There is code to guess the stack size on many platforms. But it isn&rsquo;t enabled on illumos: <code>guess_os_stack_limit</code> always returns <code>None</code>.</p>
<h2 id="putting-it-together">Putting it together</h2>
<p>With this information in hand, we can flesh out our call stack exhaustion theory:</p>
<ul>
<li>
<p>Some file in <code>cranelift-codegen</code> was triggering the crash by requiring more than 1 MiB of stack space.</p>
<ul>
<li>The <code>rustc</code> parser running against <code>cranelift-codegen</code> needed more than 1 MiB of stack space, but less than 2 MiB.</li>
</ul>
</li>
<li>
<p>Had this bug occurred on other platforms like Linux, this issue would have been a showstopper. However, it wasn&rsquo;t visible on those platforms because:</p>
<ul>
<li>Threads created by Rust use a 2 MiB stack by default.</li>
<li><code>rustc</code> requested that <code>stacker</code> create a 1 MiB stack segment, but only if less than 100 KiB of stack space was left.</li>
<li>On the other platforms, <code>stacker</code> could see that well over 100 KiB of stack space was left, and so it did not allocate a new segment.</li>
<li>On illumos, <code>stacker</code> could not see how much stack was left, and so it allocated a new 1 MiB segment.</li>
<li>This 1 MiB stack was simply not enough to parse <code>cranelift-codegen</code>.</li>
</ul>
</li>
<li>
<p><code>rustc</code> didn&rsquo;t call <code>stacker::maybe_grow</code> enough! In order for it to work, <code>stacker</code> needs to be interspersed throughout the recursive code. But some recursive parts did not appear to have called it.</p>
</li>
</ul>
<p>(It is somewhat ironic that <code>stacker</code>, a library meant to prevent call stack exhaustion, was actively making life worse here.)</p>
<p><strong>Where does the 1028 KiB come from?</strong> Looking at the <a href="https://github.com/rust-lang/stacker/blob/5df2309ccf7b1671909386c2670c7342a4d44142/src/lib.rs#L228-L234"><code>stacker</code> source code</a>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-rust" data-lang="rust"><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">page_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">page_size</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">requested_pages</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">stack_size</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">checked_add</span><span class="p">(</span><span class="n">page_size</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">&#34;unreasonably large stack requested&#34;</span><span class="p">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">page_size</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">stack_pages</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span>::<span class="n">cmp</span>::<span class="n">max</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">requested_pages</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">let</span><span class="w"> </span><span class="n">stack_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">stack_pages</span><span class="p">.</span><span class="n">checked_mul</span><span class="p">(</span><span class="n">page_size</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">&#34;unreasonably large stack requesteed&#34;</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>It looks like <code>stacker</code> first computes the number of requested pages by dividing the requested stack size by the page size, rounding up. Then it adds 2 to that. In our case:</p>
<ul>
<li>The requested stack size is 1 MiB.</li>
<li>With 4 KiB pages, this works out to 256 pages.</li>
<li><code>stacker</code> then requests 256 + 2 = 258 pages, which is 1032 KiB.</li>
</ul>
<p>This explains both the 1028 KiB allocation (one guard page after the stack), and the 4 KiB guard page we&rsquo;re crashing at (one guard page before the stack).</p>
<h2 id="triggering-the-bug-on-other-platforms">Triggering the bug on other platforms</h2>
<p>If the issue is that a 1 MiB stack isn&rsquo;t enough, it should be possible to reproduce this on other platforms by setting their stack size to something smaller than the 2 MiB default.</p>
<p>With a stack size &lt;= 1 MiB, we would expect that:</p>
<ol>
<li><code>rustc</code> calls <code>stacker</code> as before.</li>
<li>There are two possibilities: either <code>stacker</code> decides there is enough stack space and doesn&rsquo;t create a new segment, or it decides there isn&rsquo;t enough and does create a new 1 MiB segment.</li>
<li>In either case, 1 MiB is simply not enough to parse <code>cranelift-codegen</code>, and the program crashes.</li>
</ol>
<p>Let&rsquo;s try to compile <code>cranelift-codegen</code> on Linux with a reduced stack size.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> <span class="nb">cd</span> cranelift-codegen-test
</span></span><span class="line"><span class="cl"><span class="gp">$</span> <span class="nv">RUST_MIN_STACK</span><span class="o">=</span><span class="m">1048576</span> cargo +1.80.0 build
</span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="go">note: rustc unexpectedly overflowed its stack! this is a bug
</span></span></span><span class="line"><span class="cl"><span class="go">note: maximum backtrace depth reached, frames may have been lost
</span></span></span><span class="line"><span class="cl"><span class="go">note: we would appreciate a report at https://github.com/rust-lang/rust
</span></span></span><span class="line"><span class="cl"><span class="go">help: you can increase rustc&#39;s stack size by setting RUST_MIN_STACK=2097152
</span></span></span><span class="line"><span class="cl"><span class="go">note: backtrace dumped due to SIGSEGV! resuming signal
</span></span></span></code></pre></div><p>This does crash as expected. The full output is <a href="https://gist.github.com/sunshowers/3ac000e5a5022acd3f07886a16a39520">here</a>. Some of the symbols are missing, but the crash does seem to be in parser code.</p>
<p>(At this point, we could have gone further and tried to make <a href="https://github.com/rust-lang/rust/issues/116249#issuecomment-1741572717">a debug-assertions build of <code>rustc</code></a> &ndash; but it was already pretty clear why the crash was happening.)</p>
<h2 id="what-codes-failing-to-parse-anyway">What code&rsquo;s failing to parse, anyway?</h2>
<p>Call stack exhaustion in the parser suggests that the crash is happening in some kind of large, automatically generated file. But what file is it?</p>

  <figure class="floatright" >
    
    <img src="/images/strace-mascot.png"  alt="A cartoon ostrich with a light orange head, yellow eyes, orange beak and feet, and a coat of black and a couple shades of grey."   />
    
    
      <figcaption class="center" >Der Strauß, the strace mascot. CC BY-SA 4.0, by Vitaly Chaykovsky.</figcaption>
    
  </figure>


<p>It&rsquo;s hard to tell by looking at the core file itself, but we have another dimension of debugging at hand: syscall tracers! These tools print out all the <abbr title="System calls: calls from user programs into the kernel">syscalls</abbr> made by a process. Most OSes have some means to trace syscalls: <a href="https://strace.io/"><code>strace</code></a> on Linux, <a href="https://opensource.apple.com/source/dtrace/dtrace-147/DTTk/dtruss.auto.html"><code>dtruss</code></a> on macOS, <a href="https://en.wikipedia.org/wiki/Process_Monitor">Process Monitor</a> on Windows, and <a href="https://illumos.org/man/1/truss"><code>truss</code></a> on illumos<sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup>.</p>
<p>Since we&rsquo;re interested in file reads, we can try filtering it down to <a href="https://pubs.opengroup.org/onlinepubs/9699919799/">the <code>open</code> and <code>openat</code> syscalls</a>. You need to open a file to read it, after all. (Alternatively, we can also simply not filter out any syscalls, dump the entire trace to a file, and then look at it afterwards.)</p>
<p>On illumos, we tell <code>truss</code> to run <code>cargo build</code>, filtering syscalls to <code>open</code> and <code>openat</code> (<code>-t</code>), and following child processes (<code>-f</code>):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> truss -ft open,openat cargo build
</span></span></code></pre></div><p>This prints out every file that the child <code>rustc</code> tries to open (<a href="https://gist.github.com/sunshowers/1d9d67722d10b4b3f80ac2fe42d61e7a#file-gistfile0-txt-L549-L555">full output</a>):</p>
<pre tabindex="0"><code>20755/3:	open(&#34;/home/rain/dev/cranelift-codegen-test/target/debug/build/cranelift-codegen-dad37ce046df129a/out/isle_opt.rs&#34;, O_RDONLY|O_CLOEXEC) = 13
20755/3:	    Incurred fault #6, FLTBOUNDS  %pc = 0xFFFFFC7FD9E74361
20755/3:	      siginfo: SIGSEGV SEGV_ACCERR addr=0xFFFFFC7FED22CA58
20755/3:	    Received signal #11, SIGSEGV [default]
20755/3:	      siginfo: SIGSEGV SEGV_ACCERR addr=0xFFFFFC7FED22CA58
20754/3:	    Received signal #18, SIGCLD, in waitid() [default]
20754/3:	      siginfo: SIGCLD CLD_DUMPED pid=20755 status=0x000B
</code></pre><p>It looks like the crash is in a file called <code>isle_opt.rs</code> in the <code>out/</code> directory. With Cargo, a file being in an <code>out/</code> directory is a pretty strong indication that it is generated by a build script.</p>
<p>On Linux, a similar <code>strace</code> command is:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">RUST_MIN_STACK=1048576 strace -fe open,openat cargo build
</span></span></span></code></pre></div><p>This command also blames the same file, <code>isle_opt.rs</code>.</p>
<p>What does this file look like, anyway? <a href="https://gist.githubusercontent.com/sunshowers/22b1a612ee1cb88047c456532e0f6877/raw/9403b8b644f25c600d426ee1a39e2459684244dc/gistfile0.txt">Here&rsquo;s my copy.</a> It&rsquo;s pretty big and deeply nested! It does look large and complex enough to trigger call stack exhaustion.</p>
<p>Syscall traces would definitely be somewhat harder to get if the crash weren&rsquo;t so easily reproducible. Someone smarter than me should write about how to figure this out using just the core file. The file&rsquo;s fully loaded into memory so it seems like it should be possible.</p>
<h2 id="unblocking-myself">Unblocking myself</h2>
<p>Going back to the beginning: the reason I went down this adventure was because I wanted to make an unrelated change to the Rust standard library. But the stage 0 compiler being broken meant that it was impossible to get to the point where I could build the standard library as-is, let alone test that change.</p>
<p>How can we work around this? Well, going back to basics, where did the stage 0 compiler come from? It came from Rust&rsquo;s CI, and it wasn&rsquo;t actually built on illumos! (Partly because there&rsquo;s no publicly-available CI system running illumos.) Instead, it was cross-compiled from Linux to illumos.</p>
<p>Based on this, my coworker Joshua suggested that I try and do whatever Rust&rsquo;s CI does to build a stage 0 compiler for illumos.</p>
<p>Rust&rsquo;s CI uses <a href="https://github.com/rust-lang/rust/tree/75ac3b633/src/ci/docker">a set of Docker images</a> to build distribution artifacts. In theory, building a patched rustc should be as simple as running these commands on my Linux machine:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">#</span> Check out the exact version of the stage0 compiler
</span></span><span class="line"><span class="cl"><span class="gp">$</span> git checkout 75ac3b633
</span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="gp">#</span> Make changes...
</span></span><span class="line"><span class="cl"><span class="err">
</span></span></span><span class="line"><span class="cl"><span class="gp">#</span> Run Docker build
</span></span><span class="line"><span class="cl"><span class="gp">$</span> ./src/ci/docker/run.sh dist-x86_64-illumos
</span></span></code></pre></div><p>In reality, there were some Docker permissions issues due to which I had to make a couple of changes to the script. Overall, though, it was quite simple. <a href="https://gist.github.com/sunshowers/2dacc5902ad4aecc50d215f018f55232">Here&rsquo;s the patch</a> I built the compiler with, including the changes to the CI scripts.</p>
<p>The result of building the compiler was a set of <code>.tar.xz</code> files, just like <a href="https://github.com/rust-lang/rust/blob/2d5a628a1de1d38318909a710ef37da6251e362e/src/stage0">the ones published by Rust&rsquo;s CI</a>. After copying the files over to my illumos machine, I wasn&rsquo;t sure which tarballs to extract. So I made <a href="https://gist.github.com/sunshowers/9b6774edfabbea6985881617302caf34">a small change</a> to the bootstrap script to use my patched tarballs.</p>
<p>With this patch, I was able to successfully build Rust&rsquo;s standard library on illumos and test my changes. Hooray! (<a href="https://github.com/rust-lang/rust/pull/128259">Here&rsquo;s</a> what I was trying to test.)</p>
<p><em>Update 2024-08-05: After this post was published, jyn pointed out <a href="https://hachyderm.io/deck/@jyn@tech.lgbt/112906410687051157">on Mastodon</a> that <code>cranelift-codegen</code> is actually optional, and that I could have also worked around the issue by disabling it in the <code>rustc</code> build system&rsquo;s <code>config.toml</code>. Thanks!</em></p>
<h2 id="what-did-we-learn">What did we learn?</h2>
<p>The bug occurred due to a combination of several factors. It also revealed a few other issues, such as the lack of an environment variable workaround and some missing error reporting.</p>
<p>Here are some ways we can make the situation better, and help us have an easier time
debugging similar issues in the future.</p>
<ol>
<li>
<p><strong><code>rustc</code> isn&rsquo;t using <code>stacker</code> enough.</strong> The basic problem underneath it all is that the part of the <code>rustc</code> parser that triggered the bug wasn&rsquo;t calling <code>stacker</code> often enough to make new stack segments. <code>rustc</code> should be calling <code>stacker</code> more than it is today.</p>
<ul>
<li>Filed as <a href="https://github.com/rust-lang/rust/issues/128422">rust-lang/rust#128422</a>.</li>
</ul>
</li>
<li>
<p><strong><code>stacker</code> cannot detect the stack size on illumos.</strong> This is something that we should fix in <code>stacker</code>, but this is actually a secondary issue here. On other platforms, <code>stacker</code>&rsquo;s ability to detect the stack size was masking the <code>rustc</code> bug.</p>
<p>Fixing this requires two changes:</p>
<ul>
<li>A <a href="https://github.com/rust-lang/libc/pull/3788">PR to <code>libc</code></a> to add the <code>pthread_attr_get_np</code> function to it.</li>
<li>A <a href="https://github.com/rust-lang/stacker/pull/88">PR to <code>stacker</code></a> to use this function to detect the stack size on illumos.</li>
</ul>
</li>
<li>
<p><strong><code>stacker</code>-created segments don&rsquo;t print a nice message on stack exhaustion.</strong> This is a bit ironic because <code>stacker</code> is supposed to prevent stack exhaustion. But when it does happen, it would be nice if <code>stacker</code> printed out a message like standard Rust does.</p>
<ul>
<li>This is <a href="https://github.com/rust-lang/stacker/issues/59">rust-lang/stacker#59</a>.</li>
</ul>
</li>
<li>
<p><strong>On illumos, the Rust runtime doesn&rsquo;t print a message on stack exhaustion.</strong> Separate from the previous point, on illumos the Rust runtime doesn&rsquo;t print a message on stack exhaustion even when using native stacks.</p>
<ul>
<li>Filed as <a href="https://github.com/rust-lang/rust/issues/128568">rust-lang/rust#128568</a>.</li>
</ul>
</li>
<li>
<p><strong>Rust&rsquo;s CI doesn&rsquo;t run on illumos.</strong> At Oxide, we have an existential dependency on Rust targeting illumos. Even a shadow CI that ran on nightly releases would have caught this issue right away.</p>
<p>We&rsquo;re discussing the possibilities for this internally; stay tuned!</p>
</li>
<li>
<p><strong><code>stacker</code> segment sizes can&rsquo;t be controlled via the environment.</strong> Being able to control stack sizes with <code>RUST_MIN_STACK</code> is a great way to work around issues. It doesn&rsquo;t appear that <code>stacker</code> segment sizes can be controlled in this manner. Maybe that functionality should be added to <code>rustc</code>, or to <code>stacker</code> itself?</p>
<ul>
<li>Opened a <a href="https://internals.rust-lang.org/t/allow-controlling-rustc-stacker-segment-sizes-via-the-environment/21292">discussion on internals.rust-lang.org</a>.</li>
</ul>
</li>
<li>
<p><strong>Maybe a <a href="https://github.com/rust-lang/crater">crater</a> run with a smaller stack size?</strong> It would be interesting to see if there are other parts of the Rust codebase that need to call <code>stacker</code> more as well.</p>
</li>
<li>
<p><strong><code>x.py</code> suggests disabling optional components.</strong> Since <code>cranelift-codegen</code> was an optional component that can be disabled, the <code>x.py</code> tooling
could notice if a build failed in such a component, and recommend disabling that component. <em>Added 2024-08-05, suggested <a href="https://hachyderm.io/deck/@jyn@tech.lgbt/112906451310138312">by jyn</a>.</em></p>
</li>
</ol>
<p>To me, this is the most exciting part of debugging: what kinds of changes can we make, both
specific and systemic ones, to make life easier for our future selves?</p>
<h2 id="conclusion-and-credits">Conclusion and credits</h2>
<p>This was a really fun debugging experience because I got to learn about several illumos debugging
tools, and also because we could synthesize information from several sources to figure out a complex
issue. (Thankfully, the root cause was straightforward, with no memory corruption or other &ldquo;spooky
action at a distance&rdquo; involved.)</p>
<p>Debugging this was a real team effort. I couldn&rsquo;t have done it without the assistance of several of
my exceptional colleagues. In no particular order:</p>
<ul>
<li><a href="https://m.unix.house/@jmc">Joshua M. Clulow</a></li>
<li><a href="https://mattkeeter.com">Matt Keeter</a></li>
<li><a href="https://discuss.systems/@cross">Dan Cross</a></li>
<li><a href="https://cliffle.com/">Cliff Biffle</a></li>
<li><a href="https://steveklabnik.com">Steve Klabnik</a></li>
<li><a href="https://artemis.sh">artemis everfree</a></li>
</ul>
<p>Thanks to all of you!</p>
</div>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>I neglected to do this during my own debugging session, which led to some confusion when I re-ran the process and found that the core file had been overwritten.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p><a href="https://en.wikipedia.org/wiki/Name_mangling">Name
mangling</a> is a big topic of its own, but the short
version is that the Rust compiler uses <a href="https://doc.rust-lang.org/rustc/symbol-mangling/v0.html">an
algorithm</a> to encode function names into
the binary. The encoding is designed to be reversible, and the process of doing so is called
demangling. (Other languages like C++ do name mangling, too.)&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>You might have heard about &ldquo;frame pointer omission&rdquo;, which is a technique to infer the base of stack frames rather than storing it in <code>%rbp</code> explicitly. In this case, the frame pointer is not omitted.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4">
<p>A <a href="https://en.wikipedia.org/wiki/Page_%28computer_memory%29"><em>page</em></a> is the smallest amount of physical memory that can be atomically mapped to virtual memory. On x86-64, the page size is virtually always 4 KiB.&#160;<a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5">
<p>Memory being both writable and executable is dangerous, and modern systems <a href="https://en.wikipedia.org/wiki/W%5EX">do not permit this</a> by default for security reasons. Some platforms like iOS even make it impossible for memory to be writable and executable, unless the platform holder gives you the corresponding permissions.&#160;<a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6">
<p>This is generally known as a &ldquo;stack overflow&rdquo;, but that term can also mean a <a href="https://cwe.mitre.org/data/definitions/121.html">stack-based buffer overflow</a>. Throughout this document, we use &ldquo;call stack exhaustion&rdquo; to avoid confusion.&#160;<a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7">
<p>There is likely some way to get <code>rustc</code> itself to print out which files it opened, but the beauty of system call tracers is that you don&rsquo;t need to know anything about the program you&rsquo;re tracing.&#160;<a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/illumos" term="illumos" label="illumos"/><category scheme="https://sunshowers.io/tags/debugging" term="debugging" label="debugging"/></entry></feed>