<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-us"><generator uri="https://gohugo.io/" version="0.170.0">Hugo</generator><title type="html">Ecc on sunshowers</title><subtitle type="html">thoughts and ideas</subtitle><link href="https://sunshowers.io/tags/ecc/" rel="alternate" type="text/html" title="html"/><link href="https://sunshowers.io/tags/ecc/index.xml" rel="alternate" type="application/rss+xml" title="rss"/><link href="https://sunshowers.io/tags/ecc/atom.xml" rel="self" type="application/atom+xml" title="atom"/><updated>2026-11-15T23:03:09+00:00</updated><rights>© Rain 2020-present. Licensed under CC BY-NC-SA 4.0 unless marked otherwise.</rights><author><name>Rain</name></author><id>https://sunshowers.io/tags/ecc/</id><entry><title type="html">ECC RAM on AMD Ryzen 7000 desktop CPUs</title><link href="https://sunshowers.io/posts/am5-ryzen-7000-ecc-ram/?utm_source=atom_feed" rel="alternate" type="text/html"/><id>https://sunshowers.io/posts/am5-ryzen-7000-ecc-ram/</id><published>2023-10-09T00:00:00+00:00</published><updated>2023-10-09T00:00:00+00:00</updated><content type="html"><![CDATA[<blockquote>Getting ECC memory working on desktop CPUs.</blockquote><h1 id="introduction">Introduction</h1>
<p>One of the coolest features of AMD&rsquo;s Ryzen desktop CPUs, and historically a great reason to get them
over the competition, was the official support for error-corrected memory (ECC RAM)<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. With most Ryzen
1000 through 5000 series CPUs and the right motherboards, ordinary users could get ECC RAM going
without having to spring for more expensive workstation-grade CPUs.</p>

  <figure class="floatright" >
    
      <a href="/images/b550-specs.png">
    
    <img src="/images/b550-specs.png"  alt="Screenshot of B550 Steel Legend specification page, showing support for ECC &amp; non-ECC, unbuffered memory"   />
    
      </a>
    
    
      <figcaption class="center" >Specification page for the B550 Steel Legend motherboard.</figcaption>
    
  </figure>


<p>For example, here&rsquo;s the <a href="https://www.asrock.com/mb/AMD/B550%20Steel%20Legend/index.asp#Specification">specification
page</a> for the ASRock
B550 Steel Legend motherboard. This is a mainstream &ldquo;B&rdquo; series motherboard which lists detailed
compatibility information for ECC RAM by processor generation.</p>
<p>(To my knowledge ASRock has had the best support for ECC RAM in Ryzen motherboards, and I&rsquo;ve been
very happy with their motherboards in general.)</p>
<hr>

  <figure class="floatright" >
    
      <a href="/images/x670e-specs.png">
    
    <img src="/images/x670e-specs.png"  alt="Screenshot of X670E Taichi specification page, without support for ECC memory"   />
    
      </a>
    
    
      <figcaption class="center" >Specification page for the X670E Taichi motherboard, with no mention of ECC support.</figcaption>
    
  </figure>


<p>Unfortunately, when the AMD Ryzen 7000 &ldquo;Raphael&rdquo; CPUs were launched along with the brand new <a href="https://en.wikipedia.org/wiki/Socket_AM5">Socket
AM5</a>, all mention of ECC support was gone. The
<a href="https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp#Specification">specification page</a> for the
ASRock X670E Taichi, one of the most expensive AM5 motherboards you can buy, has <strong>no mention of ECC
support</strong> as of the date of writing this.</p>
<p>I still decided to upgrade to a Ryzen 7950X, and overall I&rsquo;ve been happy with the performance of the new processor. But the lack of ECC was a huge bummer at the time of purchasing my system.</p>
<h1 id="finding-a-forum-link">Finding a forum link</h1>
<p>A couple months ago I came across <a href="https://forum.asrock.com/forum_posts.asp?TID=24901">a topic on the ASRock
forums</a> talking about ECC support on AM5
motherboards, in which a user called ApplesOfEpicness said that they&rsquo;d worked with an AMD engineer
to get ECC RAM going within AMD&rsquo;s AGESA firmware. They&rsquo;d claimed to have tested it on an ASRock
motherboard with an updated UEFI, by shorting ground and data pins, and seeing errors be reported up
to the OS.</p>
<p>I was intrigued by this! Even though I didn&rsquo;t have the same motherboard that ApplesOfEpicness did, I
had chosen an ASRock board (the <a href="https://pg.asrock.com/mb/AMD/B650E%20PG%20Riptide%20WiFi/index.asp">B650E PG
Riptide</a>)&mdash;I had figured that
if ECC was possible on any AM5 board at all, it would be supported on ASRock. So based on the forum
post, last week I ordered <a href="https://v-color.net/products/ddr5-ecc-udimm-servermemory?variant=43445581906087">a pair of 32 GB server-grade ECC sticks from
v-color</a>.</p>
<p>I updated my motherboard&rsquo;s UEFI to the latest version (version 1.28 with AGESA 1.0.0.7b), and then
replaced my existing RAM with the new sticks. I started up the system, and after a very long <a href="https://www.allaboutcircuits.com/news/boosing-memory-performance-age-ddr5-introduction-ddr-training-modes/">link
training</a>
process<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>&hellip; it booted up!</p>
<h1 id="does-the-os-recognize-ecc">Does the OS recognize ECC?</h1>
<p>On the Linux side, all indications were that the ECC memory was functioning correctly. <code>sudo dmidecode -t memory</code> reported:</p>
<pre tabindex="0"><code>% sudo dmidecode -t memory
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC

... &lt;snip&gt; ...

Handle 0x0033, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x002E
        Error Information Handle: 0x0032
        Total Width: 72 bits
        Data Width: 64 bits
</code></pre><p>(The &ldquo;Total Width&rdquo; field is the important one here. For non-ECC RAM it read 64 bits, but in my case it was 72 bits because 64-bit ECC RAM has an <a href="https://www.anandtech.com/show/43/6">extra 8 bits</a> of parity data.)</p>
<p>Also, the Linux kernel reported that its error detection and correction subsystem,
<a href="https://docs.kernel.org/driver-api/edac.html">EDAC</a>, was enabled:</p>
<pre tabindex="0"><code>% sudo dmesg | grep -i EDAC
[    0.444842] EDAC MC: Ver: 3.0.0
[   25.042690] EDAC MC0: Giving out device to module amd64_edac controller F19h_M60h: DEV 0000:00:18.3 (INTERRUPT)
[   25.042693] EDAC amd64: F19h_M60h detected (node 0).
[   25.042696] EDAC MC: UMC0 chip selects:
[   25.042697] EDAC amd64: MC: 0:     0MB 1:     0MB
[   25.042699] EDAC amd64: MC: 2: 16384MB 3: 16384MB
[   25.042702] EDAC MC: UMC1 chip selects:
[   25.042703] EDAC amd64: MC: 0:     0MB 1:     0MB
[   25.042704] EDAC amd64: MC: 2: 16384MB 3: 16384MB
</code></pre><p>Looking good so far!</p>
<h1 id="wheres-this-data-coming-from">Where&rsquo;s this data coming from?</h1>
<p>At this point it&rsquo;s worth asking about the source of these messages. Where is the data coming from
and why should we believe it?</p>
<p>Let&rsquo;s look at <code>dmidecode</code> first. <code>man dmidecode</code> <a href="https://linux.die.net/man/8/dmidecode">starts with</a>:</p>
<blockquote>
<p>dmidecode is a tool for dumping a computer’s DMI (some say SMBIOS) table contents in a human‐readable format. This table contains a description of the system’s hardware components, as well as other useful pieces of information such as serial numbers and BIOS revision. Thanks to this table, you can retrieve this information without having to probe for the actual hardware. While this is a good point in terms of report speed and safeness, this also makes the presented information possibly unreliable.</p>
</blockquote>
<p>Oh, interesting, &ldquo;possibly unreliable&rdquo; is a little concerning! What is this SMBIOS thing anyway? <a href="https://en.wikipedia.org/wiki/System_Management_BIOS">Wikipedia says</a>:</p>
<blockquote>
<p>In computing, the System Management BIOS (SMBIOS) specification defines data structures (and access methods) that can be used to read management information produced by the BIOS of a computer. This eliminates the need for the operating system to probe hardware directly to discover what devices are present in the computer.</p>
</blockquote>
<p>So the data presented by <code>dmidecode</code> is coming from the <em>UEFI</em>, not from the processor<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>. What this means is that the memory is ECC-<em>capable</em>, but not necessarily that it is <em>active</em>. Whether ECC is active is ultimately determined by the <a href="https://en.wikipedia.org/wiki/Memory_controller">memory controller</a> on the system.</p>
<h1 id="querying-the-memory-controller">Querying the memory controller</h1>
<p>When I mentioned setting up ECC at <a href="https://oxide.computer/">work</a>, <a href="https://fingolfin.org/blog/">Robert
Mustacchi</a> pointed me to the excellent <a href="https://github.com/oxidecomputer/illumos-gate/blob/5f01ecd8941eadb64bc15b1a02c468604c1a503e/usr/src/uts/intel/sys/amdzen/umc.h#L22">illumos documentation about
AMD&rsquo;s Unified Memory
Controller</a>.
I did some reading and learned that essentially, AMD processors expose a bus called the System
Management Network (SMN). Among other things, this bus can be used to query and configure the AMD
Unified Memory Controller (UMC).</p>
<div class="framed">
<p><strong>NOTE:</strong> The information in the rest of this section is not part of the public AMD Processor
Programming Reference, but can be gleaned from the source code for the open-source Linux and
illumos kernels.</p>
<p><strong>WARNING:</strong> Accessing the SMN directly, and especially sending write commands to it, is dangerous
and can <strong>severely damage</strong> your computer. Do not write to the SMN unless you know what you&rsquo;re
doing.</p>
</div>
<p>The idea is that we can ask the UMC the question &ldquo;is ECC enabled&rdquo; directly, by sending a read
request over the SMN to what is called the <code>UmcCapHi</code> register. The exact addresses involved are a
little bit magical, but on illumos with a Ryzen 7000 processor, here&rsquo;s how you would query the UMC
over the SMN bus (channel 0 and channel 1 are the two memory channels on the system, and each
channel has one of the 32GB sticks plugged into it.)</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Query the UMC at address 0x50df4, representing channel 0</span>
</span></span><span class="line"><span class="cl">$ pfexec /usr/lib/usmn -d /devices/pseudo/amdzen@0/usmn@2:usmn.0 0x50df4
</span></span><span class="line"><span class="cl">0x50df4: 0x40000030
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Query the UMC at address 0x150df4, representing channel 1</span>
</span></span><span class="line"><span class="cl">$ pfexec /usr/lib/usmn -d /devices/pseudo/amdzen@0/usmn@2:usmn.0 0x150df4
</span></span><span class="line"><span class="cl">0x150df4: 0x40000030
</span></span></code></pre></div><p>(<code>pfexec</code> is the illumos equivalent to <code>sudo</code>.)</p>
<p>Also, illumos comes with a really nice way to break up a hex value into bits:</p>
<pre tabindex="0"><code>$ mdb -e &#39;0x40000030=j&#39;
                1000000000000000000000000110000
                |                        ||
                |                        |+------ bit 4  mask 0x00000010
                |                        +------- bit 5  mask 0x00000020
                +-------------------------------- bit 30 mask 0x40000000
</code></pre><p>The bit we&rsquo;re interested in here is bit 30. If it&rsquo;s set, then ECC is enabled in the memory controller.</p>
<h2 id="accessing-the-smn-on-linux-with-the-ryzen_smu-driver">Accessing the SMN on Linux with the <code>ryzen_smu</code> driver</h2>
<p>Can we replicate this query on Linux? Turns out we can! There&rsquo;s a neat little driver called
<a href="https://gitlab.com/leogx9r/ryzen_smu"><code>ryzen_smu</code></a> which provides access to the SMN bus. It&rsquo;s easy
to download and install (though on my system I needed to <a href="https://gitlab.com/leogx9r/ryzen_smu/-/merge_requests/10">apply a
patch</a>).</p>
<p>The driver exposes a <a href="https://gitlab.com/leogx9r/ryzen_smu#syskernelryzen_smu_drvsmn">file called
<code>/sys/kernel/ryzen_smu_drv/smn</code></a>
which can be used to perform a query over the SMN bus. The documentation says that to perform a
query, we must write 4 bytes to the file in <a href="https://www.section.io/engineering-education/what-is-little-endian-and-big-endian/">little-endian
format</a>, and
then read 4 bytes from the output in little-endian format. This isn&rsquo;t convenient to do via the
command line, so let&rsquo;s write a small Python script:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># smn-query-ecc.py</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Licensed under CC0-1.0</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">query</span><span class="p">(</span><span class="n">hex_str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Convert hex string to bytes in little-endian</span>
</span></span><span class="line"><span class="cl">    <span class="n">decoded</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">hex_str</span><span class="p">,</span> <span class="mi">16</span><span class="p">)</span><span class="o">.</span><span class="n">to_bytes</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="n">byteorder</span><span class="o">=</span><span class="s1">&#39;little&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">decoded</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Write 4 bytes to the SMN file</span>
</span></span><span class="line"><span class="cl">    <span class="nb">open</span><span class="p">(</span><span class="s2">&#34;/sys/kernel/ryzen_smu_drv/smn&#34;</span><span class="p">,</span> <span class="s2">&#34;wb&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">decoded</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Read 4 bytes from the SMN file, representing the return value</span>
</span></span><span class="line"><span class="cl">    <span class="n">ret</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s2">&#34;/sys/kernel/ryzen_smu_drv/smn&#34;</span><span class="p">,</span> <span class="s2">&#34;rb&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Print ret as a hex string in little-endian order</span>
</span></span><span class="line"><span class="cl">    <span class="n">ret_hex_str</span> <span class="o">=</span> <span class="nb">hex</span><span class="p">(</span><span class="nb">int</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">ret</span><span class="p">,</span> <span class="n">byteorder</span><span class="o">=</span><span class="s1">&#39;little&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;returned value for </span><span class="si">{</span><span class="n">hex_str</span><span class="si">}</span><span class="s2"> is </span><span class="si">{</span><span class="n">ret_hex_str</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="n">hex_str</span> <span class="o">=</span> <span class="s2">&#34;0x00050df4&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">query</span><span class="p">(</span><span class="s2">&#34;0x00050df4&#34;</span><span class="p">)</span>  <span class="c1"># channel 0</span>
</span></span><span class="line"><span class="cl">    <span class="n">query</span><span class="p">(</span><span class="s2">&#34;0x00150df4&#34;</span><span class="p">)</span>  <span class="c1"># channel 1</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">main</span><span class="p">()</span>
</span></span></code></pre></div><p>Running this script, I got:</p>
<pre tabindex="0"><code>$ sudo python3 smn-query-ecc.py
return value for 0x00050df4 is 0x40000000
return value for 0x00150df4 is 0x40000000
</code></pre><p>Bit 30 (the first nibble&rsquo;s <code>4</code>) is set, which means the memory controller is reporting that ECC is
enabled.</p>
<p>This query should also be possible on Windows, perhaps using <a href="https://github.com/irusanov/SMUDebugTool">this
tool</a>, though I can&rsquo;t vouch for it.</p>
<div class="framed">
<h1 id="but-is-ecc-really-working">But is ECC <em>really</em> working?</h1>
<p>The most foolproof way to test whether ECC is working is to introduce an error somehow.</p>
<ul>
<li>ApplesOfEpicness did so by shorting a data and ground pin on their motherboard.</li>
<li>Another way would be to try and overclock the RAM until it gets to an unstable point.</li>
</ul>
<p>I don&rsquo;t quite have the courage to physically short pins, nor the patience to slowly overclock my
RAM, waiting multiple minutes for DDR5 link training each time. So instead, I&rsquo;m content with knowing that the memory controller is reporting that ECC is enabled.</p>
<p>Organically, I haven&rsquo;t seen any errors so far. If a correctable or uncorrectable error does occur at
some point, I&rsquo;ll update this post with that information.</p>
</div>
<h1 id="about-those-edac-messages">About those EDAC messages</h1>
<p>Earlier in this post I&rsquo;d mentioned that the Linux kernel reported that EDAC was enabled. I was
curious what the data source for <em>that</em> was, so I dug into the Linux kernel source code.</p>
<p>Being generally unfamiliar with the Linux codebase, I used the tried and tested strategy of
searching for strings that get logged. In this case:</p>
<ul>
<li>Searching for <code>Giving out device to module</code> led me to find <a href="https://github.com/torvalds/linux/blob/82714078aee4ccbd6ee7579d5a21f8a72155d0fb/drivers/edac/edac_mc.c#L665-L668">this line</a> inside <code>edac_mc_add_mc_with_groups</code>.</li>
<li>This function is called <a href="https://github.com/torvalds/linux/blob/82714078aee4ccbd6ee7579d5a21f8a72155d0fb/drivers/edac/amd64_edac.c#L4212">here</a> inside <code>init_one_instance</code>.</li>
<li><code>init_one_instance</code> <a href="https://github.com/torvalds/linux/blob/82714078aee4ccbd6ee7579d5a21f8a72155d0fb/drivers/edac/amd64_edac.c#L4285">is only called</a> if <code>pvt-&gt;ops-&gt;ecc_enabled</code> returns true.</li>
<li>What is <code>ecc_enabled</code>? It is set to a function called <code>umc_ecc_enabled</code> in <a href="https://github.com/torvalds/linux/blob/82714078aee4ccbd6ee7579d5a21f8a72155d0fb/drivers/edac/amd64_edac.c#L3985-L3991">this code</a>. And <code>pvt-&gt;ops</code> is set to <code>umc_ops</code> <a href="https://github.com/torvalds/linux/blob/82714078aee4ccbd6ee7579d5a21f8a72155d0fb/drivers/edac/amd64_edac.c#L4023-L4024">when the processor family is &gt;= 0x17</a>. Ryzen 7000 (Zen 4) is <a href="https://en.wikipedia.org/wiki/List_of_AMD_CPU_microarchitectures#Nomenclature">family 0x19</a>.</li>
</ul>
<p>Going by just the name, <code>umc_ecc_enabled</code> sounds like it would be querying the UMC. So let&rsquo;s look at <a href="https://github.com/torvalds/linux/blob/94f6f0550c625fab1f373bb86a6669b45e9748b3/drivers/edac/amd64_edac.c#L3619C51-L3619C51">what it does</a>. It looks like it&rsquo;s checking that <code>umc_cap_hi</code>&rsquo;s <code>UMC_ECC_ENABLED</code> bit is set.</p>
<p>And what is <code>UMC_ECC_ENABLED</code>? It&rsquo;s <a href="https://github.com/torvalds/linux/blob/94f6f0550c625fab1f373bb86a6669b45e9748b3/drivers/edac/amd64_edac.h#L272">bit 30</a>!</p>
<p>So it looks like the <code>EDAC</code> messages are only shown if the UMC reports that ECC is enabled. This
means that, at least on AMD processors, the Linux kernel message <code>EDAC MC0: Giving out device to module amd64_edac</code> is a reliable indicator that ECC is enabled.</p>
<h1 id="conclusion">Conclusion</h1>
<p>ECC RAM is great, and you can easily get it working on Ryzen 7000 desktop CPUs, at least with ASRock
motherboards. I learned a ton of low-level processor interface details along the way.</p>
<h1 id="acknowledgements">Acknowledgements</h1>
<p>Thanks again to <a href="https://fingolfin.org/blog/">Robert</a> for teaching me about a lot of the details here!</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>While ECC RAM is probably overkill for most desktop users and I don&rsquo;t have it in my
gaming PC, I&rsquo;ve seen enough random bit flips on servers to know that I would like to have ECC RAM in
the computer I depend on for my livelihood.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2">
<p>The internet is replete with complaints about <a href="https://www.google.com/search?q=zen&#43;4&#43;slow&#43;boot">slow Zen 4 boot
times</a>&mdash;these is in part due to DDR5 link
training, which is a very slow process. With my system, link training almost three minutes for
just 64 GB of RAM. Thankfully, at least on Ryzen 7000 desktops, it only needs to be done once
after replacing RAM or changing timings. The UEFI caches the results of link training and reuses
those values on subsequent boots.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3">
<p>There&rsquo;s some nuance here: some information like the memory speed does come from the memory controller. The ECC information, however, comes from the UEFI.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</div>
]]></content><category scheme="https://sunshowers.io/tags/ecc" term="ecc" label="ecc"/><category scheme="https://sunshowers.io/tags/ryzen" term="ryzen" label="ryzen"/><category scheme="https://sunshowers.io/tags/amd" term="amd" label="amd"/></entry></feed>