For those wondering what the name of the audio API was that made me go “I can’t believe I am forgetting the acronym” in the middle of this video, it was ASIO.

The questions addressed in this video are:

• [00:02] “The D language actually has this because of Uniform Function Call Syntax. You could have a struct like … and have the rest of the code be none the wiser (even code like foo.x = 7). I think C# has stuff like this too. Is this what you’d be hoping for? If so, I’m surprised - I’d have guessed you usually don’t prefer implicit magic like this happening on things that don’t look like function calls.”

• [04:37] “Could someone remind me what the ‘warm’ criteria was?”

• [06:00] “I have a question about managing intrusive thoughts while writing code. When thoughts come up about edge cases, performance, reliability, or decisions that may need to be revisited later, how do you decide whether to address them immediately or set them aside? I’m finding it difficult to iterate quickly because I often get pulled into these concerns before I’ve built a working v0. I’ve tried writing TODOs in the code and keeping separate notes, but neither approach has worked well for me.”

• [15:47] “I now have a question about wasapi (I believe you now use wasapi but if I’m wrong then you can ignore this question). I’m writing the sound system for my codebase and had my wav parser ready, mixer chugging along, sat back to enjoy a song but then found out that changing the system volume causes these pops.... I know that changing the volume of a sound wave causes the discontinuity which manifests itself as audio popping but in this case where it’s not an internal volume I have control over I don’t know how to fix these. It seems to me as if windows just immediately applied the new volume and my futile attempt of trying to bypass windows’ volume logic didn’t work since they seem to first clamp my samples and then multiply by the volume... I tried some wasapi gist examples and they all had this same issue. Have you had this problem and fixed it in your own code and could you share your knowledge if you have?”

• [25:07] “I formed my question about heaps slightly misleading. It wasn’t about a memory heap, but a data structure, like in the heap sort. In the video together with Primeagen regarding code interviews you commented that heaps are quite slow. So I was wondering, what should you use instead, when the need arises?”

• [33:27] “I’m working on a project that uses a cortex-m7, a fairly fast 32-bit embedded core. It has some fancy pipeline features of big boy desktop chips, like a branch predictor, and the ability to dual issue many instructions, but doesn’t have a RAT (it’s not “out-of-order” at least as far as I know). Do I understand correctly that in CPUs without a RAT, any two adjacent instructions that use the same register are effectively serially dependent? Is this how it used to work in the desktop world back in the day?”

• [40:53] “This question relates to Part 5: Dependency Chain Stalls and In-order Interleaving. From these results, I would expect 12-way interleaving to maintain peak throughput regardless of chain length (4 FMAs/cycle * 3 cycles = 12 FMAs in flight needed to cover the latency). However, that’s not what I observe. With 12-way interleaving, throughput decreases from ~3.6 to ~3.0 FMAs/cycle as the chain length grows. I need a minimum of 16-way interleaving for throughput to stay flat at ~3.8 FMAs/cycle. My conclusion is that the CPU can’t sustain the 3-cycle latency when the FMA execution ports are saturated, resulting in an effective latency of 4 cycles under load. This would explain why I need 4 * 4 = 16 chains to obtain a flat throughput curve. I think you’ve hinted at this in some of your videos. Would you kindly explain why this happens? Thanks!”

Read more

Suppose you want to pick a random element from a set that is difficult to iterate. Perhaps the candidate elements are embedded in a spatial hierarchy, and only determined by a complex predicate.

One obvious way to solve this problem is to first build a “filtered” array of all the candidate elements, then pick a random index into that array. This always works, and is fair (assuming the RNG is fair) — but it requires reserving space for a potentially-large array, and uses up bandwidth and cache space each time a random selection is made.

A non-obvious-but-common alternative is to use a statistical method where random numbers are chosen during the initial iteration to continuously “retain or replace” a running selection. The random numbers must be carefully constructed, and typically require a modulus on each step. Proof of the fairness of the method is not obvious, and requires knowledge of the relevant underlying math.

In short, while it works, and is likely more efficient than the array method, it has drawbacks.

Inspired by the way you might write such a query in SQL, Marc “Mahk” LeBlanc shared with me a new arrayless picking method that has none of these drawbacks. It’s very similar to the statistical methods version, but without any of the drawbacks.

In this video, Mahk presents the algorithm in a series of steps, starting from a “worst” random selector, and ending with his new one.

Subscribe now

As people integrate more and more AI into their daily work, will the quality of the work they produce suffer? Will their own skills degrade as a result of relying on AI? Will there be negative psychological effects in the shift from doing work directly to doing it indirectly through AI prompting?

In this episode, Demetri and I explore these issues as we consider the broad question many people are asking themselves: “Will AI make me worse?”

Subscribe now

As always, the art for the thumbnail, podcast cover, and banner of Wading Through AI were all hand-drawn by Anna Rettberg in her signature style. We do not currently use generative AI here at Molly Rocket.

At last year’s inaugural Better Software Conference, I gave a presentation called The Big OOPs. One of the central figures in the presentation was programmer-designer Marc “Mahk” LeBlanc, whom I credited with the key phrase, “a compile-time hierarchy that matches the domain model”.

That quote, and the chronology of Looking Glass’s entity systems, both came from an extended interview I did with Mahk as part of my research for the talk. At the end of that interview, Mahk and I both had the same thought: “We should have recorded this!” There was so much Looking Glass history, nicely focused specifically on the evolution of their entity systems, that it seemed a shame nobody would get to hear it.

So this year, we did the only logical thing: we both pretended we had never had the conversation, and conducted the whole interview again. This time, we recorded it.

And here it is! The full two-and-a-half-hour interview, covering everything from Mahk’s first encounter with Looking Glass1 during Ultima Underworld, up through his work on the core engine architecture for Thief: The Dark Project.

Bonus Unboxing

Because we could not find any corroboration posted on the internet, Mahk also unboxed a still-shrink-wrapped, original boxed copy of Flight Unlimited so we could verify it did, in fact, ship with a batch file that restarted the game to fix C++-induced memory fragmentation:

After installing the game directly from the original CD, Mahk found the following batch file used to run the game:

@echo off
set dos4g=quiet
Y:\cdrom\flight.exe %1 %2 %3 %4 %5 %6 %7 %8 %9
:wacky
if errorlevel 44 goto done
if errorlevel 43 goto refuel
if errorlevel 42 goto credit
goto done
:credit
Y:\cdrom\credits.exe
Y:\cdrom\flight.exe +fromcredits %1 %2 %3 %4 %5 %6 %7 %8 %9
goto wacky
:refuel
Y:\cdrom\flight.exe +refuel %1 %2 %3 %4 %5 %6 %7 %8 %9
goto wacky
:done

To quote Mahk, “If anything marks this as a genuine LG2 product, it’s the use of ‘wacky’ as a goto label.”

Mahk Links

I hope you enjoy the interview! If you’d like to check out what Mahk’s been up to since the Looking Glass days, you can find him at the following links:

• Mahk’s YouTube: https://www.youtube.com/@algorithmancyTube

• Mahk’s Twitch: https://www.twitch.tv/algorithmancy/videos

• Mahk’s game design website: https://8kindsoffun.com

Until next time, have fun programming everyone, and I (and Mahk!) will see you on the internet.

— Casey

This is the first in a series of Big OOPs supplemental materials that I will be posting over the next several weeks. In addition to this interview with Mahk, there will be paper reviews of each of the major tranches that I used to reconstruct the history presented in the talk. If you’d like to get those delivered to your inbox when they’re published, you can take a look at our subscription options here:

Subscribe now

This is the twelfth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [0:00:03] “I would like to learn/understand more about performance oriented architecture design as most of the performance resources I see are on pinhole omptimizations, do you have specific examples and general guidelines for that?”

• [0:05:10] “What are the architectural considerations to allow the compiler to SIMDify your code?”

• [0:08:13] “You once mentioned that heaps are slow in real life. Could you elaborate on why and which algorithms may be used instead in appropriate situations?”

• [0:11:09] “I saw some advice outside of this course which was to try multiple overlapped reads at once and bypassing the file cache with FILE_FLAG_NO_BUFFERING. Trying this with 32 overlapped reads and reading the whole file into one big buffer (no chunking), I’m now reading at 6 GB/s. This is in line with my SSD read speed (7GB/s allegedly), but not all the numbers are making sense to me. I have two questions:

  How can bypassing the file cache and reading from SSD possibly be faster than reading from the file cache in memory? My main memory is 64GB so I imagine my whole file fits in the cache.

  Regardless of overlapped reads or bypassing the file cache, if I read the whole file into one big buffer, shouldn’t I be paying the cost of page faults regardless? I’d expect page faults and cache inefficiency would limit me to the original 2.5GB/s.”

• [0:19:32] “With block interleaving, we are doing extra loads/stores, how much more expensive is that in terms of power consumption? does it make sense to do the in order interleaving on some kind of mobile device specifically for battery life or to help maintain higher boost clocks via lower temperatures?”

• [0:21:54] “At my job we use Unreal and I find myself writing very defensive code. There is so much “if object exists, do thing, else, log an error and return” across the code base, it’s starting to feel like a waste of time. Especially since in a large number of cases, an early return still leaves the game in an unplayable state. The problem isn’t fixed; only moved elsewhere.

  Do you have any thoughts on when one should do this kind of granular error checking, when one should assert and when it’s okay to just let it crash?

  Is there a better way or do I just have to keep adding all this noise to my code?”

• [0:25:40] “Is it fine to expose internals of a struct together with “helper functions” to calculate derived data? Should I not worry about exposing internals? Any pointers to where I can learn more about library design in C?”

• [0:33:31] “In one QA you said that JIT and compile time are not mutually exclusive, but why I never seen some software written in a “JIT-less” language and use a JIT library to fill the compile time gap performance?”

• [0:36:40] “Something that its still not clear to me is: how do I choose a RAM frequency when buying a PC?”

• [0:40:09] “You said somewhere that you are not a fan o End To End (E2E) tests, why is that?”

• [0:45:03] “I’m curious about what to do when a procedure has only one use in the code? How to conciliate this with semantic compression? Assuming that you can only compress something if it was written at least twice. Because sometimes, even though its used once, it makes easier to read.”

• [0:47:26] “I was wondering: is a relational DB also the 35 year mistake? Since its a statically typed hierarchy that models the domain. My gut feeling is that its not, but it fits very well into the description.”

• [0:49:32] “In this article the author cites some research papers about how the SOLID principles improve software: https://florian-kraemer.net/software-architecture/2025/02/24/Are-the-SOLID-Principles-problematic.html and related to that, the book Agile software development: Principles, Patterns and Practices by Bob Martin, also tries to give a measurement ‘framework’ for SOLID. Any thoughts on this?”

• [0:53:53] “In OO code is common to use objects with private data and some methods and constructor to keep the internal data always in a valid state, but in a procedural way, like C, you can misuse a struct. How do you deal with this in C or in procedural way in general?”

• [0:59:14] “Do you think that is worth creating content like the “OOPs” talk, “Clean Code Horrible Performance”, “Where does bad software come from?” (SOLID) where it tries to persuade the viewer that the “best practice” he was taught is bad? Because even though you are very sharp in your reasoning, your content still get misinterpreted for several reasons. I was thinking of doing something similar myself, grouping everything I learned from the “Handmade way” to present a better way to program, but I’m not sure if its worth it and I don’t have decades of experience so I’m afraid my argumentation would not be good enough.”

• [1:05:04] “Related to the question above, how do I convince my coworkers that they are doing bad practices?”

• [1:06:35] “How exactly do I apply WARMED at work?”

• [1:08:59] “In the end of your IMGUI talk, you said that was excited to see what would be improved in this approach in the next years. So, after 20 years, which improvements you saw?”

• [1:11:31] “Why compilers don’t have a #pragma DONT_REMOVE, #pragma END_DONT_REMOVE to make our lives easier instead of writing assembly code to do fake reads and writes for measuring performance for a block of code?”

• [1:13:13] “I’m curious on what a web dev handmade hero would look like, because in the game version is possible to ship a commercial game without using any libraries, however I don’t think that is possible with web, since you need some libraries to deal with security/encryption like HTTPS. So how would you approach this if you were to make a handmade hero for web? Like showing how to create a substack clone, but better :)”

• [1:14:59] “Knowing the specs of the Lion Cove P-Core of that CPU with 3x load units/ports it makes sense that there is no additional speedup after Read_x3. What puzzles me a bit/makes me curious though is the fact that Read_x3 is consistently the fastest. I tried it numerous times, locked the program to the same single P-core, built dedicated executables to make sure it has nothing to do with some code alignment etc. but Read_x3 is always the fastest with some ~2% edge over Read_x4. Of course 2% is not a huge difference, but I would have assumed that Read_x3 and Read_x4 would converge more or less or if at all that Read_x4 would be slightly faster due to less loop overhead in total.”

• [1:19:22] “Now that you’re mainly on Linux, have you been using any of the profiling tools that Linux provides in <linux/perf_event.h> and if you do, what are your thoughts on them?” / “In a recent standup I heard how you are now fully switched to Linux and never going back to Windows. I would like to know, what are you now then using for debugging on Linux? As raddbg is not yet ready, though getting closer…”

Read more

A few days ago, Ramon “Ray” Santamaria, author of the popular library Raylib, posted the following code snippet and question on X:

As you can see from the screenshot, GCC and MSVC x64 appear to create binaries that behave “correctly” - or at least not obviously wrong - whereas MSVC x86 appears to produce a binary that behaves “incorrectly”.

Although many people replied to Ray’s post by saying, “it’s undefined behavior” - which is true, because there’s a missing null terminator - that’s not a complete answer. That only says why the execution is allowed to be different, not how it actually differs.

I thought this was a great opportunity to show how we would investigate behavior like this to determine the root cause. I’ve prepared two videos (and some “outtakes” with esoterica) that show this process.

In this first video, I walk through the entire “source language” side of the behavior to show what is actually causing it. I will be posting a second video shortly for people who can read ASM which shows how to go one layer deeper and look at an even more precise answer.

If you’d like to be emailed when the second video is posted, you can subscribe your email address to either our free or paid tiers using the button below.

Subscribe now

This is the eleventh video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The legal issues surrounding generative AI are something only the court system can decide. The ethical issues, on the other hand, are things that everyone must decide for themselves. Different people with different value systems will come to different conclusions.

In this episode, Demetri and I open the can of worms that is “AI ethics” and walk through a series of incremental steps people should think through when deciding for themselves if they think generative AI is ethical.

Subscribe now

As always, the art for the thumbnail, podcast cover, and banner of Wading Through AI were all hand-drawn by Anna Rettberg in her signature style. We do not currently use generative AI here at Molly Rocket.

The questions addressed in this video are:

• [00:04] “Quick question (for Casey or anyone with a suggestion!): I am neither a novice nor an experienced programmer, and would like to learn enough C++ to write some useful code, but I am struggling to find good material.”

• [02:18] “In the episode ‘In-order Interleaving’, you have a loop that handles multiple elements per loop iteration. When the list of elements isn’t a multiple of your ‘elements per loop’, we need something after the loop to handle the last few remaining elements. My question is: Is there a particular way to write this ‘residual handling’ part? Whenever I’ve had to do this in my own code it always felt a little awkward.”

• [13:25] “Hey Casey, what’s your take on Arm making their own chips now? Sincerely, an Arm engineer.”

• [17:05] “On the topic of growable arenas, concretely, how do you implement them with respect to the ‘layered’ architecture? To grow them, naturally you would need to mmap or VirtualAlloc more memory, but if you’re far removed from the platform layer and aren’t writing a program that is frame-based (for example, a compiler) and thus have no opportune time to go back to the platform layer and request more memory, what are your strategies? I can only come up with round-tripping back to the OS using a platform-layer API like ‘RequestMoreMemory()’, or something like this. For more context, this would be for the kind of program that also cannot place upper-bounds on memory usage (again, like in a compiler), but still does not want to be malloc’ing/free’ing excessively. I’m keen to hear how you would approach this problem?”

• [26:14] “Have you had occasion to use coroutines to write algorithms that must pause/resume? E.g., protocols in networking or software in a hardware simulator.”

Read more

When it comes to AI, people seem to be having very disparate experiences. Some people become enthusiastic, while others become frustrated, even when apparently attempting similar tasks.

In this episode of Wading Through AI, Demetri and I address the root causes of the feeling a lot of people - both pro- and anti-AI - are having that either they must be crazy, or everyone else must be, because their experiences seem to differ so dramatically from what they are hearing from others.

Subscribe now

As always, the art for the thumbnail, podcast cover, and banner of Wading Through AI were all hand-drawn by Anna Rettberg in her signature style. We do not currently use generative AI here at Molly Rocket.

This is the tenth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

Technically, it was dated April 1st, so it may all have been a ruse! But April Fools joke or not, the vast majority of replies took it seriously.

While anyone can subjectively agree or disagree with Notch on the value of DLSS, there’s an underlying objective question regarding his use of the phrase “THE SAME HARDWARE”. Many replies took issue with this wording, claiming it was a clear sign Notch did not understand how DLSS worked.

I’m not so sure about that.

I think there is a good case to be made that - given what little NVIDIA tells us - Notch is not objectively wrong about NVIDIA hardware. I didn’t feel like a contorted X post would be the best way to elaborate on this, so I livestreamed my thoughts on Twitch, YouTube, and here on Substack.

The video is included above for those of you that missed the stream.

In this second episode of Wading Through AI, Demetri and I discuss the recent Anthropic blog post about “parallel Claudes” building a C Compiler, and use it as a case study for how to assess AI announcements.

Subscribe now

As always, the art for the thumbnail, podcast cover, and banner of Wading Through AI were all hand-drawn by Anna Rettberg in her signature style. We do not currently use generative AI here at Molly Rocket.

The questions addressed in this video are:

• [0:00:03] “Building on using free lists to manage objects with complex lifetimes, how do you handle cases where an object type can vary in size?”

• [0:02:33] “Can you go into more detail about how to handle IO with queues vs callbacks? A small example on the fancy light-board would be nice.”

• [0:07:14] “With the recent widespread usage of AI to program in the industry and a fierce push from upper management to enforce usage of AI in development tasks, I am finding myself less and less motivated to program, and also depressed. I have mostly ceased my programming activities outside of working hours.

  Do you have any advice on how to deal with the AI wave and also to continue motivated to program as a hobby and professionally in this environment?”

• [0:20:00] “Now that you are switching to Linux in full swing have you ported your codebases or what’s your strategy? I did the opposite thing and fled from Linux because I wanted to start my first big codebase in a more stable environment so it has been the year of the Windows desktop for me which after running a debloat script hasn’t actually been bad, nevertheless seeing Microsoft’s behavior does have me yearning for the penguin, and knowing how painful releasing something for Linux is has me wondering if programming against the windows API and using Wine for most of one’s graphical software is a valid way to proceed.”

• [0:29:50] “How much of the software you currently write is based on your own thoughts and creativity? I’ve found myself relying heavily on reading and using open-source projects where similar problems were already solved. After a while, this started to make me feel like an impostor. Do great programmers like yourself write their own code, or do they adapt and build on existing work?”

• [0:35:38] “This maybe jumping the gun a fair bit. but when you say back of the envelop calculation, does that include the time complexity of whatever algorithm to do correctly?”

• [0:39:45] “Have you ever written anything in ISPC? From my experience it seems much easier to write than rolling the simd myself, and much more portable.”

• [0:42:10] “Do you have any recommendations on how to use ‘pen & paper’ in the software development process? I’ve noticed that drawing a problem helps me visualize the problem easier than jumping straight into coding. I’m working with MIDI and writing a document explaining in my own words how does MIDI encode data helped me in unexpected ways. I’m wondering if I’m missing out on any other ‘pen & paper’ practice. Any useful tips?”

• [0:46:47] “Hello I come with another question. How does one go about implementing smooth window resizing with Dangerous Thread Crews? I have the two threads and do get non-blocking resizing but the UI contents of the window jiggle and stretch a bit when resizing (which gets way worse with vsync) and nothing I try (like adding synchronization and using WM_WINDOWPOSCHANGING to allow only one resize per frame) gets rid of the apparent discrepancy between the window size and the size my renderer targets that causes some ugliness. The only reference I have of a program that has smooth resizing and doesn’t achieve this by updating on each WM_PAINT is your refterm program but I don’t see any glaring differences between the way you rendered stuff there and it seems like there you just called GetClientRect on each update with no need for anything else... so am I just doing something dumb and missing something or is there some computer wizardry I need to invoke to fix this when programming more complex UIs?”

• [0:51:06] “In this talk about pathfinding in Age of Empires 2, around the 15-16 minute mark, he mentions turning off SIMD because they were losing floating point precision. I don’t understand why that was the case for them, can you provide more insight?”

• [1:01:33] “I’ve been trying to test how many simultaneous loads can I get with the repetition tester when I do 1, 2, 3 or 4 moves in a loop on the two machines I have access to. My laptop (which has Intel Skylake chip) reports that the memory bandwidth doubles only when I go from 1 to 2 moves per loop, which is expected and reproduces your results from the course. But when I do the same on my desktop (which has Intel Raptor lake chip), the results are the same. The Raptor lake apparently have two type of cores: P-cores and E-cores, where E-cores also have 2 ports capable of executing loads, while P-cores are supposed to be equipped with 3 ports of that type (at least that’s what I read in Agner Fog manual). To my understanding it means that I should see a bandwidth bump when going from 1 to 2 moves per loop, and when going from 2 to 3 moves per loop. But that doesn’t happen, I see only one bump (from 1 to 2 moves). I guess that there are some nuances with running the tester on this system that I might not be aware of. But one of them - which is clear to me - is that it is the OS who decides on which core should the tester be run on. So I set the affinity of the tester to CPU1 to make sure that it runs on the P-core. Process Explorer confirmed that it runs on CPU1. But I still could not see the improvement when going from 2 to 3 memory reads. Then I repeated the test with all the cores (one at a time), but I saw no difference in the results.

  It is either my test setup that’s completely broken, or some other factor that I can’t see which prevents the bandwidth improvement of a 3 reads loop. I would be grateful if you could give me some pointers here.”

• [1:07:25] “What are your thoughts about the new dynamicdeopt in msvc that lets you run optimized builds that you can debug with full information because they switch the executable on the fly?”

• [1:08:15] “I’d like to ask you about the pass-by-ref vs pass-by-value ‘debate’. Traditional C++ advice is ‘always pass by const& anything bigger than 8 bytes’, but I’ve recently started seeing some people advocate that 16 byte structs should also be pass-by-value. I know that you couldn’t care less about const, and you do seem to pass by value small stuff without worrying too much about it in your own code, so .. is there anything I’m failing to consider here? is this a stupid thing to worry about in the abstract, or is there some general principle that could be useful to keep in mind here?”

Read more

A few weeks ago, I was approached by long time AI-researcher and personal friend Demetri Spanos about producing a discussion series on emerging AI topics.

Since Demetri has been working on AI technology for decades, now that it’s the hot topic, he regularly gets bombarded with questions from friends who want an opinion from the inside. The idea for the series was to see if we could get those answers out to a wider audience.

As most of you already know, I don’t work with AI myself, so I’m really looking forward to these discussions. Demetri is exceptionally good at avoiding hyperbole, and does a fantastic job providing the technical context you need to draw your own conclusions. As you’ll see, he never tries to sell a specifically positive or specifically negative view of AI, which is very refreshing compared to the normal AI discourse.

I hope you enjoy this first discussion as much as I did. If you’re a subscriber, and have questions or topics you’d like us to cover on future episodes, please feel free to leave a suggestion in the comments! I will be checking regularly to see what our readers are most interested in.

Subscribe now

As always, the art for the thumbnail, podcast cover, and banner of Wading Through AI were all hand-drawn by Anna Rettberg in her signature style. We do not currently use generative AI at Molly Rocket.

This is the ninth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

This past Wednesday, one of the topics was “the DRAM shortage”. I thought it might be nice to go through the different types of DRAM and talk about where they are made, just to give some context to the global supply situation (or lack thereof). I livestreamed it on Twitch, YouTube, and of course, right here on Substack.

Read more

The questions addressed in this video are:

• [01:02] “Could you suggest a resource to learn how to work with large datasets that don’t fit into VRAM, or even regular RAM?”

• [07:24] “I do not know how this compares to other approaches, as I’ve not really tried to optimize my profilers much as it’s not really my main area of work, but your profiling explanation made me think of a lock-debug profiler I built. It sounds similar to me at least. Do you agree?

  In my solution, each worker thread owns its profiling buffers completely. I keep a thread-local pointer to a block-based buffer that grows in chunks when needed, and all writes are done by the thread with no locks. The only shared step is that when a thread creates its first block, it registers it once by pushing it into a global linked list so the UI can later iterate all threads. To avoid stalling the writers, I double-buffered it. Each thread allocates two buffers and writes into one of them.”

• [09:46] “About memory usage and the program stack, I believed that this stack could more easily fit in the L1 cache. Isn’t there a higher risk of cache misses at that level if the data to use has been allocated elsewhere? Would it even be noticeable in terms of performance?”

• [14:32] “What’s the best way to multithread a software rasterizer? In my experience, scan line interlacing per triangle was horrific.”

• [18:18] “Do you know why in the file processing test on the same machine on linux mmaping the file could be outperforming all other methods on medium and large mapped chunk sizes, and on windows the same test was worse than everyone else?”

• [21:18] “Regarding callbacks, I think I understood the argument about moving things to a queue instead. However, I also can see the benefit of a callback because that’s synchronous. What if the file IO read some data and stored in a buffer, and the callback now is supposed to handle that read data. If instead you move things to a queue, you’d have to keep allocating new buffers to keep working on those asynchronous reads, because you don’t know when the client code will handle those reads. If instead you can very quickly handle this buffer of read data synchronously in the callback, the file IO can reuse the same buffer to store the next chunk.”

• [25:45] “Can you steelman cases for which an arena/bump allocator (are these the same thing?) is not the preferred way to allocate memory (I imagine it is when lifetimes are not apriori known, but perhaps I am missing more subtlety)? In such cases, what is your preferred method of allocation? Are you forced to go back to new/delete?”

• [33:01] “Re: Dead Code Elimination Prevention Macros, I’ve got identical results with `asm volatile (”“ : “+v”(Value));` which tells the compiler that `Value` is both input and output, forcing it to initialize it, as well as preventing from assuming a specific value. It would generate `vpxor` for `0.0` and `vmovaps` for `0.5`. The advantage here is that it’s quite generic and doen’t depend on operand size/type.Could you please elaborate on your choice of explicitly using instructions?”

• [35:14] “Concerning callbacks, is there a benefit to use callbacks for print-outs/messaging? We have a part of the program that do some calculations which can take time, and it uses callbacks to notify the user about the progress, and waiting to the end of all the computations is not good enough. We also use a callback for a type of mesh calculation that depends on things that this other part of the program shouldn’t know about. (These things were not my decision, but I guess making the module as isolated as possible makes it easier to use it as a module in another program)”

• [37:42] “How do you determine if a solution to a problem is more complex than it needs to be? And for inherently complex and interconnected problems, how do you determine if it needs to be subdivided or not? Is there a general approach for working on a complex system with lots of moving parts? (other than me complaining about it :) )”

Read more

To elaborate, in the official RCA, you will see a lengthy explanation of the state that triggers the bug, as well as the cascading series of failures that took down much of AWS. However, the bug itself is described only by the following single sentence:

Additionally, because the active plan was deleted, the system was left in an inconsistent state that prevented subsequent plan updates from being applied by any DNS Enactors.

That’s not much to go on.

Earlier this month, my hopes for a more complete explanation were bolstered by the sudden appearance of an AWS re:Invent presentation devoted entirely to this particular outage. Sadly, in the hour-long video, only twelve seconds were spent on the actual bug:

Both the RCA and the presentation do a good job of explaining the conditions that triggered the bug. Assuming I understood correctly, the setup goes like this:

1. DynamoDB uses multiple DNS “enactors”, running concurrently, to update DNS records for load balancing (among other things).

2. Enactors use locking and backoff to partially serialize their behavior and make them “easier to reason about”1.

3. Pathological lock contention can cause one enactor to be “enacting” a series of DNS records so old that another enactor will assume they couldn’t still be in use, and try to delete them.

4. An “inconsistent state” occurs when one enactor deletes DNS records it thought were sufficiently old while another enactor updates a DNS endpoint to refer to those same records as authoritative.

5. Enactors will stop operating if they attempt to update a DNS endpoint when it does not already refer to a resolvable name (which it won’t once the “inconsistent state” has occurred).

After all that setup, the story leaves me hanging. To quote the twelve seconds of the presentation:

… but when that plan’s missing, our code couldn’t handle the fact that we tried to point the rollback to nothing …

To me, this is the actual bug, so I wanted to hear about it in detail. Why didn’t the code handle that? Or more specifically, why would the code even need to do anything special to “handle” that case at all?

Let me elaborate. Whenever I’m reading something like an RCA, I’m building a model of the system in my head. I’m sure many of you do this as well - it’s how programmers think through programming problems as they are being described.

My mental model of the AWS system at this point doesn’t yet have a bug. If I imagine myself implementing the code they’re describing here, the pseudo-code version is really just three steps:

1 get the current value of dynamodb.us-east-1.api.aws
  ("plan-110.ddb.aws" in the presentation example)
2 set rollback.ddb.aws to the value from step 1
3 set dynamodb.us-east-1.api.aws to the new value
  ("plan-145.ddb.aws" in the presentation example)

When written in the straightforward way, there should be no issue with step 1 receiving an unresolvable name - some kind of “null”, garbage like 0.0.0.0, or what I assume would have happened in the actual outage: the “old” name whose record had been deleted (shown as plan-110.ddb.aws in the example from the re:Invent presentation).

As far as we’ve been told, performing these DNS record modifications does not require the enactor to actually communicate with any target load balancers, so it shouldn’t matter if the target DNS records exist or not. The rollback entrypoint (rollback.ddb.aws) would just be set to the unresolvable name, just as dynamodb.us-east-1.api.aws had an unresolvable name prior to this update. As far as the explanation goes, the enactors shouldn’t have cared whether these names were resolvable or not.

Additionally, in step 3, the endpoint (dynamodb.us-east-1.api.aws) would be updated to a valid name. Everything except debug code that looks at rollback.dds.aws would work properly had the enactor continued running. On the next update, literally everything - even rollback.ddb.aws - would be back to normal. It would then point to plan-145.ddb.aws, which remains valid and will exist for some time before being cleaned up.

So to me, we have not yet hit the root cause! What is it? Something in the way Amazon wrote the enactor code is the actual root cause. I want to know what it is.

To underscore why I think this is important: taking the explanations at face value, DynamoDB could have been taken down in exactly the same way - causing the same catastrophic outage - simply by manually deleting the target DNS record of a DynamoDB endpoint. According to the available explanation, just that action alone would cause all enactors to cease functioning, and the endpoint would remain an unresolvable name forever.

So it’s hard to call any of the “setup” the root cause, because none of it is actually required for the exact same outage. An accidental manual deletion of an endpoint’s target - even just once - would bring the entire system down, and it would be unrecoverable without operator intervention.

Although the rare, emergent appearance of the “inconsistent state” is something that you would want to fix as well, by itself it wouldn’t have caused AWS to go down catastrophically. At worst it sounds like it would have been a momentary outage. If the enactors weren’t halted by the inconsistent state, one of them would shortly enact a new DNS plan, and things would be back to normal.

In closing, I should also mention that my interest in this is more than idle curiosity. A lot of programming paradigms today rely on architectural patterns I consider dangerous for mission-critical systems. As described in the RCA and presentation, this bug sounds like one that wasn’t really caused by a latent race condition as claimed - it was merely exposed by it. Given the disclosures so far, it sounds far more likely that this bug was actually caused by a bad programming practice in a piece of practical code that communicated with Route53 to change DNS entries.

So don’t leave me hanging, DynamoDB! Tell me what the actual code looked like that took down the internet last October. I want to know how something as basic as exchanging a DNS record was written in such a way that it could permanently halt not one but three different concurrent processes on a case as simple as an unresolvable prior record value.