For those wondering what the name of the audio API was that made me go “I can’t believe I am forgetting the acronym” in the middle of this video, it was ASIO.

The questions addressed in this video are:

• [00:02] “The D language actually has this because of Uniform Function Call Syntax. You could have a struct like … and have the rest of the code be none the wiser (even code like foo.x = 7). I think C# has stuff like this too. Is this what you’d be hoping for? If so, I’m surprised - I’d have guessed you usually don’t prefer implicit magic like this happening on things that don’t look like function calls.”

• [04:37] “Could someone remind me what the ‘warm’ criteria was?”

• [06:00] “I have a question about managing intrusive thoughts while writing code. When thoughts come up about edge cases, performance, reliability, or decisions that may need to be revisited later, how do you decide whether to address them immediately or set them aside? I’m finding it difficult to iterate quickly because I often get pulled into these concerns before I’ve built a working v0. I’ve tried writing TODOs in the code and keeping separate notes, but neither approach has worked well for me.”

• [15:47] “I now have a question about wasapi (I believe you now use wasapi but if I’m wrong then you can ignore this question). I’m writing the sound system for my codebase and had my wav parser ready, mixer chugging along, sat back to enjoy a song but then found out that changing the system volume causes these pops.... I know that changing the volume of a sound wave causes the discontinuity which manifests itself as audio popping but in this case where it’s not an internal volume I have control over I don’t know how to fix these. It seems to me as if windows just immediately applied the new volume and my futile attempt of trying to bypass windows’ volume logic didn’t work since they seem to first clamp my samples and then multiply by the volume... I tried some wasapi gist examples and they all had this same issue. Have you had this problem and fixed it in your own code and could you share your knowledge if you have?”

• [25:07] “I formed my question about heaps slightly misleading. It wasn’t about a memory heap, but a data structure, like in the heap sort. In the video together with Primeagen regarding code interviews you commented that heaps are quite slow. So I was wondering, what should you use instead, when the need arises?”

• [33:27] “I’m working on a project that uses a cortex-m7, a fairly fast 32-bit embedded core. It has some fancy pipeline features of big boy desktop chips, like a branch predictor, and the ability to dual issue many instructions, but doesn’t have a RAT (it’s not “out-of-order” at least as far as I know). Do I understand correctly that in CPUs without a RAT, any two adjacent instructions that use the same register are effectively serially dependent? Is this how it used to work in the desktop world back in the day?”

• [40:53] “This question relates to Part 5: Dependency Chain Stalls and In-order Interleaving. From these results, I would expect 12-way interleaving to maintain peak throughput regardless of chain length (4 FMAs/cycle * 3 cycles = 12 FMAs in flight needed to cover the latency). However, that’s not what I observe. With 12-way interleaving, throughput decreases from ~3.6 to ~3.0 FMAs/cycle as the chain length grows. I need a minimum of 16-way interleaving for throughput to stay flat at ~3.8 FMAs/cycle. My conclusion is that the CPU can’t sustain the 3-cycle latency when the FMA execution ports are saturated, resulting in an effective latency of 4 cycles under load. This would explain why I need 4 * 4 = 16 chains to obtain a flat throughput curve. I think you’ve hinted at this in some of your videos. Would you kindly explain why this happens? Thanks!”

Read more

This is the twelfth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [0:00:03] “I would like to learn/understand more about performance oriented architecture design as most of the performance resources I see are on pinhole omptimizations, do you have specific examples and general guidelines for that?”

• [0:05:10] “What are the architectural considerations to allow the compiler to SIMDify your code?”

• [0:08:13] “You once mentioned that heaps are slow in real life. Could you elaborate on why and which algorithms may be used instead in appropriate situations?”

• [0:11:09] “I saw some advice outside of this course which was to try multiple overlapped reads at once and bypassing the file cache with FILE_FLAG_NO_BUFFERING. Trying this with 32 overlapped reads and reading the whole file into one big buffer (no chunking), I’m now reading at 6 GB/s. This is in line with my SSD read speed (7GB/s allegedly), but not all the numbers are making sense to me. I have two questions:

  How can bypassing the file cache and reading from SSD possibly be faster than reading from the file cache in memory? My main memory is 64GB so I imagine my whole file fits in the cache.

  Regardless of overlapped reads or bypassing the file cache, if I read the whole file into one big buffer, shouldn’t I be paying the cost of page faults regardless? I’d expect page faults and cache inefficiency would limit me to the original 2.5GB/s.”

• [0:19:32] “With block interleaving, we are doing extra loads/stores, how much more expensive is that in terms of power consumption? does it make sense to do the in order interleaving on some kind of mobile device specifically for battery life or to help maintain higher boost clocks via lower temperatures?”

• [0:21:54] “At my job we use Unreal and I find myself writing very defensive code. There is so much “if object exists, do thing, else, log an error and return” across the code base, it’s starting to feel like a waste of time. Especially since in a large number of cases, an early return still leaves the game in an unplayable state. The problem isn’t fixed; only moved elsewhere.

  Do you have any thoughts on when one should do this kind of granular error checking, when one should assert and when it’s okay to just let it crash?

  Is there a better way or do I just have to keep adding all this noise to my code?”

• [0:25:40] “Is it fine to expose internals of a struct together with “helper functions” to calculate derived data? Should I not worry about exposing internals? Any pointers to where I can learn more about library design in C?”

• [0:33:31] “In one QA you said that JIT and compile time are not mutually exclusive, but why I never seen some software written in a “JIT-less” language and use a JIT library to fill the compile time gap performance?”

• [0:36:40] “Something that its still not clear to me is: how do I choose a RAM frequency when buying a PC?”

• [0:40:09] “You said somewhere that you are not a fan o End To End (E2E) tests, why is that?”

• [0:45:03] “I’m curious about what to do when a procedure has only one use in the code? How to conciliate this with semantic compression? Assuming that you can only compress something if it was written at least twice. Because sometimes, even though its used once, it makes easier to read.”

• [0:47:26] “I was wondering: is a relational DB also the 35 year mistake? Since its a statically typed hierarchy that models the domain. My gut feeling is that its not, but it fits very well into the description.”

• [0:49:32] “In this article the author cites some research papers about how the SOLID principles improve software: https://florian-kraemer.net/software-architecture/2025/02/24/Are-the-SOLID-Principles-problematic.html and related to that, the book Agile software development: Principles, Patterns and Practices by Bob Martin, also tries to give a measurement ‘framework’ for SOLID. Any thoughts on this?”

• [0:53:53] “In OO code is common to use objects with private data and some methods and constructor to keep the internal data always in a valid state, but in a procedural way, like C, you can misuse a struct. How do you deal with this in C or in procedural way in general?”

• [0:59:14] “Do you think that is worth creating content like the “OOPs” talk, “Clean Code Horrible Performance”, “Where does bad software come from?” (SOLID) where it tries to persuade the viewer that the “best practice” he was taught is bad? Because even though you are very sharp in your reasoning, your content still get misinterpreted for several reasons. I was thinking of doing something similar myself, grouping everything I learned from the “Handmade way” to present a better way to program, but I’m not sure if its worth it and I don’t have decades of experience so I’m afraid my argumentation would not be good enough.”

• [1:05:04] “Related to the question above, how do I convince my coworkers that they are doing bad practices?”

• [1:06:35] “How exactly do I apply WARMED at work?”

• [1:08:59] “In the end of your IMGUI talk, you said that was excited to see what would be improved in this approach in the next years. So, after 20 years, which improvements you saw?”

• [1:11:31] “Why compilers don’t have a #pragma DONT_REMOVE, #pragma END_DONT_REMOVE to make our lives easier instead of writing assembly code to do fake reads and writes for measuring performance for a block of code?”

• [1:13:13] “I’m curious on what a web dev handmade hero would look like, because in the game version is possible to ship a commercial game without using any libraries, however I don’t think that is possible with web, since you need some libraries to deal with security/encryption like HTTPS. So how would you approach this if you were to make a handmade hero for web? Like showing how to create a substack clone, but better :)”

• [1:14:59] “Knowing the specs of the Lion Cove P-Core of that CPU with 3x load units/ports it makes sense that there is no additional speedup after Read_x3. What puzzles me a bit/makes me curious though is the fact that Read_x3 is consistently the fastest. I tried it numerous times, locked the program to the same single P-core, built dedicated executables to make sure it has nothing to do with some code alignment etc. but Read_x3 is always the fastest with some ~2% edge over Read_x4. Of course 2% is not a huge difference, but I would have assumed that Read_x3 and Read_x4 would converge more or less or if at all that Read_x4 would be slightly faster due to less loop overhead in total.”

• [1:19:22] “Now that you’re mainly on Linux, have you been using any of the profiling tools that Linux provides in <linux/perf_event.h> and if you do, what are your thoughts on them?” / “In a recent standup I heard how you are now fully switched to Linux and never going back to Windows. I would like to know, what are you now then using for debugging on Linux? As raddbg is not yet ready, though getting closer…”

Read more

This is the eleventh video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [00:04] “Quick question (for Casey or anyone with a suggestion!): I am neither a novice nor an experienced programmer, and would like to learn enough C++ to write some useful code, but I am struggling to find good material.”

• [02:18] “In the episode ‘In-order Interleaving’, you have a loop that handles multiple elements per loop iteration. When the list of elements isn’t a multiple of your ‘elements per loop’, we need something after the loop to handle the last few remaining elements. My question is: Is there a particular way to write this ‘residual handling’ part? Whenever I’ve had to do this in my own code it always felt a little awkward.”

• [13:25] “Hey Casey, what’s your take on Arm making their own chips now? Sincerely, an Arm engineer.”

• [17:05] “On the topic of growable arenas, concretely, how do you implement them with respect to the ‘layered’ architecture? To grow them, naturally you would need to mmap or VirtualAlloc more memory, but if you’re far removed from the platform layer and aren’t writing a program that is frame-based (for example, a compiler) and thus have no opportune time to go back to the platform layer and request more memory, what are your strategies? I can only come up with round-tripping back to the OS using a platform-layer API like ‘RequestMoreMemory()’, or something like this. For more context, this would be for the kind of program that also cannot place upper-bounds on memory usage (again, like in a compiler), but still does not want to be malloc’ing/free’ing excessively. I’m keen to hear how you would approach this problem?”

• [26:14] “Have you had occasion to use coroutines to write algorithms that must pause/resume? E.g., protocols in networking or software in a hardware simulator.”

Read more

This is the tenth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [0:00:03] “Building on using free lists to manage objects with complex lifetimes, how do you handle cases where an object type can vary in size?”

• [0:02:33] “Can you go into more detail about how to handle IO with queues vs callbacks? A small example on the fancy light-board would be nice.”

• [0:07:14] “With the recent widespread usage of AI to program in the industry and a fierce push from upper management to enforce usage of AI in development tasks, I am finding myself less and less motivated to program, and also depressed. I have mostly ceased my programming activities outside of working hours.

  Do you have any advice on how to deal with the AI wave and also to continue motivated to program as a hobby and professionally in this environment?”

• [0:20:00] “Now that you are switching to Linux in full swing have you ported your codebases or what’s your strategy? I did the opposite thing and fled from Linux because I wanted to start my first big codebase in a more stable environment so it has been the year of the Windows desktop for me which after running a debloat script hasn’t actually been bad, nevertheless seeing Microsoft’s behavior does have me yearning for the penguin, and knowing how painful releasing something for Linux is has me wondering if programming against the windows API and using Wine for most of one’s graphical software is a valid way to proceed.”

• [0:29:50] “How much of the software you currently write is based on your own thoughts and creativity? I’ve found myself relying heavily on reading and using open-source projects where similar problems were already solved. After a while, this started to make me feel like an impostor. Do great programmers like yourself write their own code, or do they adapt and build on existing work?”

• [0:35:38] “This maybe jumping the gun a fair bit. but when you say back of the envelop calculation, does that include the time complexity of whatever algorithm to do correctly?”

• [0:39:45] “Have you ever written anything in ISPC? From my experience it seems much easier to write than rolling the simd myself, and much more portable.”

• [0:42:10] “Do you have any recommendations on how to use ‘pen & paper’ in the software development process? I’ve noticed that drawing a problem helps me visualize the problem easier than jumping straight into coding. I’m working with MIDI and writing a document explaining in my own words how does MIDI encode data helped me in unexpected ways. I’m wondering if I’m missing out on any other ‘pen & paper’ practice. Any useful tips?”

• [0:46:47] “Hello I come with another question. How does one go about implementing smooth window resizing with Dangerous Thread Crews? I have the two threads and do get non-blocking resizing but the UI contents of the window jiggle and stretch a bit when resizing (which gets way worse with vsync) and nothing I try (like adding synchronization and using WM_WINDOWPOSCHANGING to allow only one resize per frame) gets rid of the apparent discrepancy between the window size and the size my renderer targets that causes some ugliness. The only reference I have of a program that has smooth resizing and doesn’t achieve this by updating on each WM_PAINT is your refterm program but I don’t see any glaring differences between the way you rendered stuff there and it seems like there you just called GetClientRect on each update with no need for anything else... so am I just doing something dumb and missing something or is there some computer wizardry I need to invoke to fix this when programming more complex UIs?”

• [0:51:06] “In this talk about pathfinding in Age of Empires 2, around the 15-16 minute mark, he mentions turning off SIMD because they were losing floating point precision. I don’t understand why that was the case for them, can you provide more insight?”

• [1:01:33] “I’ve been trying to test how many simultaneous loads can I get with the repetition tester when I do 1, 2, 3 or 4 moves in a loop on the two machines I have access to. My laptop (which has Intel Skylake chip) reports that the memory bandwidth doubles only when I go from 1 to 2 moves per loop, which is expected and reproduces your results from the course. But when I do the same on my desktop (which has Intel Raptor lake chip), the results are the same. The Raptor lake apparently have two type of cores: P-cores and E-cores, where E-cores also have 2 ports capable of executing loads, while P-cores are supposed to be equipped with 3 ports of that type (at least that’s what I read in Agner Fog manual). To my understanding it means that I should see a bandwidth bump when going from 1 to 2 moves per loop, and when going from 2 to 3 moves per loop. But that doesn’t happen, I see only one bump (from 1 to 2 moves). I guess that there are some nuances with running the tester on this system that I might not be aware of. But one of them - which is clear to me - is that it is the OS who decides on which core should the tester be run on. So I set the affinity of the tester to CPU1 to make sure that it runs on the P-core. Process Explorer confirmed that it runs on CPU1. But I still could not see the improvement when going from 2 to 3 memory reads. Then I repeated the test with all the cores (one at a time), but I saw no difference in the results.

  It is either my test setup that’s completely broken, or some other factor that I can’t see which prevents the bandwidth improvement of a 3 reads loop. I would be grateful if you could give me some pointers here.”

• [1:07:25] “What are your thoughts about the new dynamicdeopt in msvc that lets you run optimized builds that you can debug with full information because they switch the executable on the fly?”

• [1:08:15] “I’d like to ask you about the pass-by-ref vs pass-by-value ‘debate’. Traditional C++ advice is ‘always pass by const& anything bigger than 8 bytes’, but I’ve recently started seeing some people advocate that 16 byte structs should also be pass-by-value. I know that you couldn’t care less about const, and you do seem to pass by value small stuff without worrying too much about it in your own code, so .. is there anything I’m failing to consider here? is this a stupid thing to worry about in the abstract, or is there some general principle that could be useful to keep in mind here?”

Read more

This is the ninth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [01:02] “Could you suggest a resource to learn how to work with large datasets that don’t fit into VRAM, or even regular RAM?”

• [07:24] “I do not know how this compares to other approaches, as I’ve not really tried to optimize my profilers much as it’s not really my main area of work, but your profiling explanation made me think of a lock-debug profiler I built. It sounds similar to me at least. Do you agree?

  In my solution, each worker thread owns its profiling buffers completely. I keep a thread-local pointer to a block-based buffer that grows in chunks when needed, and all writes are done by the thread with no locks. The only shared step is that when a thread creates its first block, it registers it once by pushing it into a global linked list so the UI can later iterate all threads. To avoid stalling the writers, I double-buffered it. Each thread allocates two buffers and writes into one of them.”

• [09:46] “About memory usage and the program stack, I believed that this stack could more easily fit in the L1 cache. Isn’t there a higher risk of cache misses at that level if the data to use has been allocated elsewhere? Would it even be noticeable in terms of performance?”

• [14:32] “What’s the best way to multithread a software rasterizer? In my experience, scan line interlacing per triangle was horrific.”

• [18:18] “Do you know why in the file processing test on the same machine on linux mmaping the file could be outperforming all other methods on medium and large mapped chunk sizes, and on windows the same test was worse than everyone else?”

• [21:18] “Regarding callbacks, I think I understood the argument about moving things to a queue instead. However, I also can see the benefit of a callback because that’s synchronous. What if the file IO read some data and stored in a buffer, and the callback now is supposed to handle that read data. If instead you move things to a queue, you’d have to keep allocating new buffers to keep working on those asynchronous reads, because you don’t know when the client code will handle those reads. If instead you can very quickly handle this buffer of read data synchronously in the callback, the file IO can reuse the same buffer to store the next chunk.”

• [25:45] “Can you steelman cases for which an arena/bump allocator (are these the same thing?) is not the preferred way to allocate memory (I imagine it is when lifetimes are not apriori known, but perhaps I am missing more subtlety)? In such cases, what is your preferred method of allocation? Are you forced to go back to new/delete?”

• [33:01] “Re: Dead Code Elimination Prevention Macros, I’ve got identical results with `asm volatile (”“ : “+v”(Value));` which tells the compiler that `Value` is both input and output, forcing it to initialize it, as well as preventing from assuming a specific value. It would generate `vpxor` for `0.0` and `vmovaps` for `0.5`. The advantage here is that it’s quite generic and doen’t depend on operand size/type.Could you please elaborate on your choice of explicitly using instructions?”

• [35:14] “Concerning callbacks, is there a benefit to use callbacks for print-outs/messaging? We have a part of the program that do some calculations which can take time, and it uses callbacks to notify the user about the progress, and waiting to the end of all the computations is not good enough. We also use a callback for a type of mesh calculation that depends on things that this other part of the program shouldn’t know about. (These things were not my decision, but I guess making the module as isolated as possible makes it easier to use it as a module in another program)”

• [37:42] “How do you determine if a solution to a problem is more complex than it needs to be? And for inherently complex and interconnected problems, how do you determine if it needs to be subdivided or not? Is there a general approach for working on a complex system with lots of moving parts? (other than me complaining about it :) )”

Read more

This is the eighth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [00:04] “What are your top 5 recommendations for must have projects or applications to build at least once in your life to make you a better programmer ?”

• [14:32] “My question is regarding your previous comment on stack below, where you mentioned that we don't want to do a lot with the stack for performance oriented code. Before this course, I had impression that I should prefer stack allocation to heap allocation (maybe just C++) because I read/watched somewhere saying that 1) stack allocation/dealloc is faster (just move the stack pointer) than heap allocation 2) stack object has more obvious lifetime, 3) stack has no fragmentation concern which improves locality 4) you call on stack object directly while heap object involves indirection.

  I wonder whether my understanding above is correct or not. If not, does that mean I can prefer using heap over stack allocation most of the time? Could you please further comment? Thanks!”

• [32:48] “One idea is to avoid dynamic allocation in the critical path by pushing each measurement into a dedicated logging/profiler thread through a channel. The worker thread would record measurements while the computation threads only perform the minimal ‘send’ operation. But atomic operations, queues, and cross-thread communication can also add overhead, but also distort the original program execution … I’d like advice on how to judge whether this dedicated-thread idea is sound, and in general how to think about designing a low-overhead profiler.”

• [44:20] “What is your opinion on using callbacks for say signal handling? They often seem necessary but I may be overusing them, which leads me to believe my overall architecture could be flawed. But generally speaking, what’s your view on callbacks? Love them or hate them? Do you try to avoid them?”

Read more

This is the seventh video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [00:05] “Question regarding the ‘fat struct’ approach: do you ever find yourself thinking about excess memory consumption caused by some entities having unused fields?”

• [04:29] “About fat structs, you said we would initialize the struct to be able to be A or B depending on the situation, if you imagine that fat struct to be of type Result with cases Success or Error, how would you be able to initialize the Error in the case you have a Success as there was no Error, and vice-versa. This type for example exists in F# and it makes sense to initialize Success only or Error only.”

• [07:40] “Do you have any suggestions for an off the shelf tool to measure bandwidth and flops for a machine your run it on?”

• [13:10] “I sense a recurring theme of reliability and predictability – preferring simple control flow to early returns, preferring simple compilers to strict aliasing, preferring large blocks of memory to pointer festivals, etc. You’ve also spoken about improving _robustness_ by preferring zero/dummy values that just flow through code to null pointers, and preferring handles to pointers (from the iCloud example we just saw).

  What do you mean with “robustness”, and what other techniques can I use to make my code more robust in this way?”

• [18:00] “Do you have any thoughts on Apple’s approach to SIMD?”

• [20:10] “Hi Casey, on the recent podcast with Marco you said that if you could choose a single piece of software to be magically redesigned it would definitely be the browser because the software platform it defines is bad. Could you please elaborate on that? What are the main problems with today’s browsers from your perspective?”

• [22:30] “I’m a bit confused when analyzing the bandwidth I get when reading directly from the volume (using ReadFile with the path ‘\.\C:’ with an offset, for example). As far as I know, that’s the correct way to read system files like the $MFT (which I can currently read properly, by the way).

  When reading 20 GB of contiguous data from the beginning of the volume, I get 2.6 GB/s, and it doesn’t matter whether I use the FILE_FLAG_NO_BUFFERING flag or not—the result is the same. I’d expect something closer to a non-cached read (4.9 GB/s), but I’m getting the same throughput as a cold cached read. I’m not sure where this penalty comes from (assuming the read isn’t triggering extra cache operations since it’s non-buffered).

  Any idea what might be going on here? Do you think these read bandwidths make sense?”

• [25:46] “Will we be able to reuse the coefficients we’re currently using for f64 sine for f32 sine or will we need new ones?”

• [28:24] “I have a question about PC hardware components. I’m not clear on things like the motherboard and chipset. Do they play any role in performance? They vary a lot in price even with similar features, so I assume some aspects must affect overall system performance. Could you give a brief overview of these system parts, if possible?

  Or, to rephrase my question: When you’re building a PC, what do you specifically look at besides just the CPU, RAM, disk and GPU in terms of performance? How do you decide what’s suitable for your specific builds?”

• [36:09] “In my code, I ended up having a single loop over the input that directly produced the haversine sum, rather than splitting parsing and math into two loops. But that means if I want to time parsing vs math, I have to put blocks into the loop, which (seemingly) inevitably introduces a lot of overhead.

  Is there a good way to handle this? The best way I can think of is to instead temporarily comment out parts and just time the rest, though while that’s easy to do with the math part, it seems harder to do for the parsing part, since you still have to somehow produce dummy data for the math while making sure this doesn’t lead to any compiler optimizations you wouldn’t otherwise get.”

• [39:28] “Sorry for repeating the question from the last Q&A, but here is: I have just gotten to it, did it and looked up your solution in QA47 to cross reference. There is one thing that we got differently, and I don’t quite understand your reasoning about it:

  You said that shl rbx, 0 should be recognized by the frontend as a nop and not do anything with flags, but would produce rbx.

  1) If the frontend sees it as a nop, why would it RAW the value of rbx, and not just be a pure nop?

  2) I actually thought that it would not be recognized as a nop (I didn’t find anything about this kind of optimization, i presumed it would be somewhere near zero idiom stuff in the manual), and then it seems like shl will have not only a RAW on rbx, but also on all the flags, as it has be ready that the ALU will say that the shift was 0 and the previous value of flags should be preserved (i. e. RAW)

  So the question is, why is it that rbx is a RAW and flags are skipped, and do you know if there is any place in the docs where such a frontend optimization might be mentioned?”

Read more

This is the sixth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated, and the code repository for downloadable code listings.

Read more

The questions addressed in this video are:

• [00:00:09] “How far can we take SIMD use? What if we put a way bigger onus on eliminating branching and developing parallellizable code? The language K, the simdjson library, the Co-dfns compiler, and the new Box2D engine all blast competitors out of the water by thinking of data parallelism. SIMD use is arduous now - should we start designing our languages and APIs around it? What’s the path to broader cross pollination as an industry? It really seems an untapped potential, right?”

• [00:04:01] “Could you give examples of the kind of substrate work you’re hoping more people take seriously? Are you referring to teams like the WSL or Visual Studio performance teams, or something even deeper (or higher-level)? You’ve cited companies rewriting websites for performance and Microsoft’s console output claims - are these the kinds of things you mean? And if you mean something different, how do you think someone can get involved in that kind of work? I’m trying to understand the bigger vision. Is it that everything on the internet runs as fast as the McMaster-Carr Supply Store website?”

• [00:07:45] “When you critique tools like Git, are you expressing a personal frustration, or do you think there’s an objectively better way these tools could work? It’s hard for me to imagine a world where I don’t need to memorize Git or AWS minutiae - are there products today that represent what you think “just working” should look like?”

• [00:13:00] “What do you think about coroutines?”

• [00:15:50] “Will a VOD of the ‘Research Overview for “The Big OOPs”’ livestream be made available later?”

• [00:16:10] “Do you find that this performance aware development you are talking about also improves the quality of the software in general? Is there any connection? Further, once you decide that something needs to have its own test or tests, what techniques(as in test code organization, handling of test data etc.) and tools you find useful in creating those tests?”

• [00:23:18] “I’m currently onboarding at my first ever job. It’s an giant legacy PHP codebase which primarily uses OOP. I don’t think OOP is great. But if you were forced to use classes for everything, what would be the best way to do it?”

• [00:26:40] “It’s interesting how I get much better performance on Zen3 on Linux, than reference haversine even for the replacement case. ”

• [00:30:47] “I’ve been watching nearly all the BSC talks, and they’ve made me realize I might have a wrong understanding of what a type is. What would be the best definition?”

• [00:35:19] “It seems like there has been a push in recent years towards languages with stronger type systems and static analysis (such as Rust’s borrow checker). Do you think that this trend meaningfully improves software quality, and if so what static analysis tools (both existing and hypothetical) do you think would be the most beneficial for a performance-minded programmer?”

• [00:40:35] “I watched a YouTube video about the Montana mini-computer, and I understood how the concept of a function is implemented at the assembly level. I was wondering: what is a virtual function as defined in a high-level language, and how does that translate to a CPU? Along the same lines, I didn’t fully understand the concept of volatile. It seems to be related to the stack—could you explain how a volatile variable is represented at the assembly/CPU level?”

• [01:04:10] “With Intel AMX becoming more mature or widely-known about, do you think it is or will be possible to start doing things typically done on GPUs (texturing, filtering, convolving, etc) on CPUs in the future? Assuming it will be, do you think GPU vendors will finally start opening up a lá the 30 million line problem in a fight to remain competitive with CPU vendors?”

• [01:13:00] “Unlike games, a lot of the value in the Apple world comes from tight integration with all of their ecosystem and design language (ex: iphone widgets, watch now-playing view, siri searchability, sharing to other apps or airdrop, accessibility integration, etc...) But of course Apple is has a lot of OOP style and “declarative frameworks,” which means that I wouldn’t have control over the code that actually runs these features.

  How can I still use a handmade philosophy of writing my own simpler more focused code rather than depending on lots of slow and volatile libraries?”

• [01:15:36] “do you have any advice for gracefully avoiding or recovering when Apple helpfully deletes your stuff in the background (kills your process when you switch apps, makes you redownload files from icloud)”

• [01:19:21] “In your talk at the Better Software Conference you mention the “fat struct” as a good default option for programming in a systems level language. I think I know the gist of what you mean by this, but I am curious if you have a slightly more formal definition for the term and a general explanation for why it’s a good default approach.”

• [01:27:32] “Most of the course so far talk about programs and assembly for actual chips. How do the aspects of concern under performance aware programming change, if at all, if the target is WASM?”

• [01:30:13] “I perfectly understand why writing to al in a loop is slower than writing to rax, but I get different results for al, ax, eax and rax. The loop writing to al and ax goes at nearly 1/4 of the loop writing to rax, but the eax loop goes at 1/2 that speed. Shouldn’t they run the same since writing to eax does not preserve the upper bits?”

• [01:31:01] “If there’s really no ‘rax’ in a cpu at any given point of time, why (how?) debuggers show a single value what you stop them, or linux will only write a single value in the core dump file? Shouldn’t there be some kind of a tree? I just don’t know if I can’t trust this information for debugging... What if it shows a register value from one branch, but the bug was caused by the value from another, won’t I be misled?”

• [01:40:21] “I was trying to reproduce your results from the RAT and register file lecture. I am running them on Alder Lake chip (i7-12700H). My results were quite the opposite to yours the add only loop was either having similar performance or run much faster compared to mov and add one. I found your article that hints that Alder Lake is able to decouple those chained adds.”

Read more

This is the fifth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated.

Read more

The questions addressed in this video are:

• [00:03] “Hi, I've seen you talk before about the state of gpu apis, and I am aware that you were talking about SoC solution to this and many other problems with current computers. My question is this, how would you design a gpu api, if you had a magic stick that you could shake to make it appear and be common on all computers today? If I understand your position, you'd have the programmer talk directly(or at least through a paper thin layer) to the gpu, but how would that work in practice? If that's not too much to ask, I'd love seeing a short snippet of pseudocode of how it would work on the consumer side, thanks!”

• [04:10] “When dealing with async io, do you think an await style interface like in javascript/go is a good model, or do you think some simple state machine or any other method is better? I find that writing state machines for this kind of stuff gets overly explicit in a lot of cases. Thanks!”

• [07:52] “Given the course, I was trying to apply some techniques to my own toy problem that given a list of words and an NxN grid tries to generate a wordpuzzle where each word is added horizontally, vertically or diagonally.

  I tried to reduce pessimization as much as possible. The program performs a recursive search where each word added successfully to the grid increases the depth by one. Memory is pushed and popped with a single memory arena of max 512KiB. The hot code is the check that given a word and the current board, to see if the word will fit.

  And I am having a hard time vectorizing this loop such that it's actually more performant. The single byte checks seem to outperform vectorization by a factor 10 as the data is too sparse? I also could not find an AVX "scatter" function that does the opposite of a movemask_epi8. Was wondering if you have any thoughts on how one would optimize this further.”

• [13:03] “I was making a homework about asm volatile, and I noticed that all fma instructions were using memory operands and now I'm wondering why there are no fma instructions with immediate operands? Surely it should be beneficial to bake values in some cases, right? Or maybe there are never immediate operands for floating point instructions.”

• [17:54] “Will estimating the cost of more ‘branchy’ workloads like lexing or e.g. json parsing be covered?

  For context I am trying to apply what I've learned from this course to optimize a lexer for a programming language and have gotten from ~0.8 GB/s to ~1.3 GB/s lexing the linux source code. However, it seems impossible for me to get it to run any faster, eventhough 1.3 GB/s is nowhere near memory bandwidth and most of the work is just deciding what kind of token to spit out and how much to advance. It feels to me like ~2GB/s or so could be the limit to how fast you could lex one token at a time, and going above that would require producing more than one token like I believe simdjson does. However, I have no clue if this is remotely correct, since my intuition about the cost of branchy code is provably very bad.”

• [20:12] “Do you have any good resources that outline how to improve a user's experience. The course emphasise performance but I'm curious what other things would you consider to improve a user's experience.”

Read more

This is the fourth video in Part 5 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated. The listing referenced in the video (listing 196) is available on the github.

Read more

The questions addressed in this video are:

• [00:02] “This is more of a learning personal project and I'm looking for something I can do that has a little bigger impact. Do you have some other tool/program/improvement you'd like to see but don't have the bandwidth to do yourself that you can share or just an avenue you'd like to see explored more that I could look into to produce something of value to other fellow programmers?”

• [02:57] “Hey Casey, I posted a question about non-text programming languages and fear I was a little late on the Q&A cycle, Substack didn't send me an email for it :( , for the question to make it into this Q&A's rotation. I am still very curious what your thoughts about a non-text language are, or if you have talked about them before and have a link to those articles/videos!”

• [03:51] “I wanted to know if you have already covered ‘False Sharing’ from a cache point of view and its impact on performance. If not, do you have any plans for it? Do you plan to also cover effects of NUMA on performance?”

• [08:05] “I just rewatched your video on the GJK algorithm from 2006, and when you talk about the triangle case you say that, although there are six ifs and elses written down, you only ever execute three of them at most, so it's at most three tests and jumps. Since the logic only relies on dot products and cross products, I would assume that the multiplies and the adds would have a lesser impact on the performance compared to the penalty of mispredicting multiple branches so close to each other.

  Am I getting something wrong?”

• [11:50] “According to you, what would be the impact of AI on performance aware programming? I mean, is it likely to become more common that organizations / individuals offload this task to sophisticated AI coding agents? I know its a very vague query with possibly no clear answers but just wanted to know what this community thinks about it.”

• [17:34] “Hi Casey, I was working on a simple CSV reader to analyze some logs but I have many huge files and it takes a while. I wanted to try out Intel V Tune and it says that the code is mostly front-end bound and has 100% DSB misses. I was wondering what are those? Are they a problem? How can we solve them?”

• [23:10] “Why is memory management so obscure, and why are people inventing so many languages to ( ostensibly ) fix that, while criticizing C for being unsafe? Why are use-after-free / double-free and so on, issues mentioned when talking about C being unsafe? In other words, why would I need to free memory, when I know exactly how much memory a program will ever need and I can keep on reusing it ( you can't know at a given time how much memory you need, but there is a limit, since you can't allocate infinite memory ).

  In real life, when we discover new concepts ( like in physics ) we don't invent a new language to explains those concepts in., instead, we improve our current language / language practices.
  No worries if you can't include everything from the msg in the Q&A.”

• [29:33] “Hi Casey, before taking the course I would consider using SIMD instead of scalar operations and utilizing memory caches properly to be optimizations. But it seems like you prefer to think about them as a performance aware programming. Then I wonder what would be some concrete examples of optimizations? I just can't imagine something beyond that to get even more performance.”

• [32:59] “Hello, does anybody have any idea why could there be a huge difference between reading a file to mmap'ed memory and malloc'ed memory on Linux, AMD Zen3 chip?”

Read more

This is the fifteenth video in Part 4 of the Performance-Aware Programming series. Please see the Table of Contents to quickly navigate through the rest of the course as it is updated. The listings referenced in the video (listing 194 and 195) are available on the github.

Read more