The Pattern Behind x64 CMPPS/PD Immediates
If you're wondering how the 5 bits used in the CMPPD/PS immediate got assigned the way they did, Pete Cawley's got your back.
While I can’t say it matters for any practical purpose, I have more-than-once wondered how Intel came up with this table:
The meaning of these immediate values is straightforward enough if you understand the cmpps and cmppd instructions. First, you pick one of eight different comparisons: equal, less-than, less-than-or-equal, always false, not equal, greater-than-or-equal, greater-than, and always true. Then, you pick whether or not you want the comparison to be signaling1. Finally, you pick whether you want NaN’s to pass through as true values or false values (called “unordered” vs. “ordered” comparisons in this context).
Once you’ve made those three decisions, you combine them together and you pick the immediate value that corresponds to your choice. Eight possible comparisons, times two for signaling-or-not, times another two for ordered-or-not, equals thirty-two different values. Great! That’s exactly how many are in the table. Everything checks out.
But how did Intel choose which combination of immediate bits mapped to which combination of choices? Naively, one might think that a bit would be set whenever the comparison is signaling. Another might be set if the comparison is unordered. And so on.
To some extent, this does seem to be true. 0H, 8H, 10H, and 18H code for the four different possible equals comparisons. This would imply that the 4th and 5th bits must code for ordering and signaling, respectively, right?
Well, 0H, where the 4th bit is unset, is an ordered equals. 8H, where the 4th bit is set, is an unordered equals. So the 4th bit must code for an unordered comparison.
Except wait - 4H, which does not have the 4th bit set, is an unordered not-equals. CH, which does have the 4th bit set, is the ordered version! So the bit seems to code for the opposite in this case.
This kind of pattern continues whenever you look at the values. You can find something like a relationship between bits, but never quite what you would expect.
Recently, I asked The Internet if anyone knew what was actually going on here. While it unfortunately rewards my lazy behavior, I am delighted to report that Pete Cawley worked out what was likely going on, and wrote it up:
It’s a quick read, and covers the entire immediate encoding. So, if you’re like me and are curious as to how these sorts of things end up the way they do, I highly recommend you head over to Pete’s writeup and check it out!
Which, unless you’ve unmasked them yourself, usually won’t signal either way, since NaN exceptions are masked out by default on the most x64-based OSes.