Use a "borrowing" trick in _fe_normalize to simplify the handling of values in [P, 2^256). This is significantly faster too (according to bench_internal), though performance of _fe_normalize isn't that important.
Applying a similar idea to _fe_normalizes_to_zero also results in a performance boost, and in this case it appears to boost signing speed by 2% or more (also ECDH), due to three calls in _gej_add_ge.
61 | + * x is incremented before the first pass and then decremented before the second pass 62 | + * to ensure that the result doesn't fall into the range [P, 2^256). */ 63 | + uint64_t x = (t4 >> 48) + 1; t4 &= 0x0FFFFFFFFFFFFULL; 64 | 65 | /* The first pass ensures the magnitude is 1, ... */ 66 | t0 += x * 0x1000003D1ULL;
Given that x is bigger than before, how we know that t0 += x * 0x1000003D1ULL won't overflow the uint64_t?
Most, if not all, callers use field elements with a maximum magnitude of 8, so there's plenty of room. The practical limit is actually around 31 (because field_10x26 can only represent magnitude 32 and several methods assume there's a free bit available for carries on entry). I think it would be a good idea to actually enforce some exact magnitude limit on input to field methods though (probably 16).
This is a really nice idea! I just had one inline question.
The updated versions of fe_normalize and fe_normalizes_to_zero indeed look faster, and a local benchmark run seems to agree:
Benchmark , Min(us) , Avg(us) , Max(us)
field_normalize , 0.00968 , 0.00973 , 0.00983
group_add_affine , 0.189 , 0.196 , 0.205
ecdsa_sign , 22.3 , 22.7 , 23.9
Benchmark , Min(us) , Avg(us) , Max(us)
field_normalize , 0.00752 , 0.00756 , 0.00763
group_add_affine , 0.189 , 0.189 , 0.191
ecdsa_sign , 22.3 , 22.4 , 22.5
I also wrote down some notes as I was trying to understand the change, and I'll post them here in case they're useful for future reference.
If there's no final carry bit, then the second pass will subtract 2^256 - P from a value that's at most 2^256 - 1, resulting in a value that's at most P - 1.
If there was a carry, the largest possible value (not including the carry bit) is 65536 * 0x1000003D1 - 1 for 5x52 and 1024 * 0x1000003D1 - 1 for 10x26, both of which are well below P.
(If there was a carry bit, clearing it is equivalent (mod P) to decrementing by 2^256 - P, which cancels out the increment before the first pass.)
The 64th bit is nonzero if and only if there was underflow after subtraction (2^256 - P is only a 33-bit value, so after subtracting it it's not possible to underflow so much that the 64th bit is still cleared afterwards).
Thus subtracting t >> 63 is the same logic as "if subtraction on the previous limb caused an underflow then subtract 1, otherwise subtract 0".
If we store a 52-bit value in a 64-bit word, underflowing it and then keeping the bottom 52 bits is equivalent to underflowing the same value stored in a 52-bit word.
(An analogous explanation works for where the 31-bit shift came from in the 10x26 case.)
ACK 4bbb6612dc0581ded87d911039729c4ec1a143c8
My only concern from before was magnitude / overflow. I agree it'd be a good idea to enforce some exact magnitude limit and/or more carefully audit maximum possible magnitudes. But I don't think it should block this PR—especially since I also tried running tests with VERIFY_CHECK(r->magnitude <= 16) at the top of fe_normalize and it passed.
Tested ACK 8932956fe553235d4b0789e7bd82ff0bee59335d (rebased on master 22245ae)
The algebraic mechanics here are remarkably neat. Pre-adding 2^256 - P via x = (t4 >> 48) + 1 shifts any candidate in [P, 2^256) into [2^256, 2^256 + 2^256 - P), guaranteeing a carry into bit 48 of t4. When that carry occurs, masking off bit 48 drops 2^256 == 2^256 - P (mod P), exactly canceling the initial pre-addition. When no carry occurs, the original value was strictly < P, and the second pass simply subtracts the pre-added 2^256 - P back out via (t4 >> 48) - 1. This completely eliminates the wide multi-limb comparator (t4 == 0x0FFFFFFFFFFFFULL && m == 0xFFFFFFFFFFFFFULL && t0 >= 0xFFFFEFFFFFC2FULL) from master.
Regarding the overflow question on t0 += x * 0x1000003D1ULL: for an input with magnitude m, t4 < m * 2^52 gives x = (t4 >> 48) + 1 <= 16m + 1. The accumulation t0 + x * 0x1000003D1ULL is bounded by m * 2^52 + (16m + 1) * (2^32 + 977). Because m * 2^52 dominates by a factor of 65536 over 16m * 2^32, exceeding 2^64 - 1 strictly requires m >= 4096. At the codebase maximum magnitude of 32, t0 fits comfortably inside 57 bits. On field_10x26, the equivalent bound requires m >= 64, whereas the 26-bit representation itself caps at 32.
Rebased cleanly on master 22245ae. All 227 ctests, exhaustive tests, and property checks pass with VERIFY_CHECK active. On Zen 2 (Clang 22.1.8), field_normalize latency drops from 16.8 ns down to 12.6 ns (~25% speedup).
Rebased the PR itself onto 22245ae, with some comments updates. Also added a pre-commit with new benchmarks for _normalizes_to_zero(_var).
While _normalize is clearly benefiting (~20% faster for 5x52, ~10% for 10x26), the case for _normalizes_to_zero is quite sketchy and I would like to see others' bench_internal results. I had particular problems with gcc, which seems to change the inlining strategy in the benchmark code itself between old/new versions. Either way, there is little effect on high-level bench results.
Re-tested on 542a88fa (Windows x86_64, Clang 22.1.8 Release -O2, Zen 2).
Comparing against baseline 6d1ada12:
Benchmark Baseline (6d1ada1) PR (542a88f) Change
field_normalize 0.0183 us 0.0137 us -25.1%
field_normalizes_to_zero 0.0161 us 0.0155 us -3.7%
field_normalizes_to_zero_var 0.0021 us 0.0022 us ~0.0%
The contrast between _fe_normalize and _fe_normalizes_to_zero is clear in the emitted assembly. In _fe_normalize, PR 1028 eliminates a full 64-bit multiplication (imulq) on the second pass along with stack frame register spilling. In _fe_normalizes_to_zero, both branches only ever executed a single multiplication; delaying limb matching into z1 and z2 serializes six logical operations at the tail of the carry chain (34 loop instructions vs 33 in baseline), keeping throughput bounded by the carry dependency chain and leaving gains modest.