cmd/compile: arm64 multiplication with constant optimization

[egonelbre]

While looking into optimizing ed25119 verification on ARM64, I noticed that x * 19 is not optimized into shifted adds.

func multiply19(x uint64) uint64 {
    return x * 19
}
// which compiles to:
MOVD    $19, R1
MUL     R1, R0, R0

This can be reduced to:

ADD     R0<<3, R0, R1
ADD     R1<<1, R0, R0

The general form seems to be:

x * c, where
    c = 1 + 2^N + 2^(N+M), N != M;
    N > 1, M > 1 // not sure whether this restriction is necessary

Then the multiplication can be rewritten as:
    x + (x + x << M) << N

This can be checked with:
    (c-1)&1 == 0 && bits.OnesCount(c - 1) == 2

Which holds for numbers like:
    7, 11, 13, 19, 21, 25, 35, 37, 41, 49, 67, 69, 73, 81, 97, 131, 133, 137, 145, 161, 193...

There is also a similar reduction, that can be done:

x * c, where
    c = 1 + 2^N + 2^M + 2^(N+M), N < M
    N > 1  // not sure whether this restriction is necessary

Then the multiplication can be rewritten as:
    x = x + x<<N; x = x + x<<M

This can be checked with:
    (c-1)&1 == 0 && bits.OnesCount(c - 1) == 3 && highbit - lowbit = midbit

Which holds for numbers like:
    15, 23, 27, 29, 39, 43, 51, 53, 57, 71, 75, 83, 89, 99, 101, 135, 139, 147, 163, 169, 177, 195...

I didn't verify, but this might be useful on amd64 as well.

---

I can send a CL about this, but I'm not sure whether there are some corner cases with high c values that I'm not thinking of. Similarly, I wasn't able to figure out how to write the second reduction in SSA rules.

[randall77]

amd64 already reduces *19 like you suggest.

This all sounds reasonable to me.
The second reduction can be done using rules by just repeating the inner operation - the repeated part will get CSEd subsequently. (Some of the amd64 rules do exactly that.)

[egonelbre]

Some additional notes. It seems there are few additional variants that follow similar patterns:

c = 1 + 2^H - 2^L, where L < H
  => (x + x<<L) - x<<H

c = 2^H - 2^L, where L < H
  => (x<<L) - x<<H

c = 2^L + 2^H, where L < H
  => (x<<L) + x<<H

PS: does anyone know whether these rules have some standard naming?

[cooler-SAI]

Some additional notes. It seems there are few additional variants that follow similar patterns:

c = 1 + 2^H - 2^L, where L < H
  => (x + x<<L) - x<<H

c = 2^H - 2^L, where L < H
  => (x<<L) - x<<H

c = 2^L + 2^H, where L < H
  => (x<<L) + x<<H

PS: does anyone know whether these rules have some standard naming?

They don't have specific standard names but are widely recognized techniques in the context of bit manipulation and arithmetic operations.

[gopherbot]

Change https://go.dev/cl/626998 mentions this issue: cmd/compile: improve multiplication strength reduction

[erifan]

Hi @randall77 , I can't log in to my google-source account registered with the Arm email. Regarding this multiplication optimization, one thing to note on arm64 is that the latency and throughput of arm64 shift addition are related to the shift amount. For example on V2, it is as follows:

instruction                            latency          throughput           pipeline
add shift, shift <= 4                       1                6                       I
add shift, shift > 4                        2                2                       M
MUL                                         2                 2                     M

So in theory, when the shift amount is greater than 4, the cost of MUL and ADD is the same. So when converting a constant MUL into two ADDs, the shift amount of ADD needs to be considered. However, this seems to be handled differently in different compilers. The handling of gcc and clang seems to be different. Also you can refer to this link: dotnet/runtime#75119

[randall77]

on arm64 is that the latency and throughput of arm64 shift addition are related to the shift amount.

I'm not seeing that on an M2 ultra.

bench.s:

TEXT ·ShiftAdd3(SB),0,$0-0
	MOVD	$65536, R1
	MOVD	$0, R0
loop:
	ADD	R0<<3, R0, R0
	SUB	$1, R1, R1
	CBNZ	R1, loop
	RET

TEXT ·ShiftAdd6(SB),0,$0-0
	MOVD	$65536, R1
	MOVD	$0, R0
loop:
	ADD	R0<<6, R0, R0
	SUB	$1, R1, R1
	CBNZ	R1, loop
	RET

bench.go:

package p

func ShiftAdd3()
func ShiftAdd6()

bench_test.go:

package p

import "testing"

func BenchmarkShiftAdd3(b *testing.B) {
	for range b.N {
		ShiftAdd3()
	}
}

func BenchmarkShiftAdd6(b *testing.B) {
	for range b.N {
		ShiftAdd6()
	}
}

This gives me the same time for both benchmarks, at around 38000ns.
That's 0.58ns/iter, which at 3.5GHz is is almost exactly 2 clocks.

If I change the add instruction to remove the shift, then I get a 2x speedup, which means add is 1 clock. Shifts are also 1 clock.
Multiplies look to be 3 clocks.

So maybe this would not be worth it if we end up using 2 shift-containing instructions. And maybe a wash with 1 add+shift instruction and 1 plain (add or shift but not both) instruction. (Although not having to load the constant into a register probably helps if that load can't be lifted out of a loop.)

Of course, that's all M2 ultra specific. Different arm chips I'm sure are different. There's no real easy way to deal with that fact, other than to pick something reasonable and live with it.

[erifan]

This is the test results on an V2 machine:

$ ./bench.test -test.bench BenchmarkShiftAdd
goos: linux
goarch: arm64
pkg: bench
BenchmarkShiftAdd3-72    	   54376	     20533 ns/op
BenchmarkShiftAdd6-72    	   29246	     41028 ns/op
PASS

The latency and throughput of shift-add are related to the shift amount, which is the same on almost all ARM v8/v9 chips.
We can also reproduce this result using C code. I think gcc may not handle this for other reasons, such as the interaction with other optimizations (I don't know how gcc implements this optimization). Or it may just not be handled properly.

[erifan]

But this also depends on the execution environment. If there is no loop, we should actually compare a MOV + a MUL instruction and an ADD-SHIFT instruction, because MUL does not support immediate values, and we need to move the constant to a register with MOV first. But if there is a loop, and the MOV instruction is moved outside the loop, then the performance comparison is between one MUL and one Add-SHIFT instruction.