[Offload] regression on sm_60

[ye-luo]

I saw a regression on sm_60 caused by #122781
after the merge I got

fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "seq_cst".

on the develop, error becomes.

fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "acquire".

reproducer, code
https://github.com/TApplencourt/OvO/blob/master/test_src/cpp/hierarchical_parallelism/memcopy-complex_double/target_teams_distribute_parallel_for.cpp

clang++ -fopenmp --offload-arch=sm_60 target_teams_distribute_parallel_for.cpp

[jhuber6]

Hm, I didn't realize this was a literal limitation of older SMs. I don't think we even use this, but likely with -O0 it's not getting optimized out. @AlexMaclean would it be possible to make all non-monotonic scopes go to seq_cst or is monotonic literally the only one that's supported? Ideally this is solved in the backend.

[llvmbot]

@llvm/issue-subscribers-offload

Author: Ye Luo (ye-luo)

I saw a regression on `sm_60` caused by #122781 after the merge I got ``` fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "seq_cst". ``` on the develop, error becomes. ``` fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "acquire". ```

reproducer, code
https://github.com/TApplencourt/OvO/blob/master/test_src/cpp/hierarchical_parallelism/memcopy-complex_double/target_teams_distribute_parallel_for.cpp

clang++ -fopenmp --offload-arch=sm_60 target_teams_distribute_parallel_for.cpp

[ye-luo]

clang++ -fopenmp --offload-arch=sm_60 target_teams_distribute_parallel_for.cpp
fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "seq_cst".

with -O3

clang++ -fopenmp --offload-arch=sm_60 -O3 target_teams_distribute_parallel_for.cpp
fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "acquire".

with -ffast-math

clang++ -fopenmp --offload-arch=sm_60 -ffast-math target_teams_distribute_parallel_for.cpp # no error.

[jhuber6]

clang++ -fopenmp --offload-arch=sm_60 target_teams_distribute_parallel_for.cpp
fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "seq_cst".

with -O3

clang++ -fopenmp --offload-arch=sm_60 -O3 target_teams_distribute_parallel_for.cpp
fatal error: error in backend: PTX does not support "atomic" for orderings different than"NotAtomic" or "Monotonic" for sm_60 or older, but order is: "acquire".

with -ffast-math

clang++ -fopenmp --offload-arch=sm_60 -ffast-math target_teams_distribute_parallel_for.cpp # no error.

That's interesting. I guess for now we can just revert that patch, since I erroneously thought it wasn't necessary because the backend handled it, but I didn't consider whether or not ptxas handled it.

[ye-luo]

There were codes modified on top that patch. Could you take care of the revert?

[akshayrdeodhar]

Hm, I didn't realize this was a literal limitation of older SMs. I don't think we even use this, but likely with -O0 it's not getting optimized out. @AlexMaclean would it be possible to make all non-monotonic scopes go to seq_cst or is monotonic literally the only one that's supported? Ideally this is solved in the backend.

Looking at the PTX spec for fence- the fence instruction with memory orderings is supported for sm70+. Here's the snippet which inserts a fence to implement a memory ordering for load/store: Perhaps we could provide a brute-force emulation that uses membar.sys (which is supported) instead of a fence with a memory order- but that will likely be very inefficient.

[jhuber6]

Hm, I didn't realize this was a literal limitation of older SMs. I don't think we even use this, but likely with -O0 it's not getting optimized out. @AlexMaclean would it be possible to make all non-monotonic scopes go to seq_cst or is monotonic literally the only one that's supported? Ideally this is solved in the backend.

Looking at the PTX spec for fence- the fence instruction with memory orderings is supported for sm70+. Here's the snippet which inserts a fence to implement a memory ordering for load/store: Perhaps we could provide a brute-force emulation that uses membar.sys (which is supported) instead of a fence with a memory order- but that will likely be very inefficient.

Considering that the alternative is the program not compiling, doesn't seem that bad.