[KeyInstr] Add Atom Group waterline to LLVMContext

[OCHyams]

The waterline is managed through DILocation creation,
LLVMContext::incNextAtomGroup, and LLVMContext::updateAtomGroupWaterline.

Add unittest.

[OCHyams]

• [NFC][KeyInstr][Clang] Add some additional tests #134654
• [KeyInstr][Clang] Coerced store atoms #134653
• [KeyInstr][Clang] Ret atom #134652
• [KeyIntsr][Clang] Builtins alloca auto-init atom #134651
• [KeyInstr][Clang] Assign matrix element atom #134650
• [KeyInstr][Clang] Assign vector element atom #134649
• [KeyInstr][Clang] Bitfield atom #134648
• [KeyInstr][Clang] For range stmt atoms #134647
• [KeyInstr][Clang] For stmt atom #134646
• [KeyInstr][Clang] While stmt atom #134645
• [KeyInstr][Clang] Do stmt atom #134644
• [KeyInstr][Clang] Switch stmt atom #134643
• [KeyInstr][Clang] If stmt atom #134642
• [KeyInstr][Clang] Catch variable init atom #134641
• [KeyInstr][Clang] Member initalization atom #134640
• [KeyInstr][Clang] Aggregate init + copy #134639
• [KeyInstr] Complex assignment atoms #134638
• [KeyInstr][Clang] Assignment atom group #134637
• [KeyInstr][Clang] Static variable init atom #134636
• [KeyInstr][Clang] Agg init atom #134635
• [Clang][NFC] Move some static functions into CodeGenFunction #134634
• [KeyInstr][Clang] Scalar init atom #134633
• [KeyInstr][Clang] Add ApplyAtomGroup #134632
• [KeyInstr][Clang] Add Clang option -g[no-]key-instructions #134627
• [KeyInstr][DwarfDebug] Add is_stmt emission support #133495
• [KeyInstr] Add MIR parser support #133494
• [KeyInstr] Remap cloned PHIs in BreakCriticalEdges #133493
• Propagate DebugLocs on phis in BreakCriticalEdges #133492
• [KeyInstr][LoopUnswitch] Remap cloned instructions' atoms #133491
• [KeyInstr][LoopRotate] Remap atoms of duplicated instructions #133490
• [KeyInstr][LoopUnroll] Remap atoms while unrolling #133489
• [KeyInstr][JumpThreading] Remap atoms duping bb with cond br on phi into pred #133488
• [KeyInstr][JumpThreading] Remap atoms after threading #133487
• [KeyInstr][JumpThreading] Remap atoms in blocks duplicated for threading #133486
• [KeyInstr][Inline] Don't propagate atoms to inlined nodebug instructions #133485
• [KeyInstr][SimplifyCFG] Remap atoms after duplication for threading #133484
• [KeyInstr][debugify] Add --debugify-atoms to add key instructions metadata #133483
• [KeyInstr][SimplifyCFG] Remap atoms when folding br to common succ into pred #133482
• [KeyInstr] Inline atom info #133481
• [KeyInstr] Merge atoms in DILocation::getMergedLocation #133480
• [NFC][KeyInstr] Add Atom Group (re)mapping #133479
• [KeyInstr] Add Atom Group waterline to LLVMContext #133478 👈 (View in Graphite)
• [KeyInstr] Add fields to DILocation behind compile time flag #133477
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[llvmbot]

@llvm/pr-subscribers-llvm-ir

@llvm/pr-subscribers-debuginfo

Author: Orlando Cazalet-Hyams (OCHyams)

Changes

The waterline is managed through DILocation creation,
LLVMContext::incNextAtomGroup, and LLVMContext::updateAtomGroupWaterline.

Add unittest.

---

Full diff: https://github.com/llvm/llvm-project/pull/133478.diff

5 Files Affected:

• (modified) llvm/include/llvm/IR/LLVMContext.h (+8)
• (modified) llvm/lib/IR/DebugInfoMetadata.cpp (+3)
• (modified) llvm/lib/IR/LLVMContext.cpp (+8)
• (modified) llvm/lib/IR/LLVMContextImpl.h (+10)
• (modified) llvm/unittests/IR/MetadataTest.cpp (+38)

diff --git a/llvm/include/llvm/IR/LLVMContext.h b/llvm/include/llvm/IR/LLVMContext.h
index bbd125fd38cf1..d3cdd31e0b12f 100644
--- a/llvm/include/llvm/IR/LLVMContext.h
+++ b/llvm/include/llvm/IR/LLVMContext.h
@@ -335,6 +335,14 @@ class LLVMContext {
   StringRef getDefaultTargetFeatures();
   void setDefaultTargetFeatures(StringRef Features);
 
+  /// Key Instructions: update the highest number atom group emitted for any
+  /// function.
+  void updateAtomGroupWaterline(uint64_t G);
+
+  /// Key Instructions: get the next free atom group number and increment
+  /// the global tracker.
+  uint64_t incNextAtomGroup();
+
 private:
   // Module needs access to the add/removeModule methods.
   friend class Module;
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index f4f9fca38945c..aefda2f7be0b0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -67,6 +67,9 @@ DILocation::DILocation(LLVMContext &C, StorageType Storage, unsigned Line,
 #ifdef EXPERIMENTAL_KEY_INSTRUCTIONS
   assert(AtomRank <= 7 && "AtomRank number should fit in 3 bits");
 #endif
+  if (AtomRank)
+    C.updateAtomGroupWaterline(AtomGroup + 1);
+
   assert((MDs.size() == 1 || MDs.size() == 2) &&
          "Expected a scope and optional inlined-at");
   // Set line and column.
diff --git a/llvm/lib/IR/LLVMContext.cpp b/llvm/lib/IR/LLVMContext.cpp
index 447e5d92e0b99..4781085b30431 100644
--- a/llvm/lib/IR/LLVMContext.cpp
+++ b/llvm/lib/IR/LLVMContext.cpp
@@ -377,3 +377,11 @@ StringRef LLVMContext::getDefaultTargetFeatures() {
 void LLVMContext::setDefaultTargetFeatures(StringRef Features) {
   pImpl->DefaultTargetFeatures = Features;
 }
+
+void LLVMContext::updateAtomGroupWaterline(uint64_t V) {
+  pImpl->NextAtomGroup = std::max(pImpl->NextAtomGroup, V);
+}
+
+uint64_t LLVMContext::incNextAtomGroup() {
+  return pImpl->NextAtomGroup++;
+}
diff --git a/llvm/lib/IR/LLVMContextImpl.h b/llvm/lib/IR/LLVMContextImpl.h
index 99f0d8837de52..5a52d41834095 100644
--- a/llvm/lib/IR/LLVMContextImpl.h
+++ b/llvm/lib/IR/LLVMContextImpl.h
@@ -1832,6 +1832,16 @@ class LLVMContextImpl {
 
   std::string DefaultTargetCPU;
   std::string DefaultTargetFeatures;
+
+  /// The next available source atom group number. The front end is responsible
+  /// for assigning source atom numbers, but certain optimisations need to
+  /// assign new group numbers to a set of instructions. Most often code
+  /// duplication optimisations like loop unroll. Tracking a global maximum
+  /// value means we can know (cheaply) we're never using a group number that's
+  /// already used within this function.
+  ///
+  /// Start a 1 because 0 means the source location isn't part of an atom group.
+  uint64_t NextAtomGroup = 1;
 };
 
 } // end namespace llvm
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 94cebb0406598..8a1c1f9d306c6 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "../lib/IR/LLVMContextImpl.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
@@ -1366,6 +1367,43 @@ TEST_F(DILocationTest, discriminatorSpecialCases) {
   EXPECT_EQ(std::nullopt, L4->cloneByMultiplyingDuplicationFactor(0x1000));
 }
 
+TEST_F(DILocationTest, KeyInstructions) {
+  Context.pImpl->NextAtomGroup = 1;
+
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 1u);
+  DILocation *A1 = DILocation::get(Context, 1, 0, getSubprogram(), nullptr, false, 1, 2);
+  // The group is only applied to the DILocation if the build has opted into
+  // the additional DILocation fields needed for the feature.
+#ifdef EXPERIMENTAL_KEY_INSTRUCTIONS
+  EXPECT_EQ(A1->getAtomGroup(), 1u);
+  EXPECT_EQ(A1->getAtomRank(), 2u);
+#else
+  EXPECT_EQ(A1->getAtomGroup(), 0u);
+  EXPECT_EQ(A1->getAtomRank(), 0u);
+#endif
+
+  // Group number 1 has been "used" so next available is 2.
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 2u);
+
+  // Set a group number higher than current + 1, then check the waterline.
+  DILocation::get(Context, 2, 0, getSubprogram(), nullptr, false, 5, 1);
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 6u);
+
+  // The waterline should be unchanged (group <= next).
+  DILocation::get(Context, 3, 0, getSubprogram(), nullptr, false, 4, 1);
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 6u);
+  DILocation::get(Context, 3, 0, getSubprogram(), nullptr, false, 5, 1);
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 6u);
+
+  // Check the waterline gets incremented by 1.
+  EXPECT_EQ(Context.incNextAtomGroup(), 6u);
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 7u);
+
+  Context.updateAtomGroupWaterline(8);
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 8u);
+  Context.updateAtomGroupWaterline(7);
+  EXPECT_EQ(Context.pImpl->NextAtomGroup, 8u);
+}
 
 typedef MetadataTest GenericDINodeTest;