[lldb-dap] Re-enable the lldb-dap tests

[JDevlieghere]

Re-enable the lldb-dap tests. We've spent the last week improving the reliability of the test suite and the tests now pass reliably on macOS and Linux at desk. Let's see how things fare on the bots.

[llvmbot]

@llvm/pr-subscribers-lldb

Author: Jonas Devlieghere (JDevlieghere)

Changes

Re-enable the lldb-dap tests. We've spent the last week improving the reliability of the test suite and the tests now pass reliably on macOS and Linux at desk. Let's see how things fare on the bots.

---

Full diff: https://github.com/llvm/llvm-project/pull/138791.diff

9 Files Affected:

• (modified) lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py (-1)
• (modified) lldb/test/API/tools/lldb-dap/attach/TestDAP_attachByPortNum.py (+1)
• (modified) lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_breakpointLocations.py (+1-2)
• (modified) lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_setBreakpoints.py (+1-2)
• (modified) lldb/test/API/tools/lldb-dap/commands/TestDAP_commands.py (-2)
• (modified) lldb/test/API/tools/lldb-dap/disassemble/TestDAP_disassemble.py (+1-2)
• (modified) lldb/test/API/tools/lldb-dap/evaluate/TestDAP_evaluate.py (-2)
• (modified) lldb/test/API/tools/lldb-dap/memory/TestDAP_memory.py (+1-2)
• (modified) lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py (+1-2)

diff --git a/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py b/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py
index e0ec87bd9a8d8..c8ab5e9bedc88 100644
--- a/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py
+++ b/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py
@@ -24,7 +24,6 @@ def spawn_and_wait(program, delay):
     process.wait()
 
 
-@skip
 class TestDAP_attach(lldbdap_testcase.DAPTestCaseBase):
     def set_and_hit_breakpoint(self, continueToExit=True):
         self.dap_server.wait_for_stopped()
diff --git a/lldb/test/API/tools/lldb-dap/attach/TestDAP_attachByPortNum.py b/lldb/test/API/tools/lldb-dap/attach/TestDAP_attachByPortNum.py
index 12fba1133b95b..489beb5a55ad8 100644
--- a/lldb/test/API/tools/lldb-dap/attach/TestDAP_attachByPortNum.py
+++ b/lldb/test/API/tools/lldb-dap/attach/TestDAP_attachByPortNum.py
@@ -18,6 +18,7 @@
 import socket
 
 
+@skip("SBTarget::ConnectRemote is not synchronous")
 class TestDAP_attachByPortNum(lldbdap_testcase.DAPTestCaseBase):
     default_timeout = 20
 
diff --git a/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_breakpointLocations.py b/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_breakpointLocations.py
index 4a99cacc761a3..1058157e2c668 100644
--- a/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_breakpointLocations.py
+++ b/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_breakpointLocations.py
@@ -11,8 +11,7 @@
 import lldbdap_testcase
 import os
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
+
 class TestDAP_breakpointLocations(lldbdap_testcase.DAPTestCaseBase):
     def setUp(self):
         lldbdap_testcase.DAPTestCaseBase.setUp(self)
diff --git a/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_setBreakpoints.py b/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_setBreakpoints.py
index 6c6681804f250..26df2573555df 100644
--- a/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_setBreakpoints.py
+++ b/lldb/test/API/tools/lldb-dap/breakpoint/TestDAP_setBreakpoints.py
@@ -11,8 +11,7 @@
 import lldbdap_testcase
 import os
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
+
 class TestDAP_setBreakpoints(lldbdap_testcase.DAPTestCaseBase):
     def setUp(self):
         lldbdap_testcase.DAPTestCaseBase.setUp(self)
diff --git a/lldb/test/API/tools/lldb-dap/commands/TestDAP_commands.py b/lldb/test/API/tools/lldb-dap/commands/TestDAP_commands.py
index 4aecf9a665c06..223258fbdd3dc 100644
--- a/lldb/test/API/tools/lldb-dap/commands/TestDAP_commands.py
+++ b/lldb/test/API/tools/lldb-dap/commands/TestDAP_commands.py
@@ -6,8 +6,6 @@
 from lldbsuite.test.decorators import *
 
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
 class TestDAP_commands(lldbdap_testcase.DAPTestCaseBase):
     def test_command_directive_quiet_on_success(self):
         program = self.getBuildArtifact("a.out")
diff --git a/lldb/test/API/tools/lldb-dap/disassemble/TestDAP_disassemble.py b/lldb/test/API/tools/lldb-dap/disassemble/TestDAP_disassemble.py
index ebecb349ac177..9e8ef5b289f2e 100644
--- a/lldb/test/API/tools/lldb-dap/disassemble/TestDAP_disassemble.py
+++ b/lldb/test/API/tools/lldb-dap/disassemble/TestDAP_disassemble.py
@@ -10,8 +10,7 @@
 import lldbdap_testcase
 import os
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
+
 class TestDAP_disassemble(lldbdap_testcase.DAPTestCaseBase):
     @skipIfWindows
     def test_disassemble(self):
diff --git a/lldb/test/API/tools/lldb-dap/evaluate/TestDAP_evaluate.py b/lldb/test/API/tools/lldb-dap/evaluate/TestDAP_evaluate.py
index 19b682dfcd22d..372a9bb75e007 100644
--- a/lldb/test/API/tools/lldb-dap/evaluate/TestDAP_evaluate.py
+++ b/lldb/test/API/tools/lldb-dap/evaluate/TestDAP_evaluate.py
@@ -11,8 +11,6 @@
 from lldbsuite.test.lldbtest import *
 
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
 class TestDAP_evaluate(lldbdap_testcase.DAPTestCaseBase):
     def assertEvaluate(self, expression, regex):
         self.assertRegex(
diff --git a/lldb/test/API/tools/lldb-dap/memory/TestDAP_memory.py b/lldb/test/API/tools/lldb-dap/memory/TestDAP_memory.py
index c71ba871b8a22..ea43fccf016a7 100644
--- a/lldb/test/API/tools/lldb-dap/memory/TestDAP_memory.py
+++ b/lldb/test/API/tools/lldb-dap/memory/TestDAP_memory.py
@@ -10,8 +10,7 @@
 import lldbdap_testcase
 import os
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
+
 class TestDAP_memory(lldbdap_testcase.DAPTestCaseBase):
     def test_memory_refs_variables(self):
         """
diff --git a/lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py b/lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py
index 560207bfbb66c..3b45cdc245838 100644
--- a/lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py
+++ b/lldb/test/API/tools/lldb-dap/variables/TestDAP_variables.py
@@ -17,8 +17,7 @@ def make_buffer_verify_dict(start_idx, count, offset=0):
         verify_dict["[%i]" % (i)] = {"type": "int", "value": str(i + offset)}
     return verify_dict
 
-# DAP tests are flakey, see https://github.com/llvm/llvm-project/issues/137660.
-@skip
+
 class TestDAP_variables(lldbdap_testcase.DAPTestCaseBase):
     def verify_values(self, verify_dict, actual, varref_dict=None, expression=None):
         if "equals" in verify_dict:

[kusmour]

Let's go (fingercrossed

[DavidSpickett]

Sure, let's give this a go now.

(rather than end of week for a potential weekend of red)

[DavidSpickett]

Actually the launch sequence changes are breaking Windows tests. I need to see that all green first.

[da-viper]

is there a branch tracking the reverted changes #138219, I have an old windows 10 pc i could use to reproduce it.

[DavidSpickett]

As far as I can know, #138219 went onto main, so you can revert my revert to get that back.

Then there was 69a0af3, which was later reverted in 5e70460. I saw that cause timeouts on Linux but it didn't change the Windows results.

So it should reproduce with just that first change reapplied to main.

[JDevlieghere]

I re-landed the launch sequence patch (aeeb9a3) and the Windows bots are happy again. Given that was the only objections, I'm going to merge this so we can get some insights overnight.

As I said in the description, I expect the tests to be reliable, but if they're not, feel free to revert this again but please include a link to the failures or list the affected tests so we can continue investigating.

[JDevlieghere]

All the Linux and Windows buildbots are green, but GreenDragon is consistently failing with one of the breakpoint tests and it's not immediately obvious what the problem is. If I rerun the tests enough times, I can reproduce a failure locally. I've disabled those tests in df4eac2.

[DavidSpickett]

We have a few failures on Windows on Arm too: 2668167

I've skipped those tests for now.

[JDevlieghere]

Attaching the bot output before it gets recycled:
stdio-3.txt