Resolves #505 - Fix handling for Windows long paths #506

kcieplak · 2025-05-09T14:22:57Z

PR#369 caused the majority of tests on Windows to fail, as normalization of RelativePath was removed. This change also removed the call to 'PathAllocCanonicalize' for AbsolutePath which had the long file flag 'PATHCCH_ALLOW_LONG_PATHS'.

Reintroducing canonicalization of AbsolutePath path representation to handle long paths.

Update tests dealing with RelativePath to match implementation.
Canonicalize the path representation for AbsolutePath which also allows for long path '\?' prefix addition when path > 260 in length.
Strip trailing backslash on string representation of AbsolutePath to match definition. Only strips for non root paths.
Add helper functions:
- removeTrailingBackslash
- stripPrefix
- canonicalPathRepresentation
Add Windows API Error helpers
Add long path tests into each test case
Ran swift format on code
Update copyright dates

kcieplak · 2025-05-09T14:42:02Z

@swift-ci test

compnerd

Most of this is style or white space changes which really makes this harder to review. Can you split those out into separate PRs that can be merged independently?

compnerd · 2025-05-09T15:02:35Z

Sources/TSCBasic/Path.swift

+    static func stripPrefix(_ path: String) -> String {
+        return path.withCString(encodedAs: UTF16.self) { cStringPtr in
+            let mutableCStringPtr = UnsafeMutablePointer(mutating: cStringPtr)
+            let result = PathCchStripPrefix(mutableCStringPtr, path.count + 1)


Should this not be path.utf16.count?

compnerd · 2025-05-09T15:02:58Z

Sources/TSCBasic/Path.swift

+    static func removeTrailingBackslash(_ path: String) -> String {
+        return path.withCString(encodedAs: UTF16.self) { cStringPtr in
+            let mutableCStringPtr = UnsafeMutablePointer(mutating: cStringPtr)
+            let result = PathCchRemoveBackslash(mutableCStringPtr, path.count + 1)


kcieplak · 2025-05-09T16:22:00Z

Most of this is style or white space changes which really makes this harder to review. Can you split those out into separate PRs that can be merged independently?

Yes for sure

PR#369 - swiftlang#369 Caused the majority of tests on Windows to fail, as normalization of RelativePath was removed. This change also removed the call to 'PathAllocCanonicalize' for AbsolutePath which had the long file flag 'PATHCCH_ALLOW_LONG_PATHS'. Reintroducing canonicalization of AbsolutePath path representation to handle long paths. * Update tests dealing with RelativePath to match implementation. * Canonicalize the path representation for AbsolutePath which also allows for long path '\\?\' prefix addition when path > 260 in length. * Strip trailing backslash on string representation of AbsolutePath to match definition. Only strips for non root paths. * Add helper functions: - removeTrailingBackslash - stripPrefix - canonicalPathRepresentation * Add Windows API Error helpers * Add long path tests into each test case * Fix up .suffix. '.' has no suffix * Update copyright dates

bkhouri · 2025-05-09T17:51:08Z

Sources/TSCBasic/Path.swift

@@ -457,6 +458,79 @@ private struct WindowsPath: Path, Sendable {
        return !path.withCString(encodedAs: UTF16.self, PathIsRelativeW)
    }

+    /// When this function returns successfully, the same path string will have had the prefix removed,
+    /// if the prefix was present. If no prefix was present, the string will be unchanged.
+    static func stripPrefix(_ path: String) -> String {


suggestion: can we add automated tests that verify this function in isolation?

bkhouri · 2025-05-09T17:51:15Z

Sources/TSCBasic/Path.swift

+    ///   * Path is not a root path
+    ///   * Pash has a trailing backslash
+    /// If conditions are not met then the string is returned unchanged.
+    static func removeTrailingBackslash(_ path: String) -> String {


suggestion: can we add automated tests that verify this function in isolation?

bkhouri · 2025-05-09T17:51:34Z

Sources/TSCBasic/Path.swift

+    /// to ensure long paths greater than MAX_PATH (260) characters are handled correctly.
+    ///
+    /// - seealso: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
+    static func canonicalPathRepresentation(_ path: String) throws -> String {


suggestion: can we add automated tests that verify this function in isolation?

note: this is copied mostly verbatim from swift-foundation; I think a comment that that's the case and it generally shouldn't be changed independently, would be better than tests. Really wish we had a package that could share this implementation rather than having to duplicate it everywhere.

kcieplak · 2025-05-09T17:55:45Z

@swift-ci test

jakepetroules · 2025-05-09T20:00:43Z

Sources/TSCBasic/Path.swift

+    /// if the prefix was present. If no prefix was present, the string will be unchanged.
+    static func stripPrefix(_ path: String) -> String {
+        return path.withCString(encodedAs: UTF16.self) { cStringPtr in
+            let mutableCStringPtr = UnsafeMutablePointer(mutating: cStringPtr)


This looks unsafe, you're taking a mutable view onto an immutable pointer; I think you need to use withUnsafeTemporaryAllocation and copy to a new buffer first.

jakepetroules · 2025-05-09T20:04:58Z

Sources/TSCBasic/Path.swift

@@ -457,6 +458,79 @@ private struct WindowsPath: Path, Sendable {
        return !path.withCString(encodedAs: UTF16.self, PathIsRelativeW)
    }

+    /// When this function returns successfully, the same path string will have had the prefix removed,
+    /// if the prefix was present. If no prefix was present, the string will be unchanged.
+    static func stripPrefix(_ path: String) -> String {


I'd name this Win32PathCchStripPrefix to make it clear that it provides the semantics of that Win32 function; "stripPrefix" is quite general otherwise -- strip what prefix?

I don't think that Win32 is the right prefix, this is not Win32 related. We could use Zw as the prefix as per the NT internals.

jakepetroules · 2025-05-09T20:05:19Z

Sources/TSCBasic/Path.swift

+    /// If conditions are not met then the string is returned unchanged.
+    static func removeTrailingBackslash(_ path: String) -> String {
+        return path.withCString(encodedAs: UTF16.self) { cStringPtr in
+            let mutableCStringPtr = UnsafeMutablePointer(mutating: cStringPtr)


Same comment about memory safety.

jakepetroules · 2025-05-09T20:17:05Z

Sources/TSCBasic/Path.swift

+                // 2. Canonicalize the path.
+                // This will add the \\?\ prefix if needed based on the path's length.
+                var pwszCanonicalPath: LPWSTR?
+                let flags: ULONG = numericCast(PATHCCH_ALLOW_LONG_PATHS.rawValue) | numericCast(PATHCCH_CANONICALIZE_SLASHES.rawValue)


Why did you add PATHCCH_CANONICALIZE_SLASHES? I hadn't found it to be necessary, because GetFullPathNameW will have already done that. Our sole reason for calling PathAllocCanonicalize here is to prepend the \?\ prefix for long paths, which are otherwise effectively already normalized by the aforementioned call.

Note also that this value is only available in Windows 11 version 10.0.22000.194 and might cause problems on older OS versions, or at least will be ignored. CI is definitely on Windows 10 so it won't reflect any behaviors resulting from this.

You are correct, it is not required, and I have removed that additional flag.

jakepetroules · 2025-05-09T20:24:35Z

Sources/TSCBasic/Path.swift

+        let noPrefixPath = Self.stripPrefix(string)
+        let prefix = string.replacingOccurrences(of: noPrefixPath, with: "") // Just the prefix or empty
+
+        // Perform drive designator normalization i.e. 'c:\' to 'C:\' on string.


It's concerning that the Win32 APIs don't do this. I wonder if we want to move this part into canonicalPathRepresentation.

@compnerd Any thoughts on drive letter casing?

If we're canonicalising here so that plain string comparison works, we should use uppercase for device names/drive letters, IMO.

Paths are case insensitive, and I don't know of any win32 API that would change the drive letter. I don't know if there is a canonical spelling for the drive letter.

jakepetroules · 2025-05-09T20:25:39Z

Sources/TSCBasic/Path.swift

@@ -536,7 +614,13 @@ private struct WindowsPath: Path, Sendable {
        if !Self.isAbsolutePath(realpath) {
            throw PathValidationError.invalidAbsolutePath(path)
        }
-        self.init(string: realpath)
+        do {
+            let canonicalizedPath = try Self.canonicalPathRepresentation(realpath)


Under what circumstances are you seeing Self.canonicalPathRepresentation return a path with a trailing slash? Because it goes through GetFullPathW, I don't see how this could be the case.

If I remove the call to remove the trailing slashes

let canonicalizedPath = try Self.canonicalPathRepresentation(realpath) let normalizedPath = canonicalizedPath //let normalizedPath = Self.removeTrailingBackslash(canonicalizedPath) // AbsolutePath states paths have no trailing separator.

Then I get some failures on even simple paths.

XCTAssertEqual(AbsolutePath(#"C:\ab\cd\ef\"#).pathString, #"C:\ab\cd\ef"#) C:\Users\kcieplak\swift-tools-support-core\Tests\TSCBasicTests\PathTests.swift:143: error: PathTests.testTrailingPathSeparators : XCTAssertEqual failed: ("C:\ab\cd\ef\") is not equal to ("C:\ab\cd\ef") -

XCTAssertEqual(AbsolutePath(#"C:\ab\cd\ef\\"#).pathString, #"C:\ab\cd\ef"#) C:\Users\kcieplak\swift-tools-support-core\Tests\TSCBasicTests\PathTests.swift:144: error: PathTests.testTrailingPathSeparators : XCTAssertEqual failed: ("C:\ab\cd\ef\") is not equal to ("C:\ab\cd\ef") -

let longAbsolutePathOverPathMax = generatePath(280) XCTAssertEqual(AbsolutePath(longAbsolutePathOverPathMax + #"\"#).pathString, #"\\?\"# + longAbsolutePathOverPathMax) C:\Users\kcieplak\swift-tools-support-core\Tests\TSCBasicTests\PathTests.swift:148: error: PathTests.testTrailingPathSeparators : XCTAssertEqual failed: ("\\?\C:\0\1\2\3\4\5\6\7\8\9\10\11\12\13\14\15\16\17\18\19\20\21\22\23\24\25\26\27\28\29\30\31\32\33\34\35\36\37\38\39\40\41\42\43\44\45\46\47\48\49\50\51\52\53\54\55\56\57\58\59\60\61\62\63\64\65\66\67\68\69\70\71\72\73\74\75\76\77\78\79\80\81\82\83\84\85\86\87\88\89\90\91\92\93\94\95\") is not equal to ("\\?\C:\0\1\2\3\4\5\6\7\8\9\10\11\12\13\14\15\16\17\18\19\20\21\22\23\24\25\26\27\28\29\30\31\32\33\34\35\36\37\38\39\40\41\42\43\44\45\46\47\48\49\50\51\52\53\54\55\56\57\58\59\60\61\62\63\64\65\66\67\68\69\70\71\72\73\74\75\76\77\78\79\80\81\82\83\84\85\86\87\88\89\90\91\92\93\94\95") -

Similar question as previously, does Self. add anything? Are we expecting subclasses to replace the implementation?

jakepetroules · 2025-05-09T20:28:47Z

Tests/TSCBasicTests/PathTests.swift

+    var dirNameCount = 0
+    while currentPathLength < length {
+        let dirName = String(dirNameCount)
+        path.append("\(path.count != 0 ? separator : "")\(dirName)")


Note that path components on Windows generally have a maximum of 255 characters; you might want to account for that here.

jakepetroules · 2025-05-09T20:31:48Z

Tests/TSCBasicTests/PathTests.swift

@@ -1,31 +1,75 @@
 /*


Would be good to add some tests where the input has the \\?\ prefix already, as well as some which have the \\.\ prefix, especially with different forms of backslashes following that, e.g.:

\\?\C:\test \\?\C:/test \\.\C:\test \\.\C:/test

In my latest commit I have added a bunch of tests using the unParsed '\\?\' and device '\\.\' prefixed paths.
There are likely more edge case tests that should be added for them, but it's a start.

The tests right now are very declarative, nice to read but lots of repetition, it might be work trying to make them more programatic.

kcieplak · 2025-05-13T19:44:24Z

@swift-ci test

compnerd · 2025-05-13T20:53:40Z

Sources/TSCBasic/Path.swift

@@ -457,6 +458,83 @@ private struct WindowsPath: Path, Sendable {
        return !path.withCString(encodedAs: UTF16.self, PathIsRelativeW)
    }

+    /// When this function returns successfully, the same path string will have had the prefix removed,
+    /// if the prefix was present. If no prefix was present, the string will be unchanged.
+    static func Win32PathCchStripPrefix(_ path: String) -> String {


I don't think that this should be Win32PathCchStripPrefix. This is not a Win32 prefix but rather a kernel prefix.

Changed to 'stripRawPathPrefix'

Sorry, I suggested that name :) Should we just call it PathCchStripPrefix, same as the underlying Win32 function, and let overload resolution handle picking the right one?

Yeah, that sounds good to me. I think that I've done that in the past as well, just overloading the Win32 APIs with a more Swift friendly interface.

kcieplak · 2025-05-14T08:32:20Z

@swift-ci test

kcieplak · 2025-05-14T13:07:31Z

@swift-ci test

jakepetroules · 2025-05-14T17:14:16Z

Sources/TSCBasic/Path.swift

+        // Remove prefix from the components, allowing for comparison across normalized paths.
+        var prefixStrippedPath = Self.stripRawPathPrefix(String(cString: normalized))
+        // The '\\.\'' prefix is not removed by stripRawPathPrefix do this manually.
+        if prefixStrippedPath.starts(with: #"\\.\"#) {


What about the \\?\ prefixed? That's the more commonly used form.

var prefixStrippedPath = Self.stripRawPathPrefix(String(cString: normalized))

That will remove the \?\ prefix

There are a variety of prefixes:

\\?\ - root local path

\\.\ - device local path

\??\ - NT Object Path

\\Device\ - NT Device Path

The interesting thing is that for a number of these, it is not possible to construct a win32 representation of the path, and the prefix should not be stripped.

Consider something like: \\?\Volume{2b10d654-f5dd-4597-bd8b-480ddd40bcbc}\Users\compnerd\SourceCache\swift-project\swift-tools-support-core.

How would we handle this path? It actually may not be a mapped volume, so there is no associated drive letter to reference it. However, by using the absolute root local path, we can reference the file content routing through the MUP driver.

You are correct that there is so many more situations that the current implementation of Path, that are not covered. I believe this change based on the test cases in place gets us a bit closer, so at least we can handle longer paths i..e. \?\C:<long-path>, but lots more work is required.

There are many other path nuances to deal with as you have mentioned.

compnerd · 2025-05-14T18:03:22Z

Sources/TSCBasic/Path.swift

+    /// Removes the "\\?\" prefix, if present, from a file path. When this function returns successfully,
+    /// the same path string will have the prefix removed,if the prefix was present.
+    /// If no prefix was present,the string will be unchanged.
+    static func stripRawPathPrefix(_ path: String) -> String {


I think that PathCchStripPrefix is the agreed upon name (overloading the WinSDK interface). Although, might be nice to just make this a freestanding function. I'm not sure that there is any value in embedding this into the type.

Will do, moving back to match the Windows naming.

compnerd · 2025-05-14T18:03:45Z

Sources/TSCBasic/Path.swift

+    ///   * Path is not a root path
+    ///   * Pash has a trailing backslash
+    /// If conditions are not met then the string is returned unchanged.
+    static func removeTrailingBackslash(_ path: String) -> String {


Thoughts on just naming this PathCchRemoveBackslash? Similarly, I think that this can be a free function.

Renamed to PatchCchRemoveBackslash, and standalone.

compnerd · 2025-05-14T18:04:12Z

Sources/TSCBasic/Path.swift

-
-        return String(cString: normalized).components(separatedBy: "\\").filter { !$0.isEmpty }
+        // Remove prefix from the components, allowing for comparison across normalized paths.
+        var prefixStrippedPath = Self.stripRawPathPrefix(String(cString: normalized))


Why the Self.? Do you expect subtypes which override the behaviour?

No just an omission.

Removed in favour of a basic standalone function.

compnerd · 2025-05-14T18:13:07Z

Sources/TSCBasic/Path.swift

+        // Remove prefix from the components, allowing for comparison across normalized paths.
+        var prefixStrippedPath = Self.stripRawPathPrefix(String(cString: normalized))
+        // The '\\.\'' prefix is not removed by stripRawPathPrefix do this manually.
+        if prefixStrippedPath.starts(with: #"\\.\"#) {


There are a variety of prefixes:

\\?\ - root local path

\\.\ - device local path

\??\ - NT Object Path

\\Device\ - NT Device Path

The interesting thing is that for a number of these, it is not possible to construct a win32 representation of the path, and the prefix should not be stripped.

Consider something like: \\?\Volume{2b10d654-f5dd-4597-bd8b-480ddd40bcbc}\Users\compnerd\SourceCache\swift-project\swift-tools-support-core.

How would we handle this path? It actually may not be a mapped volume, so there is no associated drive letter to reference it. However, by using the absolute root local path, we can reference the file content routing through the MUP driver.

kcieplak · 2025-05-15T00:59:38Z

@swift-ci test

kcieplak · 2025-05-15T10:22:01Z

@swift-ci test windows

kcieplak · 2025-05-15T14:59:25Z

@swift-ci test

kcieplak · 2025-05-15T14:59:38Z

@swift-ci test windows

compnerd · 2025-05-15T15:53:42Z

Sources/TSCBasic/Path.swift

+                    throw Win32Error(GetLastError())
+                }
+                // 1.5 Leave \\.\ prefixed paths alone since device paths are already an exact representation and PathCchCanonicalizeEx will mangle these.
+                if let base = pwszFullPath.baseAddress,


Hmm, I think that we missed this on the changes in Foundation as well, but should we not ensure that pwszFullPath.count is >= 4?

Yes, good catch. I will add a conditional around that.

compnerd · 2025-05-15T15:55:20Z

Sources/TSCBasic/Path.swift

        {
-            self.string = "\(string.first!.uppercased())\(string.dropFirst(1))"
+            self.string = prefix + "\(noPrefixPath.first!.uppercased())\(noPrefixPath.dropFirst(1))"


Why not just merge this into the string interpolation?

Yup might as well, already paying the interpolation cost.

compnerd · 2025-05-15T15:56:03Z

Sources/TSCBasic/Path.swift

@@ -536,7 +614,13 @@ private struct WindowsPath: Path, Sendable {
        if !Self.isAbsolutePath(realpath) {
            throw PathValidationError.invalidAbsolutePath(path)
        }
-        self.init(string: realpath)
+        do {
+            let canonicalizedPath = try Self.canonicalPathRepresentation(realpath)


Similar question as previously, does Self. add anything? Are we expecting subclasses to replace the implementation?

- Move PathCchStripPrefix and PathCchRemoveBackslash to use a mutable temporary buffer. - Could not use buffer.withMemoryRebound and getCstring() as on Windows this seem to produce corrupt data. - Add more tests for unParsed '\\?\' and device '\\.\' paths - Remove the PATHCCH_CANONICALIZE_SLASHES flag as it is not needed. - Add Win32Error.swift to CMakeLists

kcieplak · 2025-05-15T16:54:31Z

@swift-ci test

kcieplak · 2025-05-15T16:54:47Z

@swift-ci test windows

compnerd

Thanks for sticking with this, this looks good to me (the remaining comments are minor/trivial).

compnerd · 2025-05-15T23:37:50Z

Sources/TSCBasic/Path.swift

        } else {
-            self.string = string
+            self.string = prefix + noPrefixPath
        }


I wonder if this complexity is worth it - we could just pay the small penalty and always follow the canonicalization path irrespective of if path.first?.isLowerccase (i.e. just always do the upper case conversion).

compnerd · 2025-05-15T23:39:06Z

Sources/TSCBasic/Path.swift

+            if pwszFullPath.count >= 4 {
+                if let base = pwszFullPath.baseAddress,


Style nit: I would've compressed this to:

if pwszFullPath.count >= 4, let base = pwszFullPath.baseAddress, ...

kcieplak requested review from bnbarham and MaxDesiatov as code owners May 9, 2025 14:22

MaxDesiatov requested review from compnerd and al45tair May 9, 2025 14:23

kcieplak mentioned this pull request May 9, 2025

Windows: Test failing due to assert in computing relative path. swiftlang/swift-package-manager#8602

Open

1 task

compnerd reviewed May 9, 2025

View reviewed changes

kcieplak force-pushed the topics/fix-windows-long-file-handling branch 3 times, most recently from 0fef8bc to 5f5b9e5 Compare May 9, 2025 17:38

kcieplak force-pushed the topics/fix-windows-long-file-handling branch from 5f5b9e5 to e267bc3 Compare May 9, 2025 17:55

bkhouri reviewed May 9, 2025

View reviewed changes

jakepetroules reviewed May 9, 2025

View reviewed changes

compnerd requested changes May 13, 2025

View reviewed changes

kcieplak force-pushed the topics/fix-windows-long-file-handling branch from 1e582d4 to b25da2d Compare May 14, 2025 13:06

kcieplak requested review from compnerd and jakepetroules May 14, 2025 13:43

jakepetroules reviewed May 14, 2025

View reviewed changes

jakepetroules approved these changes May 14, 2025

View reviewed changes

compnerd reviewed May 14, 2025

View reviewed changes

kcieplak force-pushed the topics/fix-windows-long-file-handling branch from b25da2d to 83d7041 Compare May 15, 2025 00:59

kcieplak force-pushed the topics/fix-windows-long-file-handling branch from 83d7041 to 8ea5813 Compare May 15, 2025 14:58

compnerd reviewed May 15, 2025

View reviewed changes

kcieplak force-pushed the topics/fix-windows-long-file-handling branch from 8ea5813 to 7de5288 Compare May 15, 2025 16:54

kcieplak requested a review from compnerd May 15, 2025 23:18

compnerd approved these changes May 15, 2025

View reviewed changes

bkhouri merged commit eb4c83d into swiftlang:main May 16, 2025
3 checks passed

		if pwszFullPath.count >= 4 {
		if let base = pwszFullPath.baseAddress,

Resolves #505 - Fix handling for Windows long paths #506

Resolves #505 - Fix handling for Windows long paths #506

Conversation

kcieplak commented May 9, 2025

kcieplak commented May 9, 2025

compnerd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcieplak commented May 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakepetroules May 9, 2025 • edited Loading

Choose a reason for hiding this comment

kcieplak commented May 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcieplak May 13, 2025 • edited Loading

Choose a reason for hiding this comment

kcieplak commented May 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcieplak commented May 14, 2025

kcieplak commented May 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcieplak commented May 15, 2025

kcieplak commented May 15, 2025

kcieplak commented May 15, 2025

kcieplak commented May 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcieplak commented May 15, 2025

kcieplak commented May 15, 2025

compnerd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakepetroules May 9, 2025 •

edited

Loading

kcieplak May 13, 2025 •

edited

Loading