Getting file descriptor count on macOS throws

[Therzok]

Description

A known way to get the file descriptor count on macOS is by using /dev/fd. Here is an example implementation in Rust.

The problem is that some file descriptors listed under /dev/fd are not stat-able, so they will fail when queried. One example is kqueues.

Reproduction Steps

Directory.EnumerateFileSystemEntries("/dev/fd").Count(); on macOS.

Expected behavior

The number of open file descriptors.

Actual behavior

C# > Directory.EnumerateFileSystemEntries("/dev/fd").Count();
╭─❌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ System.UnauthorizedAccessException: Access to the path '/dev/fd/5' is denied.                                                          │
│  ---> System.IO.IOException: Bad file descriptor                                                                                       │
│    --- End of inner exception stack trace ---                                                                                          │
│    at System.IO.FileStatus.ThrowOnCacheInitializationError(ReadOnlySpan`1 path)                                                        │
│    at System.IO.Enumeration.FileSystemEntry.get_IsSymbolicLink()                                                                       │
│    at System.IO.Enumeration.FileSystemEntry.Initialize(FileSystemEntry& entry, DirectoryEntry directoryEntry, ReadOnlySpan`1           │
│ directory, ReadOnlySpan`1 rootDirectory, ReadOnlySpan`1 originalRootDirectory, Span`1 pathBuffer)                                      │
│    at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()                                                                          │
│    at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)                                                                      │
│    at Submission#3.<<Initialize>>d__0.MoveNext()                                                                                       │
│ --- End of stack trace from previous location ---                                                                                      │
│    at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2   │
│ currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Regression?

Not sure if it's a regression.

Known Workarounds

Manually PInvoke opendir and then iterate with readdir.

Configuration

dotnet 6.0.101, macOS 10.14+, x64 and arm64

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[Therzok]

area-System.IO, I guess :D

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

A known way to get the file descriptor count on macOS is by using /dev/fd. Here is an example implementation in Rust.

The problem is that some file descriptors listed under /dev/fd are not stat-able, so they will fail when queried. One example is kqueues.

Reproduction Steps

Directory.EnumerateFileSystemEntries("/dev/fd").Count(); on macOS.

Expected behavior

The number of open file descriptors.

Actual behavior

C# > Directory.EnumerateFileSystemEntries("/dev/fd").Count();
╭─❌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ System.UnauthorizedAccessException: Access to the path '/dev/fd/5' is denied.                                                          │
│  ---> System.IO.IOException: Bad file descriptor                                                                                       │
│    --- End of inner exception stack trace ---                                                                                          │
│    at System.IO.FileStatus.ThrowOnCacheInitializationError(ReadOnlySpan`1 path)                                                        │
│    at System.IO.Enumeration.FileSystemEntry.get_IsSymbolicLink()                                                                       │
│    at System.IO.Enumeration.FileSystemEntry.Initialize(FileSystemEntry& entry, DirectoryEntry directoryEntry, ReadOnlySpan`1           │
│ directory, ReadOnlySpan`1 rootDirectory, ReadOnlySpan`1 originalRootDirectory, Span`1 pathBuffer)                                      │
│    at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()                                                                          │
│    at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)                                                                      │
│    at Submission#3.<<Initialize>>d__0.MoveNext()                                                                                       │
│ --- End of stack trace from previous location ---                                                                                      │
│    at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2   │
│ currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Regression?

Not sure if it's a regression.

Known Workarounds

Manually PInvoke opendir and then iterate with readdir.

Configuration

dotnet 6.0.101, macOS 10.14+, x64 and arm64

Other information

No response

Author:	Therzok
Assignees:	-
Labels:	area-System.IO, untriaged
Milestone:	-

[danmoseley]

@Therzok what do you propose here? Presumably in the general case, we do want to throw this exception.

[Therzok]

The API implicitly does the evaluation of dir entries and their metadata. Something like directory entry count shouldn't populate the stat information of files, unless it's explicitly requested.

The equivalent C code is something like:

DIR *dir = opendir("/dev/fd");
struct dirent *entries;

int count = 0;
while ((entries = readdir(dir)) {
    count++;
}

I'd understand if something like GetFileSystemEntries would do implicit evaluation here to have a snapshot of the filesystem data, but EnumerateFileSystemEntries is advertised as lazy. Shouldn't the entry properties be populated lazily?

[adamsitnik]

This code has been recently refactored by @tmds:

What we have in 6.0:

runtime/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEntry.Unix.cs

Lines 45 to 61 in 3d345e0

	if (isUnknown)
	{
	isSymlink = entry.IsSymbolicLink;
	// Need to fail silently in case we are enumerating
	isDirectory = entry._status.IsDirectory(entry.FullPath, continueOnError: true);
	}
	// Same idea as the directory check, just repeated for (and tweaked due to the
	// nature of) symlinks.
	// Whether we had the dirent structure or not, we treat a symlink to a directory as a directory,
	// so we need to reflect that in our isDirectory variable.
	else if (isSymlink)
	{
	// Need to fail silently in case we are enumerating
	isDirectory = entry._status.IsDirectory(entry.FullPath, continueOnError: true);
	}
	entry._status.InitiallyDirectory = isDirectory;

What we have in 7.0:

runtime/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEntry.Unix.cs

Lines 43 to 58 in e23382c

	if (isDirectory)
	{
	entry._isDirectory = true;
	}
	else if (isSymlink)
	{
	entry._isDirectory = entry._status.IsDirectory(entry.FullPath, continueOnError: true);
	}
	else if (isUnknown)
	{
	entry._isDirectory = entry._status.IsDirectory(entry.FullPath, continueOnError: true);
	if (entry._status.IsSymbolicLink(entry.FullPath, continueOnError: true))
	{
	entry._directoryEntry.InodeType = Interop.Sys.NodeType.DT_LNK;
	}
	}

in theory it should not throw, let me start my Linux machine and check it

[tmds]

This code has been recently refactored by @tmds:

Yes, this was refactored (and fixed) in #60214.
One thing that was missing is continueOnError: true for IsSymbolicLink which is the cause for this exception.

[adamsitnik]

I've added a test that ensures that the bug is not coming back: #65301

[kasperk81]

One thing that was missing is continueOnError: true for IsSymbolicLink which is the cause for this exception.

is this fix planned for backporting to net6.0? if not, any workaround for users?

[tmds]

if not, any workaround for users?

Not using .NET APIs.

Maybe you can try PInvoking the proc_pidinfo API mentioned in this article: https://zameermanji.com/blog/2021/8/1/counting-open-file-descriptors-on-macos/.