Use correct types for syscalls

[kleisauke]

Split from: #13007.

[sbc100]

If the tests pass then LGTM.

IIRC there was a reason we did it this way when porting the musl syscall layer, but I can't remember exactly what that was.. so lets just go with this if the tests pass.

[sbc100]

Should we also use pointer types here for oldpath/newpath? and elsewhere?

[sbc100]

Should list be gid_t* (avoiding the case below?)

[sbc100]

Why are these new functions needed?

[sbc100]

Can we avoid the all the changes/additions to the sanitizers? This change seems a lot more risky once you add those in?

Also changing the socket syscalls to be varargs seems a little out of scope and perhaps separable.

[sbc100]

Duplicating these definitions here seems a little sad.

[kleisauke]

Let me mark this as a draft, there are many more changes needed than expected. For example, doing this:

-int __syscall_openat(int dirfd, intptr_t path, int flags, ...); // mode is optional
+int __syscall_openat(int dirfd, const char *path, int flags, ...); // mode is optional

Will fail within the sanitizers (which uses the internal syscall function of musl):

emscripten/system/lib/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

Line 262 in b182182

return internal_syscall(SYSCALL(openat), AT_FDCWD, (uptr)filename, flags);

emscripten/tools/system_libs.py

Lines 1925 to 1928 in b182182

	class libsanitizer_common_rt(CompilerRTLibrary, MTLibrary):
	name = 'libsanitizer_common_rt'
	# TODO(sbc): We should not need musl-internal headers here.
	includes = ['system/lib/libc/musl/src/internal',

Details

/home/kleisauke/emscripten/system/lib/compiler-rt/lib/sanitizer_common/sanitizer_syscall_generic.inc:17:24: note: expanded from macro 'SYSCALL'
   17 | # define SYSCALL(name) SYS_ ## name
      |                        ^~~~~~~~~~~~
<scratch space>:132:1: note: expanded from here
  132 | SYS_openat
      | ^~~~~~~~~~
/home/kleisauke/emscripten/cache/sysroot/include/bits/syscall.h:61:21: note: expanded from macro 'SYS_openat'
   61 | #define SYS_openat              __syscall_openat
      |                                 ^~~~~~~~~~~~~~~~
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/home/kleisauke/emscripten/system/lib/libc/musl/src/internal/syscall.h:65:80: note: expanded from macro '__SYSCALL_DISP'
   65 | #define __SYSCALL_DISP(b,...) __SYSCALL_CONCAT(b,__SYSCALL_NARGS(__VA_ARGS__))(__VA_ARGS__)
      |                                                                                ^~~~~~~~~~~
/home/kleisauke/emscripten/system/lib/libc/musl/src/internal/syscall.h:55:61: note: expanded from macro '__syscall_emscripten3'
   55 | #define __syscall_emscripten3(n,a,b,c) __syscall_emscripten(n,__scc(a),__scc(b),__scc(c))
      |                                                             ^
/home/kleisauke/emscripten/system/lib/libc/musl/src/internal/syscall.h:51:38: note: expanded from macro '__syscall_emscripten'
   51 | #define __syscall_emscripten(n, ...) n(__VA_ARGS__)
      |                                      ^
/home/kleisauke/emscripten/cache/sysroot/include/syscall_arch.h:93:5: note: candidate function not viable: no known conversion from 'uptr' (aka 'unsigned long') to 'const char *' for 2nd argument
   93 | int __syscall_openat(int dirfd, const char *path, int flags, ...); // mode is optional
      |     ^
/home/kleisauke/emscripten/system/lib/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:270:27: error: no matching function for call to '__syscall_openat'
  270 |   return internal_syscall(SYSCALL(openat), AT_FDCWD, (uptr)filename, flags,

The solution is to rework the sanitizers to avoid musl's syscall layer, similar to the NetBSD and macOS implementations.

emscripten/system/lib/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp

Lines 165 to 167 in b182182

	uptr internal_open(const char *filename, int flags) {
	return open(filename, flags);
	}

[sbc100]

Yeah I think this is one reason things are the way they today.. to allow the SYSCALL stuff to work.

Perhaps we could land the initial/smaller version of this change that doesn't cause some much churn?

[kleisauke]

I've just split the compiler-rt changes to PR #24655.

[kleisauke]

This should be in good shape now, but it's still very difficult to review. I'll probably split it into 3 PRs instead (i.e. one for commit 088e789, 362e6be and efd41d6).