Validate the mode argument in access, eaccess, and faccessat for optional

POSIX compliance and to improve compatibility with Linux and NetBSD

The issue was identified with lib/libc/sys/t_access:access_inval from
NetBSD

Update the manpage accordingly

PR: 181155
Reviewed by: jilles (code), jmmv (code), wblock (manpage), wollman (code)
MFC after: 4 weeks
Phabric: D678 (code), D786 (manpage)
Sponsored by: EMC / Isilon Storage Division

lib/libc/sys/access.2

@@ -133,8 +133,16 @@ and
 .Sh RETURN VALUES
 .Rv -std
 .Sh ERRORS
-Access to the file is denied if:
+.Fn access ,
+.Fn eaccess ,
+or
+.Fn faccessat 
+will fail if:
 .Bl -tag -width Er
+.It Bq Er EINVAL
+The value of the
+.Fa mode
+argument is invalid.
 .It Bq Er ENOTDIR
 A component of the path prefix is not a directory.
 .It Bq Er ENAMETOOLONG

sys/kern/vfs_syscalls.c

@@ -2055,6 +2055,9 @@ kern_accessat(struct thread *td, int fd, char *path, enum uio_seg pathseg,
 	cap_rights_t rights;
 	int error;
 
+	if (amode != F_OK && (amode & ~(R_OK | W_OK | X_OK)) != 0)
+		return (EINVAL);
+
 	/*
 	 * Create and modify a temporary credential instead of one that
 	 * is potentially shared.