When a container using an NFS-backed volume (e.g. AWS EFS via the EFS
CSI driver) with subPathExpr is killed, kubelet may fail to restart it
with CreateContainerConfigError and "stale NFS file handle" because the
subpath bind mount target holds a cached NFS4 file handle that has been
invalidated server-side. IsMountPoint() calls lstat() on the target,
gets ESTALE, and prepareSubpathTarget() treats it as a hard failure. The
pod becomes permanently stuck and requires manual intervention.
The subpath teardown path was already hardened against stale NFS handles
in kubernetes/kubernetes#71804 (doCleanSubPaths), but the setup path in
prepareSubpathTarget() was never updated.
Detect corrupted mount points using mount.IsCorruptedMnt() and unmount
the stale bind mount before proceeding to re-create it.
Ref: https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/614
Ref: https://redhat.atlassian.net/browse/OCPBUGS-84229
mount.IsNotMount point has been deprecated and mounter.IsMountPoint
is now preffered.
This small refactor if prepareSubpathTarget() should not pose any
risk because IsNotMountPoint directly calls IsMountPoint and just
returns its negated value.
Fix several incorrect error and log messages in volume plugins that
produce confusing or incomplete output during troubleshooting:
- Fix "MountMount.NodeExpandVolume" to "MountVolume.NodeExpandVolume"
in node_expander.go (2 occurrences)
- Fix "MountVolume.NodeExapndVolume" to "MountVolume.NodeExpandVolume"
in node_expander.go and operation_generator.go (2 occurrences)
- Fix iscsi mkdir error log that prints literal "error" instead of the
actual error value
- Fix "error ummounting" to "error unmounting" in subpath handling
- Fix malformed "with :" in teardown error messages in configmap,
secret, projected, and downwardapi volume plugins
- Fix duplicated "is is" in operation_generator.go comment
Signed-off-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
This has been replaced by `//build:...` for a long time now.
Removal of the old build tag was automated with:
for i in $(git grep -l '^// +build' | grep -v -e '^vendor/'); do if ! grep -q '^// Code generated' "$i"; then sed -i -e '/^\/\/ +build/d' "$i"; fi; done
For example, we have two filesystems, one is embedded into another:
/a/test # first filesystem with a directory "/a/test/b2"
/a/test/b2 # not auto mounted yet second filesystem, notice "/a/test/b2" is
# a new directory on this filesystem after this filesystem is mounted
For subpath mount "/a/test/b2", `openat("/a/test", "b2")` gets directory "b2" on the first
filesystem, then "mount -c" will use this wrong directory as source directory.
`fstatat("/a/test", "b2/")` forces triggering auto mount of second filesystem, so
`openat("/a/test", "b2")` gets correct source directory for "mount -c".
This fixes issue https://github.com/kubernetes/kubernetes/issues/110818#issuecomment-1175736550
References:
1. https://man7.org/linux/man-pages/man2/openat.2.html
If pathname refers to an automount point that has not yet
been triggered, so no other filesystem is mounted on it,
then the call returns a file descriptor referring to the
automount directory without triggering a mount.
2. https://man7.org/linux/man-pages/man2/open_by_handle_at.2.html
name_to_handle_at() does not trigger a mount when the final
component of the pathname is an automount point. When a
filesystem supports both file handles and automount points, a
name_to_handle_at() call on an automount point will return with
error EOVERFLOW without having increased handle_bytes. This can
happen since Linux 4.13 with NFS when accessing a directory which
is on a separate filesystem on the server. In this case, the
automount can be triggered by adding a "/" to the end of the
pathname.
Errors from staticcheck:
pkg/volume/azure_dd/azure_common.go:68:2: var winDiskNumFormat is unused (U1000)
pkg/volume/csi/csi_block.go:97:2: field volumeInfo is unused (U1000)
pkg/volume/csi/csi_block_test.go:56:6: func prepareBlockUnmapperTest is unused (U1000)
pkg/volume/csi/csi_client.go:108:2: const initialDuration is unused (U1000)
pkg/volume/csi/csi_client.go:109:2: const factor is unused (U1000)
pkg/volume/csi/csi_client.go:110:2: const steps is unused (U1000)
pkg/volume/csi/csi_client_test.go:83:8: this value of err is never used (SA4006)
pkg/volume/csi/csi_mounter.go:76:2: field options is unused (U1000)
pkg/volume/csi/csi_mounter_test.go:454:13: this value of err is never used (SA4006)
pkg/volume/csi/csi_plugin_test.go:766:16: this value of err is never used (SA4006)
pkg/volume/csi/csi_plugin_test.go:861:13: this value of err is never used (SA4006)
pkg/volume/csi/csi_plugin_test.go:1186:13: this value of err is never used (SA4006)
pkg/volume/csi/csi_plugin_test.go:1249:13: this value of err is never used (SA4006)
pkg/volume/csi/csi_test.go:305:5: the goroutine calls T.Fatalf, which must be called in the same goroutine as the test (SA2002)
pkg/volume/flexvolume/probe_test.go:67:10: this value of err is never used (SA4006)
pkg/volume/iscsi/iscsi_test.go:95:2: field attachCalled is unused (U1000)
pkg/volume/iscsi/iscsi_test.go:96:2: field detachCalled is unused (U1000)
pkg/volume/iscsi/iscsi_test.go:501:24: this value of err is never used (SA4006)
pkg/volume/iscsi/iscsi_util_test.go:159:2: this value of exist is never used (SA4006)
pkg/volume/local/local.go:351:57: argument devicePath is overwritten before first use (SA4009)
pkg/volume/plugins_test.go:119:2: this value of plug is never used (SA4006)
pkg/volume/plugins_test.go:125:2: this value of plug is never used (SA4006)
pkg/volume/quobyte/quobyte.go:474:23: this result of append is never used, except maybe in other appends (SA4010)
pkg/volume/quobyte/quobyte.go:477:23: this result of append is never used, except maybe in other appends (SA4010)
pkg/volume/quobyte/quobyte.go:480:23: this result of append is never used, except maybe in other appends (SA4010)
pkg/volume/rbd/rbd.go:886:2: field adminSecret is unused (U1000)
pkg/volume/rbd/rbd.go:887:2: field adminID is unused (U1000)
pkg/volume/rbd/rbd.go:888:2: field imageFormat is unused (U1000)
pkg/volume/rbd/rbd.go:889:2: field imageFeatures is unused (U1000)
pkg/volume/storageos/storageos.go:302:2: field secretName is unused (U1000)
pkg/volume/storageos/storageos_util_test.go:43:2: field apiAddr is unused (U1000)
pkg/volume/storageos/storageos_util_test.go:44:2: field apiUser is unused (U1000)
pkg/volume/storageos/storageos_util_test.go:45:2: field apiPass is unused (U1000)
pkg/volume/storageos/storageos_util_test.go:46:2: field apiVersion is unused (U1000)
pkg/volume/util/atomic_writer_test.go:756:49: argument err is overwritten before first use (SA4009)
pkg/volume/util/fsquota/common/quota_linux_common.go:37:2: const acct is unused (U1000)
pkg/volume/util/fsquota/common/quota_linux_common.go:38:2: const enforcing is unused (U1000)
pkg/volume/util/fsquota/project.go:168:31: identical expressions on the left and right side of the '==' operator (SA4000)
pkg/volume/util/fsquota/quota_linux.go:306:50: argument poduid is overwritten before first use (SA4009)
pkg/volume/util/fsquota/quota_linux_test.go:558:16: this value of err is never used (SA4006)
pkg/volume/util/subpath/subpath_linux.go:232:81: argument err is overwritten before first use (SA4009)
pkg/volume/util/subpath/subpath_linux_test.go:579:73: argument err is overwritten before first use (SA4009)
This patch removes pkg/util/mount completely, and replaces it with the
mount package now located at k8s.io/utils/mount. The code found at
k8s.io/utils/mount was moved there from pkg/util/mount, so the code is
identical, just no longer in-tree to k/k.
This fixes the issue where "childFD" from syscall.Openat is assigned to
a local variable inside the for loop, instead of the correct one in the
function scope. This results in that when trying to close the "childFD"
in the function scope, it will be equal to "-1", instead of the correct
value.
As part of moving pkg/util/mount out of tree, the NSEnter implementation
of mount.Interface needs to be relocated out of pkg/util/mount, as it is
K8s specific. This patch relocates that mounter implementation to
pkg/volume/util/nsenter.
Since the NSEnter mounter shares a lot of its logic with the Linux
mounter implementation, many of the previously private methods of the
Linux mounter are now made public to maintain that shared code.
Additionaly, it was observed that *all* mount.Interface implemenations
were using the same common method for IsNotMountPoint, so this patch
removes that method from the mount.Interface definition and just exports
the common implementation instead.
Files in pkg/util/mount that contain significant code implementation for
subpaths are moved to a new package at pkg/volume/util/subpath. This
move is done in order to preserve git history.
