AWX- Filesystem space issue because of "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/" directory

Get Help
,
1

AWX- Filesystem space issue because of “/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/” directory.

  • Our AWX running with AWX 24.6.1 version and Filesystem filled very frequently. I have searched other forums but no luck.

I have referred following site to install AWX on my environment.

  • version details
    kubectl version
    Client Version: v1.29.6+k3s2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.6+k3s2
    AWX 24.6.1
    Our gitlab project size is around 7GB, and Filesystem size is 120GB.

  • other details

[root@server ~]# df -h /
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-RootVol00  120G   79G   41G  66% /
[root@server ~]# du -sh /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
60G     /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
[root@server ~]#
[root@server data]# kubectl events
LAST SEEN              TYPE      REASON                OBJECT                                     MESSAGE
19s (x35 over 170m)    Warning   FreeDiskSpaceFailed   Node/server                       (combined from similar events): Failed to garbage collect required amount of images. Attempted to free 39541062041 bytes, but only found 0 bytes eligible to free.
8s (x881 over 3h40m)   Warning   ProvisioningFailed    PersistentVolumeClaim/awx-projects-claim   storageclass.storage.k8s.io "awx-projects-volume" not found
[root@server data]#

@kurokobo

2

Any input will be much appreciable.

Also, sometime, usage getting reduced and again increasing
Is this expected behavior?

Is there any setting available, where we can fix the issue. Please advise.

@kurokobo