Pod process being killed for oom, but pod is not over limit #5325
Closed
bpopovich44
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We are having issues with pods being killed with cgroup OOM.
If I run kubectl top , the pod still has memory. And if I check memory from within the pod with free command, the pod still has memory, but the pod job dies. I don't see OOM with top, or describing the pod. But if I check on dmesg on the worker node, I get the following error:
[4999059.287718] Tasks state (memory values in pages):
[4999059.287858] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[4999059.288008] [ 761249] 65535 761249 243 1 28672 0 -998 pause
[4999059.288156] [ 811559] 0 811559 195 1 36864 0 999 stress
[4999059.288316] [ 811674] 0 811674 64196 25140 241664 0 999 stress
[4999059.288432] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-7005b39a75c192ef061900f1fe29f5fad82d5726d9769b709686b5be0d432740.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1162de_654f_492b_ac0c_a1eaba69b349.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1162de_654f_492b_ac0c_a1eaba69b349.slice/cri-containerd-7005b39a75c192ef061900f1fe29f5fad82d5726d9769b709686b5be0d432740.scope,task=stress,pid=811674,uid=0
[4999059.288919] Memory cgroup out of memory: Killed process 811674 (stress) total-vm:256784kB, anon-rss:100320kB, file-rss:240kB, shmem-rss:0kB, UID:0 pgtables:236kB oom_score_adj:999
[4999059.979064] IPv6: ADDRCONF(NETDEV_UP): cali62984e4bc7d: link is not ready
[4999060.063568] IPv6: ADDRCONF(NETDEV_UP): cali62984e4bc7d: link is not ready
[4999060.065830] IPv6: ADDRCONF(NETDEV_CHANGE): cali62984e4bc7d: link becomes ready
[4999231.940738] IPv6: ADDRCONF(NETDEV_UP): calic42f07e0360: link is not ready
[4999233.471091] IPv6: ADDRCONF(NETDEV_UP): cali49842b399e5: link is not ready
[4999233.699231] IPv6: ADDRCONF(NETDEV_UP): cali49842b399e5: link is not ready
[4999233.700773] IPv6: ADDRCONF(NETDEV_CHANGE): cali49842b399e5: link becomes ready
Beta Was this translation helpful? Give feedback.
All reactions