-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Open
Labels
Description
problem
Unable to remove to external node from a cks cluster
versions
ACS 4.22
The steps to reproduce the bug
-
Create a cks cluster
-
Add a external node to the cks cluster > Make sure the cluster state is in running state
-
Remove the external node from the cks cluster
-
Exception observed
logs
2026-02-12 13:16:06,724 DEBUG [c.c.a.ApiServlet] (qtp1390913202-25:[ctx-ffe63d6e]) (logid:1ce21d29) ===START=== 10.0.3.251 -- POST
command=removeNodesFromKubernetesCluster
response=json
id=e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab
nodeids=a37443b0-4424-4649-ba1d-0a5e6acc8929
sessionkey=Ds_HcBdERgXAEdh8KjTbyulPqjM
2026-02-12 13:16:06,724 DEBUG [c.c.a.ApiServlet] (qtp1390913202-25:[ctx-ffe63d6e]) (logid:1ce21d29) Two factor authentication is already verified for the user 2, so skipping
2026-02-12 13:16:06,726 DEBUG [c.c.a.ApiServer] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"28904042-0804-11f1-972e-1e00f10003ad"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-02-12 13:16:06,726 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) Account for user id 2890a7a5-0804-11f1-972e-1e00f10003ad is Root Admin or Domain Admin, all APIs are allowed.
2026-02-12 13:16:06,726 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-02-12 13:16:06,726 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-02-12 13:16:06,739 DEBUG [c.c.a.ApiServer] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) Retrieved cmdEventType from job info: KUBERNETES.CLUSTER.NODES.REMOVE
2026-02-12 13:16:06,740 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-38:[ctx-8933df3d, job-52]) (logid:8fc1114f) Add job-52 into job monitoring
2026-02-12 13:16:06,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-38:[ctx-8933df3d, job-52]) (logid:50e49cb8) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"Ds_HcBdERgXAEdh8KjTbyulPqjM\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"217\",\"id\":\"e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"a37443b0-4424-4649-ba1d-0a5e6acc8929\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab\",\"nodeids\":\"a37443b0-4424-4649-ba1d-0a5e6acc8929\",\"cmdEventType\":\"KUBERNETES.CLUSTER.NODES.REMOVE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":52,"initMsid":32989392143277,"instanceId":1,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"50e49cb8-1611-4766-8d86-cc5931dd06b0"}
2026-02-12 13:16:06,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) submit async job-52, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"Ds_HcBdERgXAEdh8KjTbyulPqjM\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"217\",\"id\":\"e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"a37443b0-4424-4649-ba1d-0a5e6acc8929\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab\",\"nodeids\":\"a37443b0-4424-4649-ba1d-0a5e6acc8929\",\"cmdEventType\":\"KUBERNETES.CLUSTER.NODES.REMOVE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":52,"initMsid":32989392143277,"instanceId":1,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"50e49cb8-1611-4766-8d86-cc5931dd06b0"}
2026-02-12 13:16:06,745 DEBUG [c.c.a.ApiServlet] (qtp1390913202-25:[ctx-ffe63d6e, ctx-af024535]) (logid:1ce21d29) ===END=== 10.0.3.251 -- POST
command=removeNodesFromKubernetesCluster
response=json
id=e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab
nodeids=a37443b0-4424-4649-ba1d-0a5e6acc8929
sessionkey=Ds_HcBdERgXAEdh8KjTbyulPqjM
2026-02-12 13:16:06,760 ERROR [c.c.k.c.a.KubernetesClusterRemoveWorker] (API-Job-Executor-38:[ctx-8933df3d, job-52, ctx-06aa0403]) (logid:50e49cb8) Error trying to remove node a37443b0-4424-4649-ba1d-0a5e6acc8929 from Kubernetes Cluster e79b3bca-0cb8-4b40-9c8a-9baaabcc9cab: Cannot invoke "String.toLowerCase(java.util.Locale)" because the return value of "com.cloud.vm.UserVmVO.getDisplayName()" is null java.lang.NullPointerException: Cannot invoke "String.toLowerCase(java.util.Locale)" because the return value of "com.cloud.vm.UserVmVO.getDisplayName()" is null
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterRemoveWorker.removeNodesFromCluster(KubernetesClusterRemoveWorker.java:99)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterRemoveWorker.removeNodesFromCluster(KubernetesClusterRemoveWorker.java:74)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.removeNodesFromKubernetesCluster(KubernetesClusterManagerImpl.java:2264)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy537.removeNodesFromKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd.execute(RemoveNodesFromKubernetesClusterCmd.java:98)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
...
The issue is not obsereved if a proper name is given to the external node
What to do about it?
The external node should be removed successfully if it contains autogenrated uuid "VM-a37443b0-4424-4649-ba1d-0a5e6acc8929"
Reactions are currently unavailable