application redeployment is getting failed for OpenShift v3.7.0

[hrishin]

Description

If f8-m-p redeploy the existing application on OpenShift v3.7 then Deployment is not happening correctly.
Its unable to pull the images and status appears ImgPullErr for a long time. After some time with 4-5 retries, it's able to pull the image and deployment happen.

Info

This might be an issue with F8-M-P or OpenShift.

• f-m-p version : v3.5.34
• Maven version (mvn -v) :

• Kubernetes / OpenShift setup and version : OpenShift v3.7.0

• If it's a bug, how to reproduce :

• deploy the application using mvn install or mvn fabric8:deploy
• again invoke mvn insall or mvn fabric8:deploy

• Sample project : [GitHub Clone URL] : https://github.com/fabric8io/fabric8-maven-plugin/tree/master/samples/spring-boot

[hrishin]

Found this issue in regression test execution

[rohanKanojia]

Confirmed, I was able to reproduce the bug you reported. Here are some pod logs:
#1st Deployment

~/work/repos/fabric8-maven-plugin/samples/spring-boot : $ oc get pods -w
NAME                                       READY     STATUS    RESTARTS   AGE
fabric8-maven-sample-spring-boot-1-vlfb8   1/1       Running   0          2m
fabric8-maven-sample-spring-boot-s2i-1-build   0/1       Completed   0         6m
fabric8-maven-sample-spring-boot-s2i-2-build   0/1       Completed   0         3m

#2nd Deployment

fabric8-maven-sample-spring-boot-s2i-3-build   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-s2i-3-build   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-s2i-3-build   0/1       Init:0/2   0         1s
fabric8-maven-sample-spring-boot-s2i-3-build   0/1       Init:0/2   0         2s
fabric8-maven-sample-spring-boot-s2i-3-build   0/1       Init:1/2   0         3s
fabric8-maven-sample-spring-boot-s2i-3-build   0/1       PodInitializing   0         4s
fabric8-maven-sample-spring-boot-s2i-3-build   1/1       Running   0         5s
fabric8-maven-sample-spring-boot-2-deploy   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-2-deploy   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-2-deploy   0/1       ContainerCreating   0         2s
fabric8-maven-sample-spring-boot-s2i-3-build   0/1       Completed   0         11s
fabric8-maven-sample-spring-boot-2-deploy   1/1       Running   0         3s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       ContainerCreating   0         1s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       Running   0         3s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       Init:0/2   0         2s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       Init:0/2   0         3s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       Init:1/2   0         5s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       PodInitializing   0         6s
fabric8-maven-sample-spring-boot-s2i-4-build   1/1       Running   0         7s
fabric8-maven-sample-spring-boot-2-57g6h   1/1       Running   0         21s
fabric8-maven-sample-spring-boot-1-vlfb8   1/1       Terminating   0         5m
fabric8-maven-sample-spring-boot-2-deploy   1/1       Terminating   0         26s
fabric8-maven-sample-spring-boot-1-vlfb8   0/1       Terminating   0         5m
fabric8-maven-sample-spring-boot-2-deploy   0/1       Terminating   0         26s
fabric8-maven-sample-spring-boot-s2i-4-build   0/1       Completed   0         13s
fabric8-maven-sample-spring-boot-3-deploy   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-3-deploy   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-3-deploy   0/1       ContainerCreating   0         1s
fabric8-maven-sample-spring-boot-3-deploy   1/1       Running   0         2s
fabric8-maven-sample-spring-boot-2-57g6h   1/1       Terminating   0         25s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       Terminating   0         26s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       ContainerCreating   0         2s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       Terminating   0         27s
fabric8-maven-sample-spring-boot-2-57g6h   0/1       Terminating   0         27s
fabric8-maven-sample-spring-boot-3-deploy   1/1       Terminating   0         6s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Running   0         4s
fabric8-maven-sample-spring-boot-3-deploy   0/1       Terminating   0         7s
fabric8-maven-sample-spring-boot-4-deploy   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-4-deploy   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-4-deploy   0/1       ContainerCreating   0         2s
fabric8-maven-sample-spring-boot-3-deploy   0/1       Terminating   0         9s
fabric8-maven-sample-spring-boot-3-deploy   0/1       Terminating   0         9s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Terminating   0         8s
fabric8-maven-sample-spring-boot-4-deploy   1/1       Running   0         4s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ContainerCreating   0         1s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Terminating   0         9s
fabric8-maven-sample-spring-boot-1-vlfb8   0/1       Terminating   0         6m
fabric8-maven-sample-spring-boot-1-vlfb8   0/1       Terminating   0         6m
fabric8-maven-sample-spring-boot-2-deploy   0/1       Terminating   0         38s
fabric8-maven-sample-spring-boot-2-deploy   0/1       Terminating   0         38s
fabric8-maven-sample-spring-boot-4-deploy   0/1       Completed   0         8s
fabric8-maven-sample-spring-boot-4-deploy   0/1       Terminating   0         8s
fabric8-maven-sample-spring-boot-4-deploy   0/1       Terminating   0         8s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ErrImagePull   0         8s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Terminating   0         20s
fabric8-maven-sample-spring-boot-3-c5qnv   0/1       Terminating   0         20s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ImagePullBackOff   0         22s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ErrImagePull   0         42s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ImagePullBackOff   0         55s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ErrImagePull   0         1m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ImagePullBackOff   0         1m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ErrImagePull   0         2m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ImagePullBackOff   0         2m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ErrImagePull   0         3m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ImagePullBackOff   0         3m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ErrImagePull   0         6m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       ImagePullBackOff   0         6m
fabric8-maven-sample-spring-boot-5-deploy   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-5-deploy   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-5-deploy   0/1       ContainerCreating   0         1s
fabric8-maven-sample-spring-boot-5-deploy   1/1       Running   0         3s
fabric8-maven-sample-spring-boot-5-rgwz4   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-5-rgwz4   0/1       Pending   0         2s
fabric8-maven-sample-spring-boot-5-rgwz4   0/1       ContainerCreating   0         2s
fabric8-maven-sample-spring-boot-5-rgwz4   0/1       Running   0         3s
fabric8-maven-sample-spring-boot-5-rgwz4   1/1       Running   0         22s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       Terminating   0         8m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       Terminating   0         8m
fabric8-maven-sample-spring-boot-5-deploy   0/1       Completed   0         26s
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       Terminating   0         8m
fabric8-maven-sample-spring-boot-4-rb9r2   0/1       Terminating   0         8m
fabric8-maven-sample-spring-boot-5-deploy   0/1       Terminating   0         26s
fabric8-maven-sample-spring-boot-5-deploy   0/1       Terminating   0         26s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       Pending   0         1s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       ContainerCreating   0         1s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       Terminating   0         1s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       Terminating   0         1s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       Terminating   0         13s
fabric8-maven-sample-spring-boot-3-wbsv7   0/1       Terminating   0         13s

Surprisingly, application pod comes back into a steady state after some time. Need to check why this behaviour is happening.

[mojsha]

I am also running into this issue in a OpenShift 3.7 cluster.

Observations:

1. Initial deploy works.

2. Subsequent deploy with my sample application having a sidecar container and an init container does NOT work for the main application container due to inability to pull image. Sidecar container and init container are able to pull their images successfully.

3. As observed by others, the image pulling does not fail immediately. It takes quite a while.