dnf update failure from category flake

[coreydaley]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_networking/595/console

Error: Failed to synchronize cache for repo 'updates'
The command '/bin/sh -c dnf -y update && dnf -y install docker glibc-langpack-en iptables openssh-clients openssh-server && dnf -y remove docker-v1.10-migrator glibc-all-langpacks selinux-policy-minimum && dnf clean all' returned a non-zero code: 1
[ERROR] PID 21263: hack/dind-cluster.sh:269: `${DOCKER_CMD} build -t "${image_name}" .` exited with status 1.
[INFO]      Stack Trace: 
[INFO]        1: hack/dind-cluster.sh:269: `${DOCKER_CMD} build -t "${image_name}" .`
[INFO]        2: hack/dind-cluster.sh:259: build-image
[INFO]        3: hack/dind-cluster.sh:356: build-images
[INFO]   Exiting with code 1.
[ERROR] PID 11020: test/extended/networking.sh:300: `${CLUSTER_CMD} build-images` exited with status 1.
[INFO]      Stack Trace: 
[INFO]        1: test/extended/networking.sh:300: `${CLUSTER_CMD} build-images`
[INFO]   Exiting with code 1.
/data/src/github.com/openshift/origin/hack/lib/log/system.sh: line 31: 21259 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"
[INFO] Stacked plot for log subset "iops" written to /tmp/openshift/test-extended/networking/logs/iops.pdf
[INFO] Stacked plot for log subset "paging" written to /tmp/openshift/test-extended/networking/logs/paging.pdf
[INFO] Stacked plot for log subset "cpu" written to /tmp/openshift/test-extended/networking/logs/cpu.pdf
[INFO] Stacked plot for log subset "queue" written to /tmp/openshift/test-extended/networking/logs/queue.pdf
[INFO] Stacked plot for log subset "memory" written to /tmp/openshift/test-extended/networking/logs/memory.pdf
[ERROR] PID 10982: test/extended/networking-minimal.sh:6: `NETWORKING_E2E_MINIMAL=1 "${OS_ROOT}/test/extended/networking.sh"` exited with status 1.
[INFO]      Stack Trace: 
[INFO]        1: test/extended/networking-minimal.sh:6: `NETWORKING_E2E_MINIMAL=1 "${OS_ROOT}/test/extended/networking.sh"`
[INFO]   Exiting with code 1.
Error while running ssh/sudo command: 
set -e
pushd /data/src/github.com/openshift//origin >/dev/null
export PATH=$GOPATH/bin:$PATH

[knobunc]

@danmcp I don't think this is a networking (well... at least the SDN) failure. The test is bombing because the dnf command failed. I'm not sure what to do with this bug... any thoughts?

[danmcp]

@knobunc AFAICT, it's logic in the networking tests failing while calling yum. I can't say I have seen that particular error.

@stevekuznetsov Does it ring a bell for you?

It looks like @juanvallejo hit the same error., so we should probably figure out what's causing it. The solution is probably going to be a retry a changing yum config.

[stevekuznetsov]

This is a dupe of #11452 which is describing network/infra flakes around the Fedora @updates repo, which other projects that interact with dnf also are seeing recently. This is best solved in Origin by breaking all of our image builds into two -- one to inject dependencies, and another to add volatile code like oc. This way we can cache the dependency layer (which changes slowly) in the base_ami. See also #9622