Large cluster has master endpoints flapping (one master is dropping in and out)

[smarterclayton]

The Kubernetes service endpoints (master IPs, used by clients on the cluster) are maintained by etcd leases on fields in etcd with a 15s TTL, and they are refreshed every 10s. On a very large cluster it appears that the TTL refresh is failing and so the endpoints drop out, which causes endpoint churn and flapping.

The field is not configurable, it should either be so or be tied to another config flag we can tweak.

[smarterclayton]

@pweil- @jupierce we probably need to hang a new flag off master-config but not sure where to do it yet. This isn't likely to be changed, but for right now we are broken

[pweil-]

autoSyncInterval in leaderlease? Maybe on the top level like ControllerLeaseTTL. It's not really an etcd config so it probably doesn't belong in there but it's probably going to be passed in through the NewEtcd call if I'm on the right track. I guess the other option may be LeaderElectionConfiguration.

[smarterclayton]

Since it's master related jordan told me to put it in kubernetesMasterConfig near masterCount and masterIP