安全升级 Kubernetes 集群

1 .Safely upgrading Kubernetes clusters 1 

2 .廖宇　Yu Liao (yliaog@github) Staff Software Engineer at Google Kubernetes team. Active contributor to Kubernetes. Confidential + Proprietary Confidential + Proprietary 

3 .徐超　Chao Xu (caesarxuchao@github) Software Engineer at Google Kubernetes team. Active contributor to Kubernetes. Confidential + Proprietary Confidential + Proprietary 

4 .Death, taxes, and upgrades 4 

5 .Death, taxes, and upgrades 5 

6 .Death, taxes, and upgrades 6 

7 .Takeaways 7 

8 .Scope of cluster upgrades • Kubernetes binaries 8 

9 .Agenda 9 

10 .Before you begin... 10 

11 .Before you begin... etcdctl snapshot save backup.db 11 

12 .Before you begin... etcdctl snapshot save backup.db etcdctl snapshot status backup.db 12 

13 .Before you begin... 13 

14 .Before you begin... 14 

15 .Before you begin... 15 

16 .Before you begin... 16 

17 .Tools 17 

18 .Reverse engineering `kubeadm upgrade` 18 

19 .Reverse engineering `kubeadm upgrade` • kubectl drain • kubectl uncordon 19 

20 .Reverse engineering `kubeadm upgrade` 20 

21 .Agenda 21 

22 .Uninterpretable data in etcd store at encoded in batch/v1/job registry/jobs/<namespace>/<name> CREATE batch/v1/job deserialize as batch/v1/job convert to extensions/v1beta1/job read from encoded in batch/v1/job registry/jobs/<namespace>/<name> GET batch/v1/job deserialize as extensions/v1beta1/job convert to batch/v1/job 22 

23 . Uninterpretable data in etcd store at encoded in batch/v1/job registry/jobs/<namespace>/<name> Created today CREATE batch/v1/job deserialize as batch/v1/job convert to extensions/v1beta1/job read from 500 internal server error registry/jobs/<namespace>/<name> Read 1 year later... GET batch/v1/job Failed to deserialize as extensions/v1beta1/job. The apiserver doesn’t have the schema. 23 

24 .Workarounds & solutions 1.KEP 2.https://github.com/kubernetes-sigs/kube-storage-version-migrator 24 

25 .Clients are outdated 25 

26 .Clients are outdated 26 

27 .Policy breaks after upgrade apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration webhooks: - name: enforce-image-policy.kubernetes.io rules: - apiGroups: - "batch" apiVersions: - "v1" resources: - jobs 27 

28 .Policy breaks after upgrade apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration webhooks: - name: enforce-image-policy.kubernetes.io rules: - apiGroups: - "batch" apiVersions: - "v1" resources: - jobs 28 

29 .Policy breaks after upgrade apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration webhooks: - name: enforce-image-policy.kubernetes.io rules: - apiGroups: - "batch" apiVersions: - "*" resources: - jobs 29