# apt-mark unhold kubeadm && apt-get update && apt-getinstall -y kubeadm=1.13.5-00 && apt-mark hold kubeadm
The following packages have unmet dependencies:
kubelet : Depends: kubernetes-cni (= 0.6.0) but 0.7.5-00isto be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.13.2
[upgrade/versions] kubeadm version: v1.13.5
I0418 02:03:25.771423 15661 version.go:237] remote version is much newer: v1.14.1; falling back to: stable-1.13
[upgrade/versions] Latest stable version: v1.13.5
[upgrade/versions] Latest version in the v1.13 series: v1.13.5
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.13.2 v1.13.5
1 x v1.13.5 v1.13.5
Upgrade to the latest version in the v1.13 series:
COMPONENT CURRENT AVAILABLE
API Server v1.13.2 v1.13.5
Controller Manager v1.13.2 v1.13.5
Scheduler v1.13.2 v1.13.5
Kube Proxy v1.13.2 v1.13.5
CoreDNS 1.2.6 1.2.6
Etcd 3.2.24 3.2.24
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.13.5_____________________________________________________________________
できそう!
ということでアップグレード
# kubeadm upgrade apply v1.13.5
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml"and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-04-18-02-04-52/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-kubernetes hash: 1b4484ac0b54c491dc5cfe4d158f31c0
Static pod: kube-apiserver-kubernetes hash: 1b4484ac0b54c491dc5cfe4d158f31c0
Static pod: kube-apiserver-kubernetes hash: 1b4484ac0b54c491dc5cfe4d158f31c0
Static pod: kube-apiserver-kubernetes hash: 1b4484ac0b54c491dc5cfe4d158f31c0
Static pod: kube-apiserver-kubernetes hash: 1b4484ac0b54c491dc5cfe4d158f31c0
[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced
Apr 18 02:09:59kuberneteskubelet[15375]: W0418 02:09:59.079140 15375 eviction_manager.go:160] Failed to admit pod kube-apiserver-kubernetes_kube-system(1b4484ac0b54c491dc5cfe4d158f31c0) - node has conditions: [DiskPressure]
あきらめずにkubeletをrestartしてみる
# systemctl restart kubelet
無事kube-apiserverが復活 何だったんだ・・
# docker ps |grep apiserver1044739d1d01 177db4b8e93a "kube-apiserver --au…" About a minute ago Up About a minute k8s_kube-apiserver_kube-apiserver-kubernetes_kube-system_1b4484ac0b54c491dc5cfe4d158f31c0_0
d0b403dc4ec2k8s.gcr.io/pause:3.1"/pause" About a minute ago Up About a minute k8s_POD_kube-apiserver-kubernetes_kube-system_1b4484ac0b54c491dc5cfe4d158f31c0_0
もう一度アップグレードを実行してみる。
# kubeadm upgrade apply v1.13.5
...
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.13.5". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
今度はうまくいった!
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetes Ready master 129d v1.13.5
worker01 Ready <none> 129d v1.13.2
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetes Ready master 129d v1.13.5
worker01 Ready <none> 129d v1.13.5
masterをv1.14.1にアップグレードする
ここからはあっさりとアップグレードできました。
# apt-mark unhold kubeadm && apt-get update && apt-getinstall -y kubeadm=1.14.1-00 && apt-mark hold kubeadm
...
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0to remove and6not upgraded.
Need toget8,150 kB of archives.
After this operation, 3,171 kB of additional disk space will be used.
Get:1https://packages.cloud.google.com/aptkubernetes-xenial/main amd64 kubeadm amd641.14.1-00 [8,150 kB]
Fetched 8,150 kB in0s (9,463 kB/s)
(Reading database ... 99264 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.14.1-00_amd64.deb ...
Unpacking kubeadm (1.14.1-00) over (1.13.5-00) ...
Setting up kubeadm (1.14.1-00) ...
kubeadm seton hold.
# kubeadm upgrade plan
...
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 2 x v1.13.5 v1.14.1
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.13.5 v1.14.1
Controller Manager v1.13.5 v1.14.1
Scheduler v1.13.5 v1.14.1
Kube Proxy v1.13.5 v1.14.1
CoreDNS 1.2.6 1.3.1
Etcd 3.2.24 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.14.1_____________________________________________________________________
$ kubeadm upgrade apply v1.14.1
...
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.14.1". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
$ apt-mark unhold kubelet &&apt-get update && apt-get install -y kubelet=1.14.1-00 kubectl=1.14.1-00 && apt-mark hold kubelet
...
# systemctl restart kubelet
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetes Ready master 129d v1.14.1
worker01 Ready <none> 129d v1.13.5
workerをv1.14.1にアップグレードする
こちらも簡単。少しマニュアルとは違う指定にしましたが、この辺はお好みで。
# apt-mark unhold kubeadm && apt-get update && apt-getinstall -y kubeadm=1.14.1-00 && apt-mark hold kubeadm
// どうせworkerが1つしかないからdrainは意図的に行わない
# kubeadm upgrade node config --kubelet-version v1.14.1
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead andupgrade the kubelet package using your package manager.
//まちがえてkubeletが上がらないようにholdしておく
# apt-mark unhold kubelet && apt-getinstall -y kubelet=1.14.1-00 kubectl=1.14.1-00 && apt-mark hold kubelet
# systemctl restart kubelet
すべてがv1.14.1になりました
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubernetes Ready master 129d v1.14.1
worker01 Ready <none> 129d v1.14.1
おまけ:metrics-serverの設定
metrics-serverを入れないとkubectl top node, kubectl top podが動作しません。