Ubuntu22.04 阿里云环境安装 k8s 以及 Cilium

这篇文章介绍如何在阿里云虚拟机上安装 k8s，并使用 Cilium 作为网络插件。

准备工作

准备如下 3 台机器，操作系统均为 ubuntu 22.04

虚拟机	IP
master	172.16.131.181
worker1	172.16.131.180
worker2	172.16.131.182

• 在每个节点上，增加主机名配置：

cat >> /etc/hosts << EOF
172.16.131.181 master
172.16.131.180 worker1
172.16.131.182 worker2
EOF

• 关闭 swap 分区

# 查看 swapoff 版本
swapoff --version

# 临时关闭
swapoff -a

# 永久关闭
sed  -ri 's/.*swap.*/#&/' /etc/fstab

# 检查的确关闭了：输出为空
swapon -v

• 修改网络配置

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

安装 k8s

安装容器运行时

• 首先先清理环境中已有的 docker 相关组件：

sudo apt-get remove docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

• 安装 containerd：

curl -# -O  https://mirrors.aliyun.com/docker-ce/linux/ubuntu/dists/jammy/pool/stable/amd64/containerd.io_1.6.25-1_amd64.deb

dpkg -i containerd.io_1.6.25-1_amd64.deb

• 导出默认配置

containerd config default > /etc/containerd/config.toml

• 修改生成的默认配置 /etc/containerd/config.toml

修改sandbox_image行替换为aliyun的pause镜像
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"

配置 systemd cgroup 驱动
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

配置镜像加速
[plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins. "io.contianerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry.aliyuncs.com"]

• 启动 containerd 服务

systemctl enable containerd
systemctl restart containerd

安装 kubeadm、kubelet 以及 kubectl

接下来在所有主机上安装 kubeadm、kubelet 以及 kubectl，有一点需要注意，需要自行确保之后 kubeadm 安装的 k8s 控制平面与 kubelet、kubectl 兼容。

同样使用阿里云镜像源来安装以上组件：

apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

初始化 master

首先生成初始化文件：

kubeadm config print init-defaults > kubeadm-init.yaml

对生成的 kubeadm-init.yaml，需要修改如下配置项：

• advertiseAddress: 1.2.3.4：修改为本机地址
• imageRepository: registry.k8s.io 替换为 imageRepository: registry.aliyuncs.com/google_containers
• name: node 需要修改为本机真实的主机名称，这里是 master

接下来下载镜像，要下载的镜像可以通过 kubeadm config images list 提前查看：

# kubeadm config images list
registry.k8s.io/kube-apiserver:v1.28.4
registry.k8s.io/kube-controller-manager:v1.28.4
registry.k8s.io/kube-scheduler:v1.28.4
registry.k8s.io/kube-proxy:v1.28.4
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1

执行如下命令下载镜像：

kubeadm config images pull --config kubeadm-init.yaml

执行初始化：

# kubeadm init --config kubeadm-init.yaml

......
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.131.181:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:96eff832d9eb08f00955d9e33e96fe9b6e560ee30cbf86e742eca781e73b6af2

接下来配置环境信息，让当前用户可以执行 kubectl 命令：

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

加入 worker 节点

根据 master 节点初始化成功后的提示，在两个 worker 节点上分别执行如下命令，将 worker 节点加入进集群：

kubeadm join 172.16.131.181:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:96eff832d9eb08f00955d9e33e96fe9b6e560ee30cbf86e742eca781e73b6af2

在 master 节点上，执行如下命令，确认所有节点都已经加入，由于当前还没有配置网络插件，所有节点状态都是 NotReady

# kubectl get node
NAME      STATUS     ROLES           AGE    VERSION
master    NotReady   control-plane   2m2s   v1.28.2
worker1   NotReady   <none>          77s    v1.28.2
worker2   NotReady   <none>          74s    v1.28.2

重新初始化集群

如果需要重新初始化集群，执行如下命令：

kubeadm reset

手动删除环境配置：

rm -rf $HOME/.kube

如果有网络相关配置，还需要注意网络配置的清理。

安装 Cilium

安装 Cilium CLI

首先需要安装 Cilium CLI，它可以用于安装 cilium：

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

确认 Cilium CLI 安装成功：

cilium version --client

安装 Cilium

执行如下命令，安装 Cilium

cilium install --version 1.14.4

通过如下命令，确认 Cilium 状态正常：

# cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:            cilium-operator    Running: 1
                       cilium             Running: 3
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.14.4
Image versions         cilium             quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.4@sha256:f0f05e4ba3bb1fe0e4b91144fa4fea637701aba02e6c00b23bd03b4a7e1dfd55: 1

可以执行如下命令，进行连通性测试：

# cilium connectivity test
......
✅ All 44 tests (295 actions) successful, 14 tests skipped, 1 scenarios skipped.

网络插件安装成功后，节点状态也变成 Ready 了：

# kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
master    Ready    control-plane   28m   v1.28.2
worker1   Ready    <none>          28m   v1.28.2
worker2   Ready    <none>          28m   v1.28.2

Reference

• 基于Ubuntu-22.04 kubeadm安装K8s-v1.25.0
• 跟着官方文档从零搭建K8S
• Installing kubeadm
• Cilium Quick Installation