CentOS 7.9基于kubeadm部署kubernetes v1.25.2

一、Kubernetes简介

Kubernetes（简称K8S）是开源的容器集群管理系统，可以实现容器集群的自动化部署、自动扩缩容、维护等功能。它既是一款容器编排工具，也是全新的基于容器技术的分布式架构领先方案。在Docker技术的基础上，为容器化的应用提供部署运行、资源调度、服务发现和动态伸缩等功能，提高了大规模容器集群管理的便捷性。

K8S集群中有管理节点与工作节点两种类型。管理节点主要负责K8S集群管理，集群中各节点间的信息交互、任务调度，还负责容器、Pod、NameSpaces、PV等生命周期的管理。工作节点主要为容器和Pod提供计算资源，Pod及容器全部运行在工作节点上，工作节点通过kubelet服务与管理节点通信以管理容器的生命周期，并与集群其他节点进行通信。

二、环境部署准备

1、环境架构

2、配置主机名（以下操作所有节点需要执行）

# Master

[root@localhost ~]# hostnamectl set-hostname k8s-master --static

# Node1

[root@localhost ~]# hostnamectl set-hostname k8s-node1 --static

# Node2

[root@localhost ~]# hostnamectl set-hostname k8s-node2 --static

[root@k8s-master ~]# cat >>/etc/hosts <<EOF

192.168.0.199 k8s-master

192.168.0.198 k8s-node2

192.168.0.197 k8s-node1

EOF

3、关闭防火墙和selinux

[root@k8s-master ~]# systemctl stop firewalld.service

[root@k8s-master ~]# systemctl disable firewalld.service

[root@k8s-master ~]# setenforce 0

[root@k8s-master ~]# sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

4、关闭swap，注释swap分区

[root@k8s-master ~]# swapoff -a

[root@k8s-master ~]# sed -i '/swap/s/^/#/g' /etc/fstab

5、配置内核参数，将桥接的IPv4流量传递到iptables的链

[root@k8s-master ~]# cat >>/etc/modules-load.d/k8s.conf <<EOF

overlay

br_netfilter

EOF

[root@k8s-master ~]# modprobe overlay

[root@k8s-master ~]# modprobe br_netfilter

[root@k8s-master ~]# cat > /etc/sysctl.d/k8s.conf <<EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_forward = 1

EOF

[root@k8s-master ~]# sysctl --system

6、安装Docker

[root@k8s-master ~]# curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

[root@k8s-master ~]# systemctl enable docker

[root@k8s-master ~]# systemctl start docker

7、添加阿里云docker仓库加速器及cgroup驱动

[root@k8s-master ~]# mkdir -p /etc/docker

[root@k8s-master ~]# cat >/etc/docker/daemon.json <<EOF

{

    "registry-mirrors": ["https://fl791z1h.mirror.aliyuncs.com"],

    "exec-opts": ["native.cgroupdriver=systemd"]

}

EOF

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl restart docker

8、安装cri-docker

# 参考: https://github.com/Mirantis/cri-dockerd可以从release版本中直接下载rpm包安装比较简单

# Docker Engine没有实现CRI，而这是容器运行时在Kubernetes中工作所需要的。为此，必须安装一个额外的服务cri-dockerd。cri-dockerd是一个基于传统的内置Docker引擎支持的项目，它在1.24版本从kubelet中移除。

[root@k8s-master ~]# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.6/cri-dockerd-0.2.6-3.el7.x86_64.rpm

[root@k8s-master ~]# yum -y install cri-dockerd-0.2.6-3.el7.x86_64.rpm

9、配置沙箱（pause）镜像

# 需要修改cri-docker.service, 指定network-plugin和pod-infra-container-image(否则后续kubeadm init会无法启动容器报错)

[root@k8s-master ~]# sed -i '/ExecStart/s#dockerd#& --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.8#' /usr/lib/systemd/system/cri-docker.service

[root@k8s-master ~]# systemctl start cri-docker && systemctl enable cri-docker

三、安装kubectl、kubelet、kubeadm

1、添加阿里kubernetes源

[root@k8s-master ~]# cat >/etc/yum.repos.d/kubernetes.repo <<EOF

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

2、安装kubectl、kubelet、kubeadm

[root@k8s-master ~]# yum -y install kubectl-1.25.2 kubelet-1.25.2 kubeadm-1.25.2

[root@k8s-master ~]# systemctl enable kubelet

四、部署Kubernetes Master

1、初始化K8s集群

[root@k8s-master ~]# kubeadm init --kubernetes-version=1.25.2 \

--apiserver-advertise-address=192.168.0.199 \

--image-repository registry.aliyuncs.com/google_containers \

--service-cidr=172.15.0.0/16 --pod-network-cidr=172.16.0.0/16 \

--cri-socket unix:///var/run/cri-dockerd.sock

注：pod的网段为: 172.16.0.0/16，api server地址为Master本机IP，网段可以自定义，不冲突即可。

这一步很关键，由于kubeadm 默认从官网k8s.grc.io下载所需镜像，国内无法访问，因此需要通过--image-repository指定阿里云镜像仓库地址。

集群初始化成功后返回如下信息：

[init] Using Kubernetes version: v1.25.2

[preflight] Running pre-flight checks

[preflight] Pulling images required for setting up a Kubernetes cluster

[preflight] This might take a minute or two, depending on the speed of your internet connection

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

[certs] Using certificateDir folder "/etc/kubernetes/pki"

[certs] Generating "ca" certificate and key

[certs] Generating "apiserver" certificate and key

[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.15.0.1 192.168.0.199]

[certs] Generating "apiserver-kubelet-client" certificate and key

[certs] Generating "front-proxy-ca" certificate and key

[certs] Generating "front-proxy-client" certificate and key

[certs] Generating "etcd/ca" certificate and key

[certs] Generating "etcd/server" certificate and key

[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.199 127.0.0.1 ::1]

[certs] Generating "etcd/peer" certificate and key

[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.199 127.0.0.1 ::1]

[certs] Generating "etcd/healthcheck-client" certificate and key

[certs] Generating "apiserver-etcd-client" certificate and key

[certs] Generating "sa" key and public key

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

[kubeconfig] Writing "admin.conf" kubeconfig file

[kubeconfig] Writing "kubelet.conf" kubeconfig file

[kubeconfig] Writing "controller-manager.conf" kubeconfig file

[kubeconfig] Writing "scheduler.conf" kubeconfig file

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Starting the kubelet

[control-plane] Using manifest folder "/etc/kubernetes/manifests"

[control-plane] Creating static Pod manifest for "kube-apiserver"

[control-plane] Creating static Pod manifest for "kube-controller-manager"

[control-plane] Creating static Pod manifest for "kube-scheduler"

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[apiclient] All control plane components are healthy after 15.006652 seconds

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster

[upload-certs] Skipping phase. Please see --upload-certs

[mark-control-plane] Marking the node k8s-master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]

[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]

[bootstrap-token] Using token: hpg3xk.pgaajueu0r9fpqpl

[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes

[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key

[addons] Applied essential addon: CoreDNS

[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.199:6443 --token hpg3xk.pgaajueu0r9fpqpl \

--discovery-token-ca-cert-hash sha256:186d7bffdba732c14e342a39c16842bb7388f4b43b8766d97dfaacb6af1974e4

注：记录生成的最后部分内容，此内容需要在其它节点加入Kubernetes集群时执行。

2、根据提示创建kubectl

[root@k8s-master ~]# mkdir -p $HOME/.kube

[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

[root@k8s-master ~]# export KUBECONFIG=/etc/kubernetes/admin.conf

3、查看node节点和pod

[root@k8s-master ~]# kubectl get node

NAME          STATUS      ROLES               AGE     VERSION

k8s-master   NotReady   control-plane   7m42s   v1.25.2

[root@k8s-master ~]# kubectl get pod -A

NAMESPACE   NAME                                 READY   STATUS    RESTARTS   AGE

kube-system   coredns-c676cc86f-hz2sz     0/1         Pending   0                7m32s

kube-system   coredns-c676cc86f-mlzs6     0/1         Pending   0                7m32s

kube-system   etcd-k8s-master                   1/1         Running   0               7m45s

kube-system   kube-apiserver-k8s-master            1/1     Running   0          7m45s

kube-system   kube-controller-manager-k8s-master   1/1     Running   0          7m45s

kube-system   kube-proxy-qqrgn                     1/1     Running   0          7m32s

kube-system   kube-scheduler-k8s-master       1/1     Running   0          7m45s

注：node节点为NotReady，因为corednspod没有启动，缺少网络pod

4、安装Pod网络插件calico（CNI）

[root@k8s-master ~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

poddisruptionbudget.policy/calico-kube-controllers created

serviceaccount/calico-kube-controllers created

serviceaccount/calico-node created

configmap/calico-config created

customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created

clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrole.rbac.authorization.k8s.io/calico-node created

clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrolebinding.rbac.authorization.k8s.io/calico-node created

daemonset.apps/calico-node created

deployment.apps/calico-kube-controllers created

5、再次查看pod和node

[root@k8s-master ~]# kubectl get pod -A

NAMESPACE   NAME                                       READY   STATUS    RESTARTS   AGE

kube-system   calico-kube-controllers-59697b644f-9txn4   1/1     Running   0          6m6s

kube-system   calico-node-pf6kt                      1/1     Running   0          6m6s

kube-system   coredns-c676cc86f-hz2sz           1/1     Running   0          14m

kube-system   coredns-c676cc86f-mlzs6           1/1     Running   0          14m

kube-system   etcd-k8s-master                         1/1     Running   0          15m

kube-system   kube-apiserver-k8s-master         1/1     Running   0          15m

kube-system   kube-controller-manager-k8s-master         1/1     Running   0          15m

kube-system   kube-proxy-qqrgn                      1/1     Running   0          14m

kube-system   kube-scheduler-k8s-master        1/1     Running   0          15m

[root@k8s-master ~]# kubectl get node

NAME         STATUS   ROLES             AGE   VERSION

k8s-master   Ready    control-plane   15m   v1.25.2

6、从节点加入Kubernetes集群，注意添加--cri-socket

[root@k8s-node1 ~]# kubeadm join 192.168.0.199:6443 --token hpg3xk.pgaajueu0r9fpqpl \

    --discovery-token-ca-cert-hash sha256:186d7bffdba732c14e342a39c16842bb7388f4b43b8766d97dfaacb6af1974e4 \

    --cri-socket unix:///var/run/cri-dockerd.sock

[root@k8s-node2 ~]# kubeadm join 192.168.0.199:6443 --token hpg3xk.pgaajueu0r9fpqpl \

    --discovery-token-ca-cert-hash sha256:186d7bffdba732c14e342a39c16842bb7388f4b43b8766d97dfaacb6af1974e4 \

    --cri-socket unix:///var/run/cri-dockerd.sock

7、再次查看Node

[root@k8s-master ~]# kubectl get nodes

NAME          STATUS   ROLES             AGE      VERSION

k8s-master   Ready     control-plane   24m      v1.25.2

k8s-node1    Ready    <none>            3m22s   v1.25.2

k8s-node2    Ready    <none>            2m18s   v1.25.2

8、kubectl命令补全功能

[root@k8s-master kube-monitor]# yum -y install bash-completion

[root@k8s-master kube-monitor]# echo "source <(kubectl completion bash)" >> /etc/profile

[root@k8s-master kube-monitor]# source /etc/profile

五、安装kubernetes-dashboard

注：官方部署dashboard的服务没使用nodeport，将yaml文件下载到本地，在service里添加nodeport

1、下载配置文件

[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

2、修改配置文件

[root@master ~]# vim recommended.yaml

# 需要修改的内容如下所示

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort       # 增加内容
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000  # 增加内容
  selector:
    k8s-app: kubernetes-dashboard

[root@k8s-master ~]# kubectl apply -f recommended.yaml

namespace/kubernetes-dashboard created

serviceaccount/kubernetes-dashboard created

service/kubernetes-dashboard created

secret/kubernetes-dashboard-certs created

secret/kubernetes-dashboard-csrf created

secret/kubernetes-dashboard-key-holder created

configmap/kubernetes-dashboard-settings created

role.rbac.authorization.k8s.io/kubernetes-dashboard created

clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created

rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

deployment.apps/kubernetes-dashboard created

service/dashboard-metrics-scraper created

deployment.apps/dashboard-metrics-scraper created

3、查看pod和service

[root@k8s-master ~]# kubectl get pod,svc -n kubernetes-dashboard

NAME                                                                     READY   STATUS    RESTARTS   AGE

pod/dashboard-metrics-scraper-64bcc67c9c-t7hsl   1/1         Running   0               43s

pod/kubernetes-dashboard-5c8bd6b59-82nvx         1/1         Running  0                43s

NAME                                              TYPE         CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE

service/dashboard-metrics-scraper   ClusterIP   172.15.108.161   <none>        8000/TCP         43s

service/kubernetes-dashboard         NodePort  172.15.174.174   <none>        443:30000/TCP   43s

4、通过页面访问，推荐使用firefox浏览器

# 浏览器输入https://172.168.0.199:30000/,如下图所示

5、创建用户

[root@k8s-master ~]# vim dashboard-admin.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kubernetes-dashboard  

---
apiVersion: v1
kind: Secret
metadata:
  name: kubernetes-dashboard-admin
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/service-account.name: "admin"
type: kubernetes.io/service-account-token

[root@k8s-master ~]# kubectl apply -f dashboard-admin.yaml

serviceaccount/admin created

clusterrolebinding.rbac.authorization.k8s.io/admin created

secret/kubernetes-dashboard-admin created

6、创建Token

[root@k8s-master ~]# kubectl -n kubernetes-dashboard create token admin

eyJhbGciOiJSUzI1NiIsImtpZCI6IjE0aExvNVB0N2lLb1lpZTFvaGQwejM5VjFXSWUtV3dsNnR4b3BIcWRNdVEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjY3MDMwOTM1LCJpYXQiOjE2NjcwMjczMzUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbiIsInVpZCI6IjVkZGViNzkxLWNlMDEtNGE3Mi05ZDg1LTcwYWYzZTFjNWY2YyJ9fSwibmJmIjoxNjY3MDI3MzM1LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4ifQ.GYmv7T2ba_2x-w7lGzXP8ijw33cFN2Z1ZgJtjFn-0Uss5E-Y3em915zdj8eqvAmTc_AvkDFJgrfK8JW_CsXWv6efy4SNjX4a-Z3jsGZG-1Lg6X4fwhLwctJSyd5znG1O0xtqiLWbfLgwT2g0VybcKQwRpq-IQe4quHl9gKF9nS4wAO7Fd3H_CJOO5tEratHxrb7FKqjWMSfJiSkVHtQ_6gwTqpLIDWe_f8R5rPwP24GUNr7_-oXx05J-uLHnXefXbb8zlc3BnsL4GmV6Xq9J1pTN0FEJWfBNg610OWtecnVPcyhHF8fLLvrPXIr4MHNfqgpR2RcrHhKeTg9FrIVaUQ

7、输入刚才创建的Token登录，如下图所示

注：登录后如果没有namespace可选，并且提示找不到资源，那么就是权限不足问题导致，可通过以下命令授权

[root@k8s-master ~]# kubectl create clusterrolebinding serviceaccount-cluster-admin --clusterrole=cluster-admin --user=system:serviceaccount:kubernetes-dashboard:kubernetes-dashboard

8、Token获取方式

Token=$(kubectl -n kubernetes-dashboard get secret |awk '/kubernetes-dashboard-admin/ {print $1}')

[root@k8s-master ~]# kubectl describe secrets -n kubernetes-dashboard kubernetes-dashboard-admin

至此，kubeadm在CentOS 7.9部署kubernetes1.25.2完毕。