☸️ 二进制部署 Kubernetes 集群

你需要先阅读的内容

二进制安装相较于其他版本无太大区别，只需要区分每个组件版本的对应关系，重点是在通过一步步部署来掌握 K8S 理论知识
如果你是使用虚拟机部署集群，请不要使用带中文的版本和克隆的虚拟机，主机配置静态 IP 地址
关于集群网络划分，请参考关于 K8S 集群网络划分
我在集群部署前提前准备好了整个部署阶段需要用到的安装包、服务部署文件及相关证书文件
教程中需要用到的相关文件：k8s-install

集群信息相关信息

Master01（192.168.2.13/24）：Master节点(16C 16GB 100GB)
Worker01（192.168.2.14/24）：Worker节点(48C 64GB 1024GB)
K8sVersion：1.23.17
SystemVersion：Rockylinux-8.10
DockerVersion：20.10.X
PodNetWork: 10.244.0.0/12
ServiceNetWork：10.96.0.0/16

系统环境初始化

🔔 环境初始化操作在所有节点执行。因为初始化配置内容比较简单，我这边就没有写出具体执行过程和结果，方便大家阅读

关闭防火墙和安全策略

systemctl disable --now firewalld 
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config && setenforce 0 
echo -n "当前SELinux状态：" && getenforce

配置阿里云源

sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
    -i.bak \
    /etc/yum.repos.d/Rocky-*.repo
dnf clean all && dnf makecache

安装必备软件

dnf install telnet lsof vim wget tcpdump bash-completion net-tools epel-release dnsutils chrony ipvsadm ipset sysstat conntrack libseccomp -y

配置时间同步

sed -i '/^pool 2.rocky.pool.ntp.org iburst/d' /etc/chrony.conf 
echo "server ntp.aliyun.com iburst" >> /etc/chrony.conf 
echo "server ntp.tuna.tsinghua.edu.cn iburst" >> /etc/chrony.conf 
systemctl enable --now chronyd 
chronyc -a makestep 
echo "---当前系统时间：$(date)---"

配置主机名及 Hosts 解析

# 2个节点分别设置主机名、并配置 host 解析
hostnamectl set-hostname master01
hostnamectl set-hostname worker01
vim /etc/hosts 
#添加以下内容 
192.168.2.13 master01 
192.168.2.14 worker01

所有节点关闭Swap分区，建议在部署系统时不添加 Swap 分区

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

所有节点配置 limit

ulimit -SHn 65535 
vim /etc/security/limits.conf 
# 末尾添加如下内容 
* soft nofile 65536 
* hard nofile 131072 
* soft nproc 65535 
* hard nproc 655350 
* soft memlock unlimited 
* hard memlock unlimited

配置节点免密登录

🔔 Master01 节点免密钥登录其他节点，安装过程中生成配置文件和证书均在 Master01 上操作，集群管理默认在 Master01 节点

ssh-keygen -t rsa 
for i in master01 worker01;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

升级内核版本

# 请参考【Rocky Linux 8.X 内核升级至5.X】 
bash kernel_update.sh

所有节点配置 IPVS 模块

modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh 
modprobe -- nf_conntrack

vim /etc/modules-load.d/ipvs.conf 
ip_vs 
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip 

# 然后开启模块加载服务 
systemctl enable --now systemd-modules-load.service

1. IP Virtual Server（IPVS）核心模块 IP_VS

用途：提供 L4 层负载均衡框架，是后续所有 IPVS 调度算法的基础依赖
典型场景：Kubernetes（IPVS 模式）、LVS 集群

2. IP_VS_RR 轮询调度算法

用途：将请求依次均匀分发给后端服务器（无权重区分）

3. IP_VS_WRR 加权轮询调度算法

功能：Weighted Round-Robin（加权轮询）调度算法
用途：根据服务器权重分配流量，性能高的服务器获得更多请求

4. IP_VS_SH 哈希算法

功能：Source Hashing（源地址哈希）调度算法
用途：固定客户端IP与后端服务器的映射，实现会话保持（Session Persistence）

5. NF_CONNTRACK (nf_conntrack 模块)

功能：Netfilter连接跟踪模块
用途：跟踪网络连接状态（如TCP/UDP/ICMP），是 NAT、防火墙、负载均衡的基础功能。
典型场景：Kubernetes Service（iptables模式）；防火墙规则（如 -m state –state ESTABLISHED）；Docker/NAT网络地址转换。

配置 K8S 集群内核参数

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
net.ipv4.conf.all.route_localnet = 1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
# 系统配置调整后，需要重启系统或者运行 sysctl 命令方能生效
sysctl --system

🔔 所有初始化配置完成后，重启所有节点操作系统

reboot
# 重启完成后检查模块加载
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

基本组件安装

🔔 完成 Docker、Kubernetes等组件安装

Docker 作为 Runtime

dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
dnf config-manager --enable docker-ce-stable
dnf install docker-ce-20.10.* docker-ce-cli-20.10.* -y
mkdir -p /etc/docker
# 重点配置是将 Docker 的 CgroupDriver 修改成 Systemd
cat > /etc/docker/daemon.json <<-EOF
{
    "registry-mirrors": [
        "https://dockerhub.xisoul.cn",
        "https://hub.littlediary.cn"
    ],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 5,
    "log-opts": {
        "max-size": "300m",
        "max-file": "2"  
    },
    "live-restore": true
}
EOF
systemctl daemon-reload && systemctl enable --now docker

我们为什么要修改 CgroupDriver ？或许这里能找到你想要的答案

K8S 及 Etcd 安装

Master01 节点下载 Kubernetes 安装包，Kubernetes下载地址，Etcd下载地址，我这边提前下载好了

# 解压 Etcd 软件包至 /usr/local/bin 目录
[root@master01 k8s-install]# tar -zxvf etcd-v3.5.6-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.6-linux-amd64/etcd{,ctl}
etcd-v3.5.6-linux-amd64/etcdctl
etcd-v3.5.6-linux-amd64/etcd
[root@master01 k8s-install]# ll /usr/local/bin/
total 40608
-rwxr-xr-x 1 528287 89939 23691264 Nov 21  2022 etcd
-rwxr-xr-x 1 528287 89939 17891328 Nov 21  2022 etcdctl

# 解压 Kubernetes 安装包至 /usr/local/bin 目录
[root@master01 k8s-install]# tar -zxvf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
kubernetes/server/bin/kubelet
kubernetes/server/bin/kube-apiserver
kubernetes/server/bin/kubectl
kubernetes/server/bin/kube-proxy
kubernetes/server/bin/kube-controller-manager
kubernetes/server/bin/kube-scheduler
[root@master01 k8s-install]# ll /usr/local/bin/
total 526076
-rwxr-xr-x 1 528287 89939  23691264 Nov 21  2022 etcd
-rwxr-xr-x 1 528287 89939  17891328 Nov 21  2022 etcdctl
-rwxr-xr-x 1 root   root  126132224 Feb 22  2023 kube-apiserver
-rwxr-xr-x 1 root   root  116068352 Feb 22  2023 kube-controller-manager
-rwxr-xr-x 1 root   root   45174784 Feb 22  2023 kubectl
-rwxr-xr-x 1 root   root  119091888 Feb 22  2023 kubelet
-rwxr-xr-x 1 root   root   42672128 Feb 22  2023 kube-proxy
-rwxr-xr-x 1 root   root   47976448 Feb 22  2023 kube-scheduler

# 核对组件版本信息
[root@master01 k8s-install]# etcdctl version 
etcdctl version: 3.5.6
API version: 3.5
[root@master01 k8s-install]# kubelet --version 
Kubernetes v1.23.17

# 发送组件至其他节点，如果你需要部署多 Master 节点你需要将所有的组件包发送到其他 Master节点
[root@master01 k8s-install]# scp /usr/local/bin/kube{let,-proxy} @worker01:/usr/local/bin/
kubelet                                                                                                       100%  114MB 103.3MB/s   00:01    
kube-proxy                                                                                                    100%   41MB  58.5MB/s   00:00

相关组件证书生成

Master01 节点下载证书生成工具，接下来的操作务必谨慎小心，使用虚拟机环境的同学，建议做个快照

# 因为网络问题，建议使用浏览器访问下载地址提前下载好，记得修改文件名哦
wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson

[root@master01 k8s-install]# mv cfss* /usr/local/bin/
[root@master01 k8s-install]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
[root@master01 k8s-install]# ll /usr/local/bin/
total 538440
-rwxr-xr-x 1 root   root   10376657 Apr 29 09:46 cfssl
-rwxr-xr-x 1 root   root    2277873 Apr 29 09:46 cfssljson
-rwxr-xr-x 1 528287 89939  23691264 Nov 21  2022 etcd
-rwxr-xr-x 1 528287 89939  17891328 Nov 21  2022 etcdctl
-rwxr-xr-x 1 root   root  126132224 Feb 22  2023 kube-apiserver
-rwxr-xr-x 1 root   root  116068352 Feb 22  2023 kube-controller-manager
-rwxr-xr-x 1 root   root   45174784 Feb 22  2023 kubectl
-rwxr-xr-x 1 root   root  119091888 Feb 22  2023 kubelet
-rwxr-xr-x 1 root   root   42672128 Feb 22  2023 kube-proxy
-rwxr-xr-x 1 root   root   47976448 Feb 22  2023 kube-scheduler

Kubernetes 需要 PKI 证书才能进行基于 TLS 的身份验证，如果你想了解证书相关要求，请参阅PKI 证书和要求，如果你想要了解怎么手动配置证书，请参阅手动生成证书，官网给出了详细的描述

示例证书设置的100年过期时间，生产环境不推荐 100 年有效期（安全风险高）

Master01节点创建相关证书目录，生成 Etcd 、K8S 组件证书，我这里提前创建好了 Json 文件

# etcd-ca-csr.json 文件示例，用于 CA 证书签名请求（CSR）
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Chengdu",
      "L": "Chengdu",
      "O": "etcd",
      "OU": "Etcd Security Company"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
# ca-config.json CA文件示例
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}

# 创建相关目录
[root@master01 k8s-install]# mkdir /etc/etcd/ssl -p
[root@master01 k8s-install]# mkdir -p /etc/kubernetes/pki

生成 Etcd 证书

[root@master01 pki]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca
2025/05/02 04:14:23 [INFO] generating a new CA key and certificate from CSR
2025/05/02 04:14:23 [INFO] generate received request
2025/05/02 04:14:23 [INFO] received CSR
2025/05/02 04:14:23 [INFO] generating key: rsa-2048
2025/05/02 04:14:23 [INFO] encoded CSR
2025/05/02 04:14:23 [INFO] signed certificate with serial number 124432598626891262203497234748492141578462897848

# 如果你是多 Master 节点 hostname 赋值示例如下，请勿照抄，注意替换
hostname=127.0.0.1,master01主机名,master02主机名,master03主机名,master01节点IP,master02节点IP，master03节点IP

[root@master01 pki]# cfssl gencert \
>    -ca=/etc/etcd/ssl/etcd-ca.pem \
>    -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
>    -config=ca-config.json \
>    -hostname=127.0.0.1,master01,192.168.2.13 \
>    -profile=kubernetes \
>    etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd
2025/05/02 04:16:50 [INFO] generate received request
2025/05/02 04:16:50 [INFO] received CSR
2025/05/02 04:16:50 [INFO] generating key: rsa-2048
2025/05/02 04:16:50 [INFO] encoded CSR
2025/05/02 04:16:50 [INFO] signed certificate with serial number 262079655992902293098952518220498170673577531012

生成 kubernetes 组件证书

[root@master01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
2025/06/05 14:13:18 [INFO] generating a new CA key and certificate from CSR
2025/06/05 14:13:18 [INFO] generate received request
2025/06/05 14:13:18 [INFO] received CSR
2025/06/05 14:13:18 [INFO] generating key: rsa-2048
2025/06/05 14:13:19 [INFO] encoded CSR
2025/06/05 14:13:19 [INFO] signed certificate with serial number 143803215971407258365554410565371615958052045171

# 10.96.0.1 是 Service 网段，如果你设置的网段跟我不一样，那么你需要修改网段地址
[root@master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,192.168.2.13,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.2.13 -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver
2025/06/05 14:20:31 [INFO] generate received request
2025/06/05 14:20:31 [INFO] received CSR
2025/06/05 14:20:31 [INFO] generating key: rsa-2048
2025/06/05 14:20:31 [INFO] encoded CSR
2025/06/05 14:20:31 [INFO] signed certificate with serial number 498100617916002473219259412806491567242799799331

# 配置 APIServer 的聚合证书
[root@master01 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
2025/06/05 22:17:34 [INFO] generating a new CA key and certificate from CSR
2025/06/05 22:17:34 [INFO] generate received request
2025/06/05 22:17:34 [INFO] received CSR
2025/06/05 22:17:34 [INFO] generating key: rsa-2048
2025/06/05 22:17:34 [INFO] encoded CSR
2025/06/05 22:17:34 [INFO] signed certificate with serial number 392506862833057430110783212276875059568475484702
# 关于聚合证书，APIServer 聚合证书
[root@master01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes  front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
2025/06/05 22:22:54 [INFO] generate received request
2025/06/05 22:22:54 [INFO] received CSR
2025/06/05 22:22:54 [INFO] generating key: rsa-2048
2025/06/05 22:22:54 [INFO] encoded CSR
2025/06/05 22:22:54 [INFO] signed certificate with serial number 74206394661826901417184553530586754741407520281
2025/06/05 22:22:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

# 生成 Controller-Manage 证书
[root@master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -profile=kubernetes manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
2025/06/05 22:31:43 [INFO] generate received request
2025/06/05 22:31:43 [INFO] received CSR
2025/06/05 22:31:43 [INFO] generating key: rsa-2048
2025/06/05 22:31:43 [INFO] encoded CSR
2025/06/05 22:31:43 [INFO] signed certificate with serial number 602018262810732932752040658946404569974660447340
2025/06/05 22:31:43 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

# 设置集群项、环境项、用户项及默认环境（如果你创建的是HA高可用集群，那么这个 --server 赋值应该如何写？）
[root@master01 pki]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.2.13:6443 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Cluster "kubernetes" set.

[root@master01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Context "system:kube-controller-manager@kubernetes" created.

[root@master01 pki]# kubectl config set-credentials system:kube-controller-manager --client-certificate=/etc/kubernetes/pki/controller-manager.pem --client-key=/etc/kubernetes/pki/controller-manager-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
User "system:kube-controller-manager" set.

[root@master01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
Switched to context "system:kube-controller-manager@kubernetes".

# 生成 Scheduler 证书
[root@master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -profile=kubernetes scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
2025/06/05 22:40:53 [INFO] generate received request
2025/06/05 22:40:53 [INFO] received CSR
2025/06/05 22:40:53 [INFO] generating key: rsa-2048
2025/06/05 22:40:53 [INFO] encoded CSR
2025/06/05 22:40:53 [INFO] signed certificate with serial number 687828539097819378318835495427708252840647603882
2025/06/05 22:40:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@master01 pki]# kubectl config set-cluster kubernetes \
>      --certificate-authority=/etc/kubernetes/pki/ca.pem \
>      --embed-certs=true \
>      --server=https://192.168.2.13:6443 \
>      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Cluster "kubernetes" set.

[root@master01 pki]# kubectl config set-credentials system:kube-scheduler \
>      --client-certificate=/etc/kubernetes/pki/scheduler.pem \
>      --client-key=/etc/kubernetes/pki/scheduler-key.pem \
>      --embed-certs=true \
>      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
User "system:kube-scheduler" set.

[root@master01 pki]# kubectl config set-context system:kube-scheduler@kubernetes \
>      --cluster=kubernetes \
>      --user=system:kube-scheduler \
>      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Context "system:kube-scheduler@kubernetes" created.

[root@master01 pki]# kubectl config use-context system:kube-scheduler@kubernetes \
>      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Switched to context "system:kube-scheduler@kubernetes".

# 定义集群链接信息，将名为 kubernetes 的集群配置写入 /etc/kubernetes/admin.kubeconfig 文件中
[root@master01 pki]# cfssl gencert \
>    -ca=/etc/kubernetes/pki/ca.pem \
>    -ca-key=/etc/kubernetes/pki/ca-key.pem \
>    -config=ca-config.json \
>    -profile=kubernetes \
>    admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
2025/06/05 22:48:29 [INFO] generate received request
2025/06/05 22:48:29 [INFO] received CSR
2025/06/05 22:48:29 [INFO] generating key: rsa-2048
2025/06/05 22:48:29 [INFO] encoded CSR
2025/06/05 22:48:29 [INFO] signed certificate with serial number 183586339220662533589282650788255156090853568734
2025/06/05 22:48:29 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@master01 pki]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.2.13:6443  --kubeconfig=/etc/kubernetes/admin.kubeconfig
Cluster "kubernetes" set.

[root@master01 pki]# kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem     --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig
User "kubernetes-admin" set.

[root@master01 pki]# kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes  --user=kubernetes-admin  --kubeconfig=/etc/kubernetes/admin.kubeconfig
Context "kubernetes-admin@kubernetes" created.

[root@master01 pki]# kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig
Switched to context "kubernetes-admin@kubernetes"

# 创建ServiceAccount Key
[root@master01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
....................+++++
.......................................................+++++
e is 65537 (0x010001)
[root@master01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub
writing RSA key

[root@master01 pki]# ll /etc/kubernetes/
total 28
-rw------- 1 root root 6448 Jun  5 22:53 admin.kubeconfig
-rw------- 1 root root 6584 Jun  5 22:40 controller-manager.kubeconfig
drwxr-xr-x 2 root root 4096 Jun  5 22:55 pki
-rw------- 1 root root 6508 Jun  5 22:48 scheduler.kubeconfig

[root@master01 pki]# ls /etc/kubernetes/pki/ | wc -l 
23

Kubernetes 组件配置

🔔 所有节点创建相关目录

mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

Master 节点启动 Etcd Service

vim /etc/etcd/etcd.config.yml 编辑配置文件，注意修改节点IP地址和主机名

name: 'master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.2.13:2380'
listen-client-urls: 'https://192.168.2.13:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.2.13:2380'
advertise-client-urls: 'https://192.168.2.13:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master01=https://192.168.2.13:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

# 编辑 Service 文件 vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

Master 节点创建数据库证书目录，并启动 Etcd Service

[root@master01 pki]# vim /usr/lib/systemd/system/etcd.service
[root@master01 pki]# mkdir /etc/kubernetes/pki/etcd
[root@master01 pki]# ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
[root@master01 pki]# systemctl daemon-reload
[root@master01 pki]# systemctl enable --now etcd
Created symlink /etc/systemd/system/etcd3.service → /usr/lib/systemd/system/etcd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /usr/lib/systemd/system/etcd.service.

[root@master01 pki]# systemctl status etcd.service 
● etcd.service - Etcd Service
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 14:58:29 CST; 7s ago
     Docs: https://coreos.com/etcd/docs/latest/
 Main PID: 13976 (etcd)
    Tasks: 12 (limit: 50001)
   Memory: 21.2M
   CGroup: /system.slice/etcd.service
           └─13976 /usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml

Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.625+0800","caller":"etcdserver/server.go>
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.625+0800","caller":"embed/serve.go:100",>
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.625+0800","caller":"membership/cluster.g>
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.625+0800","caller":"etcdmain/main.go:44">
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.625+0800","caller":"api/capability.go:75>
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.625+0800","caller":"etcdserver/server.go>
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.626+0800","caller":"etcdmain/main.go:50">
Jun 06 14:58:29 master01 systemd[1]: Started Etcd Service.
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.626+0800","caller":"embed/serve.go:146",>
Jun 06 14:58:29 master01 etcd[13976]: {"level":"info","ts":"2025-06-06T14:58:29.627+0800","caller":"embed/serve.go:198",>
lines 1-20/20 (END)

查看 Etcd 服务状态

[root@master01 pki]# export ETCDCTL_API=3
[root@master01 pki]# etcdctl --endpoints="192.168.2.13:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.2.13:2379 | 80891ac0b42748fb |   3.5.6 |   20 kB |      true |      false |         2 |          4 |                  4 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Master 节点启动 APIServer Service

🔔 你是否需要调整集群网络端口范围？
🔔 你规划得集群网络地址段是多少？
🔔 Kubernetes API Server 认证机制有哪些？本次教程使用的是那种认证模式？

编辑 Service 文件 vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --logtostderr=true  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --insecure-port=0  \
      --advertise-address=192.168.2.13 \
      --service-cluster-ip-range=10.96.0.0/16  \
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.2.13:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

启动 Kube-APIServer

[root@master01 pki]# vim /usr/lib/systemd/system/kube-apiserver.service
[root@master01 pki]# systemctl daemon-reload 
[root@master01 pki]# systemctl enable --now kube-apiserver.service 
Created symlink /etc/systemd/system/multi-user.target.wants/kube-apiserver.service → /usr/lib/systemd/system/kube-apiserver.service.
[root@master01 pki]# systemctl status kube-apiserver.service 
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 16:36:54 CST; 6s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 14179 (kube-apiserver)
    Tasks: 26 (limit: 50001)
   Memory: 158.5M
   CGroup: /system.slice/kube-apiserver.service
           └─14179 /usr/local/bin/kube-apiserver --v=2 --logtostderr=true --allow-privileged=true --bind-address=0.0.0.0 --secure-port=6443 --insecure-port=0 --advertise-addr>

Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.330293   14179 storage_rbac.go:315] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-c>
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.335311   14179 storage_rbac.go:315] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-s>
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.340518   14179 storage_rbac.go:315] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-s>
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.345783   14179 storage_rbac.go:315] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provi>
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.350703   14179 storage_rbac.go:315] created rolebinding.rbac.authorization.k8s.io/system:controller:token-clean>
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.355853   14179 storage_rbac.go:315] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-s>
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.380055   14179 alloc.go:329] "allocated clusterIPs" service="default/kubernetes" clusterIPs=map[IPv4:10.96.0.1]
Jun 06 16:36:59 master01 kube-apiserver[14179]: W0606 16:36:59.386144   14179 lease.go:234] Resetting endpoints for master service "kubernetes" to [192.168.2.13]
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.387077   14179 controller.go:611] quota admission added evaluator for: endpoints
Jun 06 16:36:59 master01 kube-apiserver[14179]: I0606 16:36:59.399367   14179 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io

Master 节点启动 ControllerManager Service

🔔 你规划的 Pod 网段是多少？

编辑 Service 文件 vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
      --v=2 \
      --logtostderr=true \
      --address=127.0.0.1 \
      --root-ca-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
      --leader-elect=true \
      --use-service-account-credentials=true \
      --node-monitor-grace-period=40s \
      --node-monitor-period=5s \
      --pod-eviction-timeout=2m0s \
      --controllers=*,bootstrapsigner,tokencleaner \
      --allocate-node-cidrs=true \
      --cluster-cidr=10.244.0.0/12 \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --node-cidr-mask-size=24
      
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

[root@master01 pki]# vim /usr/lib/systemd/system/kube-controller-manager.service
[root@master01 pki]# systemctl daemon-reload 
[root@master01 pki]# systemctl enable --now kube-controller-manager.service 
Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /usr/lib/systemd/system/kube-controller-manager.service.
[root@master01 pki]# systemctl status kube-controller-manager.service 
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 16:45:56 CST; 5s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 14277 (kube-controller)
    Tasks: 17 (limit: 50001)
   Memory: 35.8M
   CGroup: /system.slice/kube-controller-manager.service
           └─14277 /usr/local/bin/kube-controller-manager --v=2 --logtostderr=true --address=127.0.0.1 --root-ca-file=/etc/kubernetes/pki/ca.pem --cluster-signing-cert-file=/>

Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.005925   14277 shared_informer.go:240] Waiting for caches to sync for namespace
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.139967   14277 controllermanager.go:605] Started "replicaset"
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.140035   14277 controllermanager.go:576] Starting "csrapproving"
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.140089   14277 replica_set.go:186] Starting replicaset controller
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.140107   14277 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.189125   14277 controllermanager.go:605] Started "csrapproving"
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.189164   14277 controllermanager.go:576] Starting "nodeipam"
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.189164   14277 certificate_controller.go:118] Starting certificate controller "csrapproving"
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.189189   14277 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving
Jun 06 16:46:00 master01 kube-controller-manager[14277]: I0606 16:46:00.238308   14277 node_ipam_controller.go:91] Sending events to api server.
lines 1-20/20 (END)

Master 节点启动 Scheduler Service

编辑 Service 文件 vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler --v=2 --logtostderr=true --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

[root@master01 pki]# vim /usr/lib/systemd/system/kube-scheduler.service
[root@master01 pki]# systemctl daemon-reload 
[root@master01 pki]# systemctl enable --now kube-scheduler.service 
Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /usr/lib/systemd/system/kube-scheduler.service.
[root@master01 pki]# systemctl status kube-scheduler.service 
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 16:50:55 CST; 7s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 14375 (kube-scheduler)
    Tasks: 20 (limit: 50001)
   Memory: 24.6M
   CGroup: /system.slice/kube-scheduler.service
           └─14375 /usr/local/bin/kube-scheduler --v=2 --logtostderr=true --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Jun 06 16:50:56 master01 kube-scheduler[14375]:     score: {}
Jun 06 16:50:56 master01 kube-scheduler[14375]:   schedulerName: default-scheduler
Jun 06 16:50:56 master01 kube-scheduler[14375]: ------------------------------------Configuration File Contents End Here---------------------------------
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.552411   14375 server.go:139] "Starting Kubernetes Scheduler" version="v1.23.17"
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.554177   14375 tlsconfig.go:200] "Loaded serving cert" certName="Generated self signed cert" certDetail="\"loca>
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.554495   14375 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail=">
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.554548   14375 secure_serving.go:200] Serving securely on [::]:10259
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.555071   14375 tlsconfig.go:240] "Starting DynamicServingCertificateController"
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.655166   14375 leaderelection.go:248] attempting to acquire leader lease kube-system/kube-scheduler...
Jun 06 16:50:56 master01 kube-scheduler[14375]: I0606 16:50:56.662927   14375 leaderelection.go:258] successfully acquired lease kube-system/kube-scheduler

TLS 客户端证书引导配置

在一个 Kubernetes 集群中，工作节点上的组件（kubelet 和 kube-proxy）需要与 Kubernetes 控制平面组件通信，尤其是 kube-apiserver。为了确保通信本身是私密的、不被干扰，并且确保集群的每个组件都在与另一个可信的组件通信，我们强烈建议使用节点上的客户端 TLS 证书。参考文献：TLS 启动引导

🔔 该配置只需要在 Master 主节点执行

将 Kubernetes 集群的连接信息（如 API Server 地址、CA 证书）写入指定的 Kubeconfig 文件

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.2.13:6443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

配置用户认证 Token

kubectl config set-credentials tls-bootstrap-token-user --token=07401b.f395accd246ae52d --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

创建一个上下文（Context），将集群（Cluster）和用户（User）关联起来

kubectl config set-context tls-bootstrap-token-user@kubernetes     --cluster=kubernetes     --user=tls-bootstrap-token-user     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

激活上下文配置

kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

执行配置结果

[root@master01 bootstrap]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.2.13:6443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
Cluster "kubernetes" set.

[root@master01 bootstrap]# kubectl config set-credentials tls-bootstrap-token-user --token=07401b.f395accd246ae52d --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
User "tls-bootstrap-token-user" set.

[root@master01 bootstrap]# kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
Context "tls-bootstrap-token-user@kubernetes" created.

[root@master01 bootstrap]# kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
Switched to context "tls-bootstrap-token-user@kubernetes".

将集群的 admin 配置文件复制到默认路径

mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

检查集群核心组件健康状态，如果状态异常，则不可继续进行后续操作，请检查前期配置是否有误

[root@master01 bootstrap]# kubectl get cs 
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}

创建 Kubernetes TLS 引导资源，为新节点（如 kubelet）加入集群时提供安全的自动证书签发机制

[root@master01 bootstrap]# kubectl create -f bootstrap.secret.yaml
secret/bootstrap-token-07401b created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

Worker 节点配置

复制集群证书文件

cd /etc/kubernetes/
for NODE in worker01; do
    ssh $NODE mkdir -p /etc/kubernetes/pki
    for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
    scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
    done
done

[root@master01 kubernetes]# ls
admin.kubeconfig  bootstrap-kubelet.kubeconfig  controller-manager.kubeconfig  pki  scheduler.kubeconfig
[root@master01 kubernetes]# for NODE in worker01; do
>      ssh $NODE mkdir -p /etc/kubernetes/pki
>      for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
>        scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
>  done
>  done
ca.pem                                                                                                                                                                                                             100% 1411   904.1KB/s   00:00    
ca-key.pem                                                                                                                                                                                                         100% 1679   804.4KB/s   00:00    
front-proxy-ca.pem                                                                                                                                                                                                 100% 1143   490.5KB/s   00:00    
bootstrap-kubelet.kubeconfig                                                                                                                                                                                       100% 2427     1.2MB/s   00:00 

# 核对 Worker 节点证书信息
[root@worker01 ~]# ll /etc/kubernetes/
total 4
-rw------- 1 root root 2427 Jun  6 23:00 bootstrap-kubelet.kubeconfig
drwxr-xr-x 2 root root   64 Jun  6 23:00 pki
[root@worker01 ~]# ll /etc/kubernetes/pki/
total 12
-rw------- 1 root root 1679 Jun  6 23:00 ca-key.pem
-rw-r--r-- 1 root root 1411 Jun  6 23:00 ca.pem
-rw-r--r-- 1 root root 1143 Jun  6 23:00 front-proxy-ca.pem

所有节点启动 Kubelet 服务

所有节点创建相关目录

for NODE in master01 worker01;do
    ssh root@$NODE "mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/"
done

所有节点创建 Service 配置文件，master 和 worker 节点配置一样

[root@worker01 ~]# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kubelet
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

所有节点创建 /etc/kubernetes/kubelet-conf.yml 文件，master 和 worker 节点配置一样，注意修改 DNS 地址

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

所有节点启动 Kubelet 服务

[root@master01 kubernetes]# systemctl daemon-reload 
[root@master01 kubernetes]# systemctl enable --now kubelet.service 
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /usr/lib/systemd/system/kubelet.service.
[root@master01 kubernetes]# systemctl status kubelet.service 
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 23:21:39 CST; 6s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 15537 (kubelet)
    Tasks: 24 (limit: 50001)
   Memory: 56.3M
   CGroup: /system.slice/kubelet.service
           └─15537 /usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hang>

Jun 06 23:21:40 master01 kubelet[15537]: I0606 23:21:40.679025   15537 kubelet.go:2031] "Starting kubelet main sync loop"
Jun 06 23:21:40 master01 kubelet[15537]: E0606 23:21:40.679105   15537 kubelet.go:2055] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 06 23:21:40 master01 kubelet[15537]: I0606 23:21:40.743844   15537 kuberuntime_manager.go:1105] "Updating runtime config through cri with podcidr" CIDR="10.240.0.0/24"
Jun 06 23:21:40 master01 kubelet[15537]: I0606 23:21:40.744334   15537 docker_service.go:364] "Docker cri received runtime config" runtimeConfig="&RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.240.0.0/24,},}"
Jun 06 23:21:40 master01 kubelet[15537]: I0606 23:21:40.744550   15537 kubelet_network.go:76] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.240.0.0/24"
Jun 06 23:21:40 master01 kubelet[15537]: E0606 23:21:40.751785   15537 kubelet.go:2394] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config >
Jun 06 23:21:41 master01 kubelet[15537]: I0606 23:21:41.429353   15537 apiserver.go:52] "Watching apiserver"
Jun 06 23:21:41 master01 kubelet[15537]: I0606 23:21:41.650809   15537 reconciler.go:167] "Reconciler: start to sync state"
Jun 06 23:21:45 master01 kubelet[15537]: I0606 23:21:45.409038   15537 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Jun 06 23:21:45 master01 kubelet[15537]: E0606 23:21:45.610777   15537 kubelet.go:2394] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config >
lines 1-20/20 (END)

验证集群节点信息

[root@master01 kubernetes]# kubectl get nodes -owide
NAME       STATUS     ROLES    AGE     VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                            KERNEL-VERSION                CONTAINER-RUNTIME
master01   NotReady   <none>   8m2s    v1.23.17   192.168.2.13   <none>        Rocky Linux 8.10 (Green Obsidian)   5.4.292-1.el8.elrepo.x86_64   docker://20.10.24
worker01   NotReady   <none>   7m39s   v1.23.17   192.168.2.14   <none>        Rocky Linux 8.10 (Green Obsidian)   5.4.292-1.el8.elrepo.x86_64   docker://20.10.24

配置 Kube-Proxy,只在 Master01 节点操作,注意修改 IP 地址

kubectl -n kube-system create serviceaccount kube-proxy
kubectl create clusterrolebinding system:kube-proxy  --clusterrole system:node-proxier  --serviceaccount kube-system:kube-proxy

SECRET=$(kubectl -n kube-system get sa/kube-proxy --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET --output=jsonpath='{.data.token}' | base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes

kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=https://192.168.2.13:6443     --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig
kubectl config set-credentials kubernetes --token=${JWT_TOKEN}     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config set-context kubernetes     --cluster=kubernetes     --user=kubernetes     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config use-context kubernetes     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
# 复制配置文件至 Worker 节点
scp /etc/kubernetes/kube-proxy.kubeconfig root@worker01:/etc/kubernetes/

所有节点配置 kube-proxy 和 service 文件
vim /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --v=2

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

编辑 vim /etc/kubernetes/kube-proxy.yaml，注意修改clusterCIDR，修改成你定义的 Pod 网络

apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 10.244.0.0/12 
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms

[root@master01 k8s-install]#systemctl daemon-reload 
[root@master01 k8s-install]#systemctl enable --now kube-proxy.service
Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /usr/lib/systemd/system/kube-proxy.service.
[root@master01 k8s-install]# systemctl status kube-proxy.service 
● kube-proxy.service - Kubernetes Kube Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 23:39:47 CST; 20h ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 18415 (kube-proxy)
    Tasks: 20 (limit: 50001)
   Memory: 41.1M
   CGroup: /system.slice/kube-proxy.service
           └─18415 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy.yaml --v=2

Jun 06 23:39:47 master01 kube-proxy[18415]: I0606 23:39:47.328062   18415 shared_informer.go:247] Caches are synced for endpoint slice config
Jun 06 23:39:47 master01 kube-proxy[18415]: I0606 23:39:47.328072   18415 proxier.go:995] "Not syncing ipvs rules until Services and Endpoints have been received from master"
Jun 06 23:39:47 master01 kube-proxy[18415]: I0606 23:39:47.328117   18415 shared_informer.go:247] Caches are synced for node config
Jun 06 23:39:47 master01 kube-proxy[18415]: I0606 23:39:47.328191   18415 service.go:422] "Adding new service port" portName="default/kubernetes:https" servicePort="10.96.0.1:443/TCP"
Jun 06 23:40:25 master01 kube-proxy[18415]: I0606 23:40:25.835067   18415 service.go:306] "Service updated ports" service="kube-system/calico-typha" portCount=1
Jun 06 23:40:25 master01 kube-proxy[18415]: I0606 23:40:25.835189   18415 service.go:422] "Adding new service port" portName="kube-system/calico-typha:calico-typha" servicePort="10.96.108.0:5473/TCP"
Jun 06 23:49:19 master01 kube-proxy[18415]: I0606 23:49:19.954062   18415 service.go:306] "Service updated ports" service="kube-system/calico-typha" portCount=0
Jun 06 23:49:19 master01 kube-proxy[18415]: I0606 23:49:19.954192   18415 service.go:447] "Removing service port" portName="kube-system/calico-typha:calico-typha"
Jun 06 23:49:36 master01 kube-proxy[18415]: I0606 23:49:36.152121   18415 service.go:306] "Service updated ports" service="kube-system/calico-typha" portCount=1
Jun 06 23:49:36 master01 kube-proxy[18415]: I0606 23:49:36.152232   18415 service.go:422] "Adding new service port" portName="kube-system/calico-typha:calico-typha" servicePort="10.96.120.75:5473/TCP"

复制相关配置文件至 Worker 节点

scp /usr/lib/systemd/system/kube-proxy.service  root@worker01:/usr/lib/systemd/system/
scp /etc/kubernetes/kube-proxy.yaml  root@worker01:/etc/kubernetes/

# 启动 Kube-Proxy 服务
[root@worker01]#systemctl daemon-reload 
[root@worker01]#systemctl enable --now kube-proxy.service
Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /usr/lib/systemd/system/kube-proxy.service.

[root@worker01 ~]# systemctl status kube-proxy.service 
● kube-proxy.service - Kubernetes Kube Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2025-06-06 23:39:55 CST; 20h ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 10815 (kube-proxy)
    Tasks: 22 (limit: 411943)
   Memory: 41.7M
   CGroup: /system.slice/kube-proxy.service
           └─10815 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy.yaml --v=2

Jun 06 23:39:55 worker01 kube-proxy[10815]: I0606 23:39:55.210337   10815 proxier.go:995] "Not syncing ipvs rules until Services and Endpoints have been received from master"
Jun 06 23:39:55 worker01 kube-proxy[10815]: I0606 23:39:55.210334   10815 shared_informer.go:247] Caches are synced for node config
Jun 06 23:39:55 worker01 kube-proxy[10815]: I0606 23:39:55.210392   10815 shared_informer.go:247] Caches are synced for endpoint slice config
Jun 06 23:39:55 worker01 kube-proxy[10815]: I0606 23:39:55.210636   10815 service.go:422] "Adding new service port" portName="default/kubernetes:https" servicePort="10.96.0.1:443/TCP"
Jun 06 23:40:25 worker01 kube-proxy[10815]: I0606 23:40:25.832732   10815 service.go:306] "Service updated ports" service="kube-system/calico-typha" portCount=1
Jun 06 23:40:25 worker01 kube-proxy[10815]: I0606 23:40:25.832851   10815 service.go:422] "Adding new service port" portName="kube-system/calico-typha:calico-typha" servicePort="10.96.108.0:5473/TCP"
Jun 06 23:49:19 worker01 kube-proxy[10815]: I0606 23:49:19.951749   10815 service.go:306] "Service updated ports" service="kube-system/calico-typha" portCount=0
Jun 06 23:49:19 worker01 kube-proxy[10815]: I0606 23:49:19.951882   10815 service.go:447] "Removing service port" portName="kube-system/calico-typha:calico-typha"
Jun 06 23:49:36 worker01 kube-proxy[10815]: I0606 23:49:36.150349   10815 service.go:306] "Service updated ports" service="kube-system/calico-typha" portCount=1
Jun 06 23:49:36 worker01 kube-proxy[10815]: I0606 23:49:36.150460   10815 service.go:422] "Adding new service port" portName="kube-system/calico-typha:calico-typha" servicePort="10.96.120.75:5473/TCP"

安装 Calico

🔔 请安装官方推荐版本，我们这里部署 Calico v3.22.0，注意修改 POD_CIDR 值

我这边忘记复制操作过程了，大家执行步骤命令即可

# 编辑 Calico 配置，执行部署
sed -i "s#POD_CIDR#10.244.0.0/12#g" calico.yaml
kubectl create -f calico.yaml 
# 验证 Calico 状态及集群状态
[root@master01 calico]# kubectl get pods -A 
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-6f6595874c-swkjg   1/1     Running   0          20h
kube-system   calico-node-fcnfs                          1/1     Running   0          20h
kube-system   calico-node-krns9                          1/1     Running   0          20h
kube-system   calico-typha-6b6cf8cbdf-fgbtx              1/1     Running   0          20h
[root@master01 calico]# kubectl get nodes -owide
NAME       STATUS   ROLES    AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                            KERNEL-VERSION                CONTAINER-RUNTIME
master01   Ready    <none>   20h   v1.23.17   192.168.2.13   <none>        Rocky Linux 8.10 (Green Obsidian)   5.4.292-1.el8.elrepo.x86_64   docker://20.10.24
worker01   Ready    <none>   20h   v1.23.17   192.168.2.14   <none>        Rocky Linux 8.10 (Green Obsidian)   5.4.292-1.el8.elrepo.x86_64   docker://20.10.24

安装 CoreDNS 服务

🔔 在第188行处将 KUBEDNS_SERVICE_IP 修改为你集群网络的第10个IP，我们这边设置为：10.96.0.10
🔔 CoreDNS 版本 V1.8.6

[root@master01 coredns]# kubectl create -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
[root@master01 coredns]# kubectl get pods -n kube-system | grep coredns
coredns-5db5696c7-p2rwf                    1/1     Running   0          35s

安装 Metrics-Server

在新版的Kubernetes中系统资源的采集均使用Metrics-Server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

# 我们直接使用准备好的部署文件安装 Metrics-Server V0.5.0

[root@master01 metrics-server]# kubectl create -f metrics-server-deployment.yaml 
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
# 验证功能
[root@master01 metrics-server]# kubectl top pod -A 
NAMESPACE     NAME                                       CPU(cores)   MEMORY(bytes)   
kube-system   calico-kube-controllers-6f6595874c-swkjg   3m           43Mi            
kube-system   calico-node-fcnfs                          47m          199Mi           
kube-system   calico-node-krns9                          65m          195Mi           
kube-system   calico-typha-6b6cf8cbdf-fgbtx              4m           42Mi            
kube-system   coredns-5db5696c7-p2rwf                    2m           26Mi            
kube-system   metrics-server-6bf7dcd649-vdrpc            3m           22Mi            
[root@master01 metrics-server]# kubectl top node
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master01   276m         1%     1967Mi          25%       
worker01   158m         0%     969Mi           1%

Kubecc

要想成为扫地僧，需要不断的学习进步，这个世界，在悄悄惩罚那些不改变的人

你需要先阅读的内容

集群信息相关信息

系统环境初始化

基本组件安装

Docker 作为 Runtime

K8S 及 Etcd 安装

相关组件证书生成

生成 Etcd 证书

生成 kubernetes 组件证书

Kubernetes 组件配置

Master 节点启动 Etcd Service

Master 节点启动 APIServer Service

Master 节点启动 ControllerManager Service

Master 节点启动 Scheduler Service

TLS 客户端证书引导配置

Worker 节点配置

复制集群证书文件

所有节点启动 Kubelet 服务

配置 Kube-Proxy,只在 Master01 节点操作,注意修改 IP 地址

安装 Calico

安装 CoreDNS 服务

安装 Metrics-Server

发送评论 编辑评论

发送评论编辑评论