首页
关于
留言
接口
搜索
资讯
技术
资源
悦读
杂记
首页
登录
登录
搜索
emer
累计撰写
58
篇文章
累计收到
0
条评论
首页
栏目
资讯
技术
资源
悦读
杂记
首页
登录
页面
首页
关于
留言
接口
包含标签 【K8s安装】 的文章
2023-10-27
K8s集群安装记录及证书更新(1.20.6 docker版)
1、升级系统到centos7.9.2009yum update -y2、修改主机名以示区分(不要带特别符号,最好是字母+数字即可)hostnamectl set-hostname XXXX1 &&/bin/bash[root@centos7demo ~]# hostnamectl set-hostname node1&&/bin/bash3、修改网卡IP[root@master1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33注:/etc/sysconfig/network-scripts/ifcfg-ens33文件里的配置说明:NAME=ens33 #网卡名字,跟DEVICE名字保持一致即可DEVICE=ens33 #网卡设备名,大家ip addr可看到自己的这个网卡设备名,每个人的机器可能这个名字不一样,需要写自己的BOOTPROTO=static #static表示静态ip地址ONBOOT=yes #开机自启动网络,必须是yesIPADDR=192.168.40.180 #ip地址,需要跟自己电脑所在网段一致NETMASK=255.255.255.0 #子网掩码,需要跟自己电脑所在网段一致GATEWAY=192.168.40.2 #网关,在自己电脑打开cmd,输入ipconfig /all可看到DNS1=192.168.40.2 #DNS,在自己电脑打开cmd,输入ipconfig /all可看到 4、检查selinux是否关闭[root@master1 ~]# getenforceDisabled #显示Disabled说明selinux已经关闭,如果是如果未关闭,需要执行命令修改:sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config注意:修改selinux配置文件之后,重启机器,selinux配置才会生效5、配置主机hosts文件,相互之间通过主机名互相访问,修改每台机器的/etc/hosts文件,增加如下三行:192.168.40.180 master1192.168.40.181 master2192.168.40.182 node16、配置主机间免密码登陆a、在master1主机执行:[root@master1 ~]# ssh-keygen -t rsa[root@master1 ~]# ssh-copy-id master1[root@master1 ~]# ssh-copy-id master2[root@master1 ~]# ssh-copy-id node1在master2主机执行:[root@master2 ~]# ssh-keygen -t rsa[root@master2 ~]# ssh-copy-id master2[root@master2 ~]# ssh-copy-id master1[root@master2 ~]# ssh-copy-id node1在node1主机执行:[root@node1 ~]#ssh-keygen -t rsa[root@node1 ~]# ssh-copy-id node1[root@node1 ~]# ssh-copy-id master1[root@node1 ~]# ssh-copy-id master2测试 :ssh + 主机名[root@master2 ~]# ssh master1Last login: Fri Oct 13 21:16:02 2023 from node1[root@master1 ~]# 7、关闭交换分区:a、#临时关闭 swapoff -ab、直接注释掉/etc/fstab 的交换分区挂载,重启后自动生效问题:为什么要关闭swap交换分区?答: Swap的交换分区是硬盘划分出来的,当如果机器内存不够,会使用swap分区,但是swap分区的性能相对内存要低很多,k8s设计的时候为了能提升性能,默认是不允许使用交换分区的。Kubeadm初始化的时候会检测swap是否关闭,如果没关闭,那就会导致初始化失败。如果不想要关闭交换分区,安装k8s的时候可以指定--ignore-preflight-errors=Swap来解决。 8、修改机器内核参数[root@master1 ~]# modprobe br_netfilter[root@master1 ~]# cat /etc/profile[root@master1 ~]# echo "modprobe br_netfilter" >> /etc/profile[root@master1 ~]# cat > /etc/sysctl.d/k8s.conf <<EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1EOF[root@master1 ~]# cat /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1同理其他两台主机操作类似,加载内核配置文件,[root@master1 ~]# sysctl -p /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 19、关闭firewalld防火墙并设置为不再随机启动:systemctl stop firewalld ; systemctl disable firewalld10、配置阿里云的repo源安装rzsz命令:[root@master1 ~]# yum install lrzsz -y安装scp:[root@master1 ~]# yum install openssh-clients -y备份基础repo源[root@master1 ~]# cd /etc/yum.repos.d/[root@master1 yum.repos.d]# mkdir /root/repo.bak[root@master1 yum.repos.d]# mv * /root/repo.bak/下载阿里云的repo源把资料包里的CentOS-Base.repo和epel.repo文件上传到master1主机的/etc/yum.repos.d/目录下或者从网上下载。master2和node1节点的配置可以采用类似操作。也可以备份好原yum源,删除旧yum文件,直接从master1上复制到master2和node1上[root@master1 yum.repos.d]# scp CentOS-Base.repo epel.repo master2:/etc/yum.repos.dCentOS-Base.repo 100% 2523 3.1MB/s 00:00epel.repo 100% 1050 1.2MB/s 00:00[root@master1 yum.repos.d]# scp CentOS-Base.repo epel.repo node1:/etc/yum.repos.dCentOS-Base.repo 100% 2523 1.5MB/s 00:00epel.repo 配置国内阿里云docker的repo源1、安装yum-utils ,命令:yum install yum-utils -y.=2、下载安装docker的repo源 :yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo[root@master1 yum.repos.d]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo已加载插件:fastestmirroradding repo from: http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repograbbing file http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.reporepo saved to /etc/yum.repos.d/docker-ce.repo3、配置安装k8s组件需要的阿里云的repo源 11、配置时间同步(1、#安装时间同步命令ntpdate[root@master1 yum.repos.d]# yum install ntpdate -y(2、执行同步命令:[root@master1 yum.repos.d]# ntpdate cn.pool.ntp.org14 Oct 00:35:14 ntpdate[9954]: no server suitable for synchronization found(3、编写计划任务,每隔一小时同步一次(所有主机节点一样的规则)[root@master1 yum.repos.d]# crontab -e[root@master1 yum.repos.d]# crontab -l/1 /usr/sbin/ntpdate cn.pool.ntp.org(4、#重启crond服务[root@node1 ~]#service crond restart12 安装基础软件包三台主机可以时执行:yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet ipvsadm遇到问题:警告:/var/cache/yum/x86_64/7/epel/packages/epel-release-7-14.noarch.rpm: 头V4 RSA/SHA256 Signature, 密钥 ID 352c64e5: NOKEY从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 检索密钥获取 GPG 密钥失败:[Errno 14] curl#37 - "Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7"解决办法:缺失RPM-GPG-KEY-EPEL-7,进入目录直接wget下载cd /etc/pki/rpm-gpgwget https://archive.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7需要执行清理命令:yum clean all然后就可以正常安装了。13、安装docker服务,docker-ce的20.10.6版本安装命令:yum install docker-ce-20.10.6 docker-ce-cli-20.10.6 containerd.io -y启动docker,设置为开机启动并查看当前docker情况:systemctl start docker && systemctl enable docker && systemctl status docker14、配置docker镜像加速器和驱动[root@master1 ~]# vim /etc/docker/daemon.json {"registry-mirrors":["https://w70c62mv.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com","http://qtid6917.mirror.aliyuncs.com", "https://rncxm540.mirror.aliyuncs.com"],"exec-opts": ["native.cgroupdriver=systemd"]}保存退出来,scp命令复制到master2和node1,scp daemon.json master2:/etc/dockerscp daemon.json node1:/etc/docker修改docker文件驱动为systemd,默认为cgroupfs,kubelet默认使用systemd,两者必须一致才可以。[root@master1 ~]# systemctl daemon-reload&&systemctl restart docker[root@master1 ~]# systemctl status docker14、安装初始化k8s需要的软件包发送安装命令到三台机器 ,yum install -y kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6[root@master1 ~]# yum install -y kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6[root@master2 ~]# yum install -y kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6[root@node1 ~]# yum install -y kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6注:每个软件包的作用Kubeadm: kubeadm是一个工具,用来初始化k8s集群的kubelet: 安装在集群所有节点上,用于启动Pod的kubectl: 通过kubectl可以部署和管理应用,查看各种资源,创建、删除和更新各种组件15、通过keepalive+nginx实现k8s apiserver节点高可用1、安装nginx主备:在master1和master2上做nginx主备安装[root@master1 ~]# yum install nginx keepalived -y[root@master2 ~]# yum install nginx keepalived -y2、修改nginx配置文件。主备的配置需要保持一致 vim /etc/nginx/nginx.com 增加: #四层负载均衡,为两台Master Apiserver组件提供负载均衡stream {log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';access_log /var/log/nginx/k8s-access.lgo main;upstream k8s-apiserver {server 192.168.40.180:6443 weight=5 max_fails=3 fail_timeout=30s;server 192.168.40.181:6443 weight=5 max_fails=3 fail_timeout=30s;}server{操作如下,也可以直接备份原配置文件,然后rz上传本地写好的配置文件。[root@master1 ~]# vim /etc/nginx/nginx.confuser nginx;worker_processes auto;error_log /var/log/nginx/error.log;pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;events {worker_connections 1024;}四层负载均衡,为两台Master apiserver组件提供负载均衡stream {log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';access_log /var/log/nginx/k8s-access.log main;upstream k8s-apiserver { server 192.168.40.180:6443 weight=5 max_fails=3 fail_timeout=30s; server 192.168.40.181:6443 weight=5 max_fails=3 fail_timeout=30s;}server { listen 16443; # 由于nginx与master节点复用,这个监听端口不能是6443,否则会冲突 proxy_pass k8s-apiserver;}}http {log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log /var/log/nginx/access.log main;sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;types_hash_max_size 2048;include /etc/nginx/mime.types;default_type application/octet-stream;server { listen 80 default_server; server_name _; location / { }}}listen 16443;#由于nginx与master节点复用,这个监听端口不能是6443,否则会有冲突proxy_pass k8s-apiserver;}}master2也采用相同操作。3)、keepalived配置 a、主keepalived[root@master1 ~]# vim /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER} vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh"}vrrp_instance VI_1 { state MASTER interface ens33 # 修改为实际网卡名 virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的 priority 100 # 优先级,备服务器设置 90 advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒 authentication { auth_type PASS auth_pass 1111 } # 虚拟IP virtual_ipaddress { 192.168.40.199/24 } track_script { check_nginx } }#vrrp_script:指定检查nginx工作状态脚本(根据nginx状态判断是否故障转移)#virtual_ipaddress:虚拟IP(VIP)[root@master1 ~]# vim /etc/keepalived/check_nginx.sh !/bin/bash1、判断Nginx是否存活counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )if [ $counter -eq 0 ]; then #2、如果不存活则尝试启动Nginx service nginx start sleep 2 #3、等待2秒后再次获取一次Nginx状态 counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" ) #4、再次进行判断,如Nginx还不存活则停止Keepalived,让地址进行漂移 if [ $counter -eq 0 ]; then service keepalived stop fifi[root@master1 ~]# chmod +x /etc/keepalived/check_nginx.shmaster2采用相同的操作,配置文件保存一致。4、启动服务:启动之前先安装nginx-stream模块[root@master1 ~]# yum install nginx-mod-stream -y[root@master1 ~]# systemctl daemon-reload启动nginx和keepalived[root@master1 ~]# systemctl start nginx keepalived&&systemctl enable nginx keepalived查看状况[root@master1 ~]# systemctl status keepalived● keepalived.service - LVS and VRRP High Availability MonitorLoaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)Active: active (running) since 六 2023-10-14 10:59:10 CST; 16s agoMain PID: 19607 (keepalived)CGroup: /system.slice/keepalived.service├─19607 /usr/sbin/keepalived -D├─19608 /usr/sbin/keepalived -D└─19609 /usr/sbin/keepalived -D5、测试vip是否绑定成功[root@master1 ~]# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:0c:29:81:87:5a brd ff:ff:ff:ff:ff:ffinet 192.168.40.180/24 brd 192.168.40.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet 192.168.40.199/24 scope global secondary ens33valid_lft forever preferred_lft foreverinet6 fe80::aaff:e4a0:d160:38d3/64 scope link noprefixroutevalid_lft forever preferred_lft forever6、测试keepalived:停掉master1上的keepalived,Vip会漂移到master2[root@master1 ~]# service keepalived stop[root@master2]# ip addr16、kubeadm初始化k8s集群a、在master1节点上创建kubeadm-config.yml文件[root@master1 ~]# cat kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta2kind: ClusterConfigurationkubernetesVersion: v1.20.6controlPlaneEndpoint: 192.168.40.199:16443imageRepository: registry.aliyuncs.com/google_containersapiServer:certSANs:192.168.40.180192.168.40.181192.168.40.182192.168.40.199networking:podSubnet: 10.244.0.0/16serviceSubnet: 10.96.0.0/16 apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvsb、把初始化k8s集群需要的离线镜像包k8simage-1-20-6.tar.gz上传到master1、master2、node1机器上,手动解压:[root@master1 ~]# docker load -i k8simage-1-20-6.tar.gz[root@master2 ~]# docker load -i k8simage-1-20-6.tar.gz[root@node1 ~]# docker load -i k8simage-1-20-6.tar.gz初始化k8s集群命令:[root@master1]# kubeadm init --config kubeadm-config.yaml --ignore-preflight-errors=SystemVerification特别提醒:--image-repository registry.aliyuncs.com/google_containers为保证拉取镜像不到国外站点拉取,手动指定仓库地址为registry.aliyuncs.com/google_containers。kubeadm默认从k8s.gcr.io拉取镜像。 我们本地有导入到的离线镜像,所以会优先使用本地的镜像。mode: ipvs 表示kube-proxy代理模式是ipvs,如果不指定ipvs,会默认使用iptables,但是iptables效率低,所以我们生产环境建议开启ipvs,阿里云和华为云托管的K8s,也提供ipvs模式。安装成功最后有如下提示:Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.40.199:16443 --token qnmgnl.mk5nisfwa4lbzsc4 \--discovery-token-ca-cert-hash sha256:99902cb959dda1bb32061bedcc364233a6cc5091e0c5c0832277a44f31abc74f 配置kubectl的配置文件config,相当于对kubectl进行授权,这样kubectl命令可以使用这个证书对k8s集群进行管理[root@master1 ~]# mkdir -p $HOME/.kube[root@master1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config[root@master1 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config查看安装状态:[root@master1 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONmaster1 NotReady control-plane,master 5m45s v1.20.6此时集群状态还是NotReady状态,是因为还没有安装网络插件17、扩容k8s集群-添加master节点把master1节点的证书拷贝到master2上,在master2创建证书存放目录:[root@master2 ~]# cd /root/&&mkdir -p /etc/kubernetes/pki/etcd&&mkdir -p ~/.kube/返回master1节点上,把节点证书传一份给master2上。[root@master1 pki]# scp ca.crt ca.key master2:/etc/kubernetes/pki/[root@master1 pki]# scp sa.key sa.pub front-proxy-ca.crt front-proxy-ca.key master2:/etc/kubernetes/pki/[root@master1 pki]# scp ./etcd/ca.crt master2:/etc/kubernetes/pki/etcd/[root@master1 pki]# scp ./etcd/ca.key master2:/etc/kubernetes/pki/etcd/然后返回master1上获取加入集群命令 ,执行:kubeadm token create --print-join-command[root@master1 pki]# kubeadm token create --print-join-commandkubeadm join 192.168.40.199:16443 --token 5u8ixe.x8kcchoipnuoqtt6 --discovery-token-ca-cert-hash sha256:99902cb959dda1bb32061bedcc364233a6cc5091e0c5c0832277a44f31abc74f 然后在master2上执行:(管理节点 加上 --control-plane )kubeadm join 192.168.40.199:16443 --token 5u8ixe.x8kcchoipnuoqtt6 --discovery-token-ca-cert-hash sha256:99902cb959dda1bb32061bedcc364233a6cc5091e0c5c0832277a44f31abc74f --control-plane --ignore-preflight-errors=SystemVerification然后在node1上执行:kubeadm join 192.168.40.199:16443 --token 5u8ixe.x8kcchoipnuoqtt6 --discovery-token-ca-cert-hash sha256:99902cb959dda1bb32061bedcc364233a6cc5091e0c5c0832277a44f31abc74f --ignore-preflight-errors=SystemVerification返回master1查看集群状况:kubectl get nodes[root@master1 pki]# kubectl get nodesNAME STATUS ROLES AGE VERSIONmaster1 NotReady control-plane,master 96m v1.20.6master2 NotReady control-plane,master 37s v1.20.6node1 NotReady 11m v1.20.6可以看到node1的ROLES角色为空,就表示这个节点是工作节点。可以把node1的ROLES变成work,按照如下方法:[root@master1 ~]# kubectl label node node1 node-role.kubernetes.io/worker=worker[root@master1 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONmaster1 NotReady control-plane,master 111m v1.20.6master2 NotReady control-plane,master 15m v1.20.6node1 NotReady worker 26m v1.20.6注意:上面可以看出集群主机的状态都是NotReady状态,说明没有安装网络插件[root@master1 ~]# kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcoredns-7f89b7bc75-dqrxj 0/1 Pending 0 112mcoredns-7f89b7bc75-qzc9p 0/1 Pending 0 112metcd-master1 1/1 Running 0 112metcd-master2 1/1 Running 0 16mkube-apiserver-master1 1/1 Running 0 112mkube-apiserver-master2 1/1 Running 0 16mkube-controller-manager-master1 1/1 Running 1 112mkube-controller-manager-master2 1/1 Running 0 16mkube-proxy-dh22b 1/1 Running 0 112mkube-proxy-mp5xm 1/1 Running 0 27mkube-proxy-rp972 1/1 Running 0 16mkube-scheduler-master1 1/1 Running 1 112mkube-scheduler-master2 1/1 Running 0 16m18、安装kubernetes网络组件-Calico上传calico.yaml到master1上,使用yaml文件安装calico 网络插件安装命令:[root@master1 ~]# kubectl apply -f calico.yaml查询运行状况:[root@master1 ~]# kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcalico-kube-controllers-6949477b58-nzh84 1/1 Running 0 70scalico-node-cggnz 1/1 Running 0 70scalico-node-fm7rv 1/1 Running 0 70scalico-node-k28fk 1/1 Running 0 70scoredns-7f89b7bc75-dqrxj 1/1 Running 0 117mcoredns-7f89b7bc75-qzc9p 1/1 Running 0 117metcd-master1 1/1 Running 0 117metcd-master2 1/1 Running 0 21mkube-apiserver-master1 1/1 Running 0 117mkube-apiserver-master2 1/1 Running 0 21mkube-controller-manager-master1 1/1 Running 1 117mkube-controller-manager-master2 1/1 Running 0 21mkube-proxy-dh22b 1/1 Running 0 117mkube-proxy-mp5xm 1/1 Running 0 32mkube-proxy-rp972 1/1 Running 0 21mkube-scheduler-master1 1/1 Running 1 117mkube-scheduler-master2 1/1 Running 0 21m查看集群运行状况[root@master1 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONmaster1 Ready control-plane,master 118m v1.20.6master2 Ready control-plane,master 21m v1.20.6node1 Ready worker 32m v1.20.619、测试在k8s创建pod是否可以正常访问网络把busybox-1-28.tar.gz上传到node1节点,手动解压[root@node1 ~]# docker load -i busybox-1-28.tar.gz在master1节点上执行:kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh进入容器测试网络:/ # ping www.baidu.comPING www.baidu.com (39.156.66.18): 56 data bytes64 bytes from 39.156.66.18: seq=0 ttl=127 time=39.3 ms通过上面可以看到能访问网络,说明calico网络插件已经被正常安装了20 、测试coredns是否正常[root@master1 ~]# kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- shIf you don't see a command prompt, try pressing enter./ # nslookup kubernetes.default.svc.cluster.localServer: 10.96.0.10Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localName: kubernetes.default.svc.cluster.localAddress 1: 10.96.0.1 kubernetes.default.svc.cluster.local/ # 21、延长k8s证书查看证书有效时间:openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text |grep Not显示如下,通过下面可看到ca证书有效期是10年[root@master1 ~]# openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text |grep NotNot Before: Oct 14 04:45:12 2023 GMTNot After : Oct 11 04:45:12 2033 GMTopenssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep Not显示如下,通过下面可看到apiserver证书有效期是1年[root@master1 ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep NotNot Before: Oct 14 04:45:12 2023 GMTNot After : Oct 13 04:45:12 2024 GMT 把资料包里的update-kubeadm-cert.sh文件上传到master1和master2节点,分别执行如下操作:1)给update-kubeadm-cert.sh证书授权可执行权限[root@master1~]#chmod +x update-kubeadm-cert.sh2)执行下面命令,修改证书过期时间,把时间延长到10年[root@master1 ~]# ./update-kubeadm-cert.sh all3)给update-kubeadm-cert.sh证书授权可执行权限[root@master2~]#chmod +x update-kubeadm-cert.sh4)执行下面命令,修改证书过期时间,把时间延长到10年[root@master2 ~]# ./update-kubeadm-cert.sh all3)在master1节点查询Pod是否正常,能查询出数据说明证书签发完成kubectl get pods -n kube-system显示如下,能够看到pod信息,说明证书签发正常:
2023年-10月-27日
401 阅读
0 评论
技术
2023-10-27
K8s单节点高可用集群安装记录(1-20-6 docker版)
1、通过centos模板机创建三台主机,修改主机名,设置网络静态地址如下:主机1:hmaster1 IP 192.168.40.180主机2:hnode1 IP 192.168.40.181主机3:hnode2 IP 192.168.40.1822、检查SELINUX是否关闭,执行getenforce打开修改 vim /etc/selinux/config编辑 SELINUX=disabled保存退出,重启服务器才可生效。其他命令: sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config #修改selinux配置文件之后,重启机器,selinux配置才能永久生效3、配置主机hosts文件,相互之间通过主机名互相访问,修改每台机器的/etc/hosts文件,增加如下三行:192.168.40.180 hmaster1192.168.40.181 hnode1192.168.40.182 hnode2[root@hmaster1 ~]# vim /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.40.180 hmaster1192.168.40.181 hnode1192.168.40.182 hnode24、配置主机之间无密码登录命令在三主机分别执行:ssh-keygen -t rsa #生成登陆秘钥ssh-copy-id hmaster1ssh-copy-id hnode1ssh-copy-id hnode25、关闭交换分区swap,提升性能临时关闭 root@hmaster1 ~]# swapoff -a#永久关闭:注释掉/etc/fstab里的swap挂载,给swap这行开头加一下注释问题1:为什么要关闭swap交换分区?Swap是交换分区,如果机器内存不够,会使用swap分区,但是swap分区的性能较低,k8s设计的时候为了能提升性能,默认是不允许使用交换分区的。Kubeadm初始化的时候会检测swap是否关闭,如果没关闭,那就初始化失败。如果不想要关闭交换分区,安装k8s的时候可以指定--ignore-preflight-errors=Swap来解决。6、修改机器内核参数,设置IP转发[root@hmaster1 ~]# modprobe br_netfilter[root@hmaster1 ~]# echo "modprobe br_netfilter" >> /etc/profile[root@hmaster1 ~]# cat > /etc/sysctl.d/k8s.conf <<EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1EOF[root@hmaster1 ~]# sysctl -p /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1hnode1和hnode2采用相同操作。7、关闭firewalld防火墙,并设置随机不启动命令1:systemctl stop firewalld ; systemctl disable firewalld也可以执行命令2:systemctl disable firewalld --now如下:[root@hmaster1 ~]# ps -ef|grep firewallroot 753 1 0 12:18 ? 00:00:00 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopidroot 18709 18598 0 18:12 pts/0 00:00:00 grep --color=auto firewall[root@hmaster1 ~]# systemctl disable firewalld --nowRemoved symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.[root@hmaster1 ~]# ps -ef|grep firewallroot 18768 18598 0 18:12 pts/0 00:00:00 grep --color=auto firewall8、配置阿里云的repo源命令: yum install -y yum-utilsyum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo9、安装基础依赖包yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet ipvsadm10、 配置安装k8s组件需要的阿里云的repo源[root@hmaster1 ~]#vim /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/enabled=1gpgcheck=0将新创建好的repo源文件复制到其他节点相应的目录下。[root@hmaster1 ~]# scp /etc/yum.repos.d/kubernetes.repo hnode1:/etc/yum.repos.d/kubernetes.repo 100% 128 160.2KB/s 00:00 [root@hmaster1 ~]# scp /etc/yum.repos.d/kubernetes.repo hnode2:/etc/yum.repos.d/kubernetes.repo 100% 128 167.7KB/s 00:00 11、配置时间同步 ntpdate chrony安装ntpdate命令 [root@hmaster1 ~]# yum install ntpdate -y #跟网络时间做同步 [root@hmaster1 ~]# ntpdate cn.pool.ntp.org #把时间同步做成计划任务 [root@hmaster1 ~]# crontab -e * */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org wq保存退出,重启计划任务二、安装docker服务1、安装 docker-ce多主机同步执行:yum install -y docker-ce-20.10.62、启动docker:systemctl enable docker --now查看docker运行状况systemctl status docker3、配置docker镜像加速器和驱动vim /etc/docker/daemon.json{"registry-mirrors":["https://rsbud4vc.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"],"exec-opts": ["native.cgroupdriver=systemd"]} 修改docker文件驱动为systemd,默认为cgroupfs,kubelet默认使用systemd,两者必须一致才可以。systemctl daemon-reload #加载镜像加速器配置文件systemctl restart docker #重启dockersystemctl status docker #检查docker是否正常,如果不能正常启动,检查配置文件4、安装初始化k8s需要的软件包yum install -y kubelet-1.20.6 kubeadm-1.20.6 kubectl-1.20.6systemctl enable kubelet #设置kubelet随机启动,不然会导致pod无法启动。注:每个软件包的作用Kubeadm: kubeadm是一个工具,用来初始化k8s集群的kubelet: 安装在集群所有节点上,用于启动Pod的kubectl: 通过kubectl可以部署和管理应用,查看各种资源,创建、删除和更新各种组件5、kubeadm初始化k8s集群将镜像包k8simage-1-20-6.tar.gz上传到三台服务器上,手动解压:docker load -i k8simage-1-20-6.tar.gz三个机器都需要解压6、使用kubeadm初始化k8s集群[root@hmaster1~]# kubeadm config print init-defaults > kubeadm.yaml #获取初始化模板文件kubeadm.yaml的修改配置如下:[root@hmaster1 ~]# cat kubeadm.yamlapiVersion: kubeadm.k8s.io/v1beta2bootstrapTokens:groups:system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:signingauthenticationkind: InitConfigurationlocalAPIEndpoint:advertiseAddress: 192.168.40.180bindPort: 6443nodeRegistration:criSocket: /var/run/dockershim.sockname: hmaster1taints:effect: NoSchedulekey: node-role.kubernetes.io/masterapiServer:timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta2certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns:type: CoreDNSetcd:local:dataDir: /var/lib/etcdimageRepository: registry.aliyuncs.com/google_containerskind: ClusterConfigurationkubernetesVersion: v1.20.6networking:dnsDomain: cluster.localserviceSubnet: 10.96.0.0/12podSubnet: 10.244.0.0/16 #指定pod网段, 需要新增加这个scheduler: {}追加如下几行apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationmode: ipvsapiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationcgroupDriver: systemdkubeadm.yaml修改后执行以下命令初始化kubernetes集群kubeadm init --config=kubeadm.yaml --ignore-preflight-errors=SystemVerification初始化成功后会有 kubeadm join ****类似字样,如本次实验的结果如下:********Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.40.180:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:7d94828271cba0b82e91497c7780d0079241c29937e2abd4d814579cb3317de0 扩展:kubeadm init初始化流程分析1) 检查执行 init 命令的用户是否为 root,如果不是 root,直接快速失败(fail fast);2) 检查待安装的 k8s 版本是否被当前版本的 kubeadm 支持(kubeadm 版本 >= 待安装 k8s 版本);3) 检查防火墙,如果防火墙未关闭,提示开放端口 10250;4) 检查端口是否已被占用,6443(或你指定的监听端口)、10257、10259;5) 检查文件是否已经存在,/etc/kubernetes/manifests/*.yaml;6) 检查是否存在代理,连接本机网络、服务网络、Pod网络,都会检查,目前不允许代理;7) 检查容器运行时,使用 CRI 还是 Docker,如果是 Docker,进一步检查 Docker 服务是否已启动,是否设置了开机自启动;8) 对于 Linux 系统,会额外检查以下内容:8.1) 检查以下命令是否存在:crictl、ip、iptables、mount、nsenter、ebtables、ethtool、socat、tc、touch;8.2) 检查 /proc/sys/net/bridge/bridge-nf-call-iptables、/proc/sys/net/ipv4/ip-forward 内容是否为 1;8.3) 检查 swap 是否是关闭状态;9) 检查内核是否被支持,Docker 版本及后端存储 GraphDriver 是否被支持;对于 Linux 系统,还需检查 OS 版本和 cgroup 支持程度(支持哪些资源的隔离);10) 检查主机名访问可达性;11) 检查 kubelet 版本,要高于 kubeadm 需要的最低版本,同时不高于待安装的 k8s 版本;12) 检查 kubelet 服务是否开机自启动;13) 检查 10250 端口是否被占用;14) 如果开启 IPVS 功能,检查系统内核是否加载了 ipvs 模块;15) 对于 etcd,如果使用 Local etcd,则检查 2379 端口是否被占用, /var/lib/etcd/ 是否为空目录;如果使用 External etcd,则检查证书文件是否存在(CA、key、cert),验证 etcd 服务版本是否符合要求;16) 如果使用 IPv6,检查 /proc/sys/net/bridge/bridge-nf-call-iptables、/proc/sys/net/ipv6/conf/default/forwarding 内容是否为 1;以上就是 kubeadm init 需要检查的所有项目了!完成安装前的配置1) 在 kube-system 命名空间创建 ConfigMap kubeadm-config,同时对其配置 RBAC 权限;2) 在 kube-system 命名空间创建 ConfigMap kubelet-config-,同时对其配置 RBAC 权限;3) 为当前节点(Master)打标记:node-role.kubernetes.io/master=;4) 为当前节点(Master)补充 Annotation;5) 如果启用了 DynamicKubeletConfig 特性,设置本节点 kubelet 的配置数据源为 ConfigMap 形式;6) 创建 BootStrap token Secret,并对其配置 RBAC 权限;7) 在 kube-public 命名空间创建 ConfigMap cluster-info,同时对其配置 RBAC 权限;8) 与 apiserver 通信,部署 DNS 服务;9) 与 apiserver 通信,部署 kube-proxy 服务;10) 如果启用了 self-hosted 特性,将 Control Plane 转为 DaemonSet 形式运行;11) 打印 join 语句; #配置kubectl的配置文件config,相当于对kubectl进行授权,这样kubectl命令可以使用这个证书对k8s集群进行管理 [root@hmaster1 ~]# mkdir -p $HOME/.kube [root@hmaster1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@hmaster1 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config [root@hmaster1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION hmaster1 NotReady control-plane,master 18m v1.20.6在hnode1、hnode2工作节点上执行以下命令,加入集群(获取的token一般24小时内有效)kubeadm join 192.168.40.180:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:7d94828271cba0b82e91497c7780d0079241c29937e2abd4d814579cb3317de0 --ignore-preflight-errors=SystemVerification返回hmaster1节点查看[root@hmaster1 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONhmaster1 NotReady control-plane,master 26m v1.20.6hnode1 NotReady <none> 4m49s v1.20.6hnode2 NotReady <none> 10s v1.20.6给工作节点打标签可执行以下命令:kubectl label node hnode2 node-role.kubernetes.io/worker=worker [root@hmaster1 ~]# kubectl label node hnode2 node-role.kubernetes.io/worker=worker node/hnode2 labeled [root@hmaster1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION hmaster1 NotReady control-plane,master 37m v1.20.6 hnode1 NotReady <none> 15m v1.20.6 hnode2 NotReady worker 10m v1.20.6 [root@hmaster1 ~]# kubectl label node hnode1 node-role.kubernetes.io/worker=worker node/hnode1 labeled [root@hmaster1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION hmaster1 NotReady control-plane,master 37m v1.20.6 hnode1 NotReady worker 16m v1.20.6 hnode2 NotReady worker 11m v1.20.6 集群状态还是NotReady状态,因为没有安装网络插件
2023年-10月-27日
350 阅读
0 评论
技术