kubernetes/kubernetes-MD/gitlab-ci基于kubernetes+harbo...

26 KiB
Raw Blame History

gitlab-ci基于kubernetes+harbor

著作:行癫 <盗版必究>


image-20220514010742541

一:环境准备

1.kubernetes集群

集群环境检查

[root@master ~]# kubectl get node
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   17d   v1.23.1
node-1   Ready    <none>                 17d   v1.23.1
node-2   Ready    <none>                 17d   v1.23.1
node-3   Ready    <none>                 17d   v1.23.1

注意node节点的运行内存需要大于5G以上

2.harbor仓库

仓库检查

image-20220513232903669

3.NFS提供PVC

[root@nfs-harbor ~]# exportfs -rv
exporting *:/data/storage/k8s/gitlab/gitlab
exporting *:/data/storage/k8s/gitlab/redis
exporting *:/data/storage/k8s/gitlab/postgresql

gitlab-ce部署

1.创建命名空间

[root@master gitlab]# kubectl create namespace kube-ops

2.postgresql数据库

Deployment yaml文件

[root@master gitlab]# cat gitlab-postgresql.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgresql
  namespace: kube-ops
  labels:
    name: postgresql
spec:
  replicas: 1
  selector:
    matchLabels:
      name: postgresql
  template:
    metadata:
      name: postgresql
      labels:
        name: postgresql
    spec:
      containers:
      - name: postgresql
        image: daocloud.io/library/postgres:9.0.20
        env:
        - name: DB_USER
          value: gitlab
        - name: DB_PASS
          value: passw0rd
        - name: DB_NAME
          value: gitlab_production
        - name: DB_EXTENSION
          value: pg_trgm
        ports:
        - name: postgres
          containerPort: 5432
        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: data
      volumes:
      - name: data
        nfs:
          server: 10.0.0.230
          path: /data/storage/k8s/gitlab/postgresql
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  name: postgresql
  namespace: kube-ops
  labels:
    name: postgresql
spec:
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres
  selector:
    name: postgresql

创建Depolyment

[root@master gitlab]# kubelet create -f gitlab-postgresql.yaml

查看Deployment和Pod

[root@master gitlab]# kubectl get deployment -n kube-ops
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
postgresql   1/1     1            1           97m
[root@master gitlab]# kubectl get pod -n kube-ops
NAME                         READY   STATUS    RESTARTS      AGE
postgresql-cccb54fff-2gczp   1/1     Running   0             99m

image-20220513233712291

image-20220513233722978

查看service

[root@master gitlab]# kubectl get svc -n kube-ops
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                     AGE
postgresql   ClusterIP   10.103.7.249    <none>        5432/TCP                    100m

3.redis部署

创建Deployment yaml文件

[root@master gitlab]# cat gitlab-redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: kube-ops
  labels:
    name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      name: redis
  template:
    metadata:
      name: redis
      labels:
        name: redis
    spec:
      containers:
      - name: redis
        image: 10.0.0.230/xingdian/redis:v1
        imagePullPolicy: IfNotPresent
        ports:
        - name: redis
          containerPort: 6379
        volumeMounts:
        - mountPath: /var/lib/redis
          name: data
        livenessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        nfs:
          server: 10.0.0.230
          path: /data/storage/k8s/gitlab/redis
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: kube-ops
  labels:
    name: redis
spec:
  ports:
    - name: redis
      port: 6379
      targetPort: redis
  selector:
    name: redis

创建Deployment

[root@master gitlab]# kubectl create -f gitlab-redis.yaml

查看Deployment和Pod

[root@master gitlab]# kubectl get deployment -n kube-ops
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
redis        1/1     1            1           104m
[root@master gitlab]# kubectl get pod -n kube-ops
NAME                         READY   STATUS    RESTARTS       AGE
redis-7786bc4f96-lxhlj       1/1     Running   0              104m

4.gitlab-ce部署

创建Deployment yaml文件

[root@master gitlab]# cat gitlab.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: kube-ops
  labels:
    name: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab
  template:
    metadata:
      name: gitlab
      labels:
        name: gitlab
    spec:
      containers:
      - name: gitlab
        image: 10.0.0.230/xingdian/gitlab-ce:v1
        imagePullPolicy: IfNotPresent
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: GITLAB_TIMEZONE
          value: Beijing
        - name: GITLAB_SECRETS_DB_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_SECRETS_SECRET_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_SECRETS_OTP_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_ROOT_PASSWORD
          value: xingdian123
        - name: GITLAB_ROOT_EMAIL
          value: zhuangyaovip@163.com
        - name: GITLAB_HOST
          value: 0.0.0.0:30004
        - name: GITLAB_PORT
          value: "80"
        - name: GITLAB_SSH_PORT
          value: "22"
        - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
          value: "true"
        - name: GITLAB_NOTIFY_PUSHER
          value: "false"
        - name: GITLAB_BACKUP_SCHEDULE
          value: daily
        - name: GITLAB_BACKUP_TIME
          value: 01:00
        - name: DB_TYPE
          value: postgres
        - name: DB_HOST
          value: postgresql
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          value: gitlab
        - name: DB_PASS
          value: passw0rd
        - name: DB_NAME
          value: gitlab_production
        - name: REDIS_HOST
          value: redis
        - name: REDIS_PORT
          value: "6379"
        ports:
        - name: http
          containerPort: 80
        - name: ssh
          containerPort: 22
        volumeMounts:
        - mountPath: /home/git/data
          name: data
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 180
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        nfs:
          server: 10.0.0.230
          path: /data/storage/k8s/gitlab/gitlab
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  name: gitlab
  namespace: kube-ops
  labels:
    name: gitlab
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30004
    - name: ssh
      port: 22
      targetPort: ssh
  selector:
    name: gitlab

创建Deployment

[root@master gitlab]# kubectl create -f gitlab.yaml

查看Deployment和Pod

[root@master gitlab]# kubectl get deployment -n kube-ops
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
gitlab       1/1     1            1           108m
[root@master gitlab]# kubectl get pod -n kube-ops
NAME                         READY   STATUS    RESTARTS       AGE
gitlab-696d568999-8zg75      1/1     Running   1 (105m ago)   109m

注意gitlab启动较慢我们需要耐心等待

image-20220513235023237

image-20220513235044997

gitlab-runner部署

1.创建configmap

创建config-configmap.yaml文件

[root@master gitlab-runner]# cat config-configmap.yaml
apiVersion: v1
data:
  REGISTER_NON_INTERACTIVE: "true"
  REGISTER_LOCKED: "false"
  METRICS_SERVER: "0.0.0.0:9100"
  CI_SERVER_URL: "http://10.0.0.220:30004/ci"  # k8s内gitlab服务的通信地址格式:svc.namespace.svc.cluster.local, 同时加上/ci这个prefix这里也可以使用外网访问地址
  RUNNER_REQUEST_CONCURRENCY: "4"
  RUNNER_EXECUTOR: "kubernetes"
  KUBERNETES_NAMESPACE: "kube-ops" # 服务运行的namespace
  KUBERNETES_PRIVILEGED: "true"
  KUBERNETES_CPU_LIMIT: "1"
  KUBERNETES_MEMORY_LIMIT: "1Gi"
  KUBERNETES_SERVICE_CPU_LIMIT: "1"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
  KUBERNETES_HELPER_CPU_LIMIT: "500m"
  KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-cm
  namespace: kube-ops

创建scripts-configmap.yaml文件

[root@master gitlab-runner]# cat scripts-configmap.yaml
apiVersion: v1
data:
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml
    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
    wait
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-scripts
  namespace: kube-ops

创建configmap

[root@master gitlab-runner]# kubectl create -f scripts-configmap.yaml
[root@master gitlab-runner]# kubectl create -f config-configmap.yaml

查看:

[root@master gitlab-runner]# kubectl get configmap -n kube-ops
NAME                       DATA   AGE
gitlab-ci-runner-cm        18     22m
gitlab-ci-runner-scripts   1      21m

2.创建secret

需要创建一个Secret用来存储gitlab的token

创建secret.yaml文件

[root@master gitlab-runner]# cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-token
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
data:
  GITLAB_CI_TOKEN: R1IxMzQ4OTQxVVZVWS1yUjh4YXl6aE1zaVp3eDU=   # 这是base64加密Gitlab runner token之后的字符串

生产tokne

[root@master gitlab-runner]# echo -n 'GR1348941UVUY-rR8xayzhMsiZwx5' | openssl base64
R1IxMzQ4OTQxVVZVWS1yUjh4YXl6aE1zaVp3eDU=

创建secret

[root@master gitlab-runner]# kubectl create -f secret.yaml

查看:

[root@master gitlab-runner]# kubectl get secret -n kube-ops
NAME                    TYPE                                  DATA   AGE
default-token-5xql2     kubernetes.io/service-account-token   3      24h
gitlab-ci-token         Opaque                                1      18m
gitlab-ci-token-dwvd5   kubernetes.io/service-account-token   3      10m

3.创建ServiceAccount

创建一个ServiceAccount来有足够的权限做一些事情,因此我们创建一个gitlab-ciServiceAccount

创建ServiceAccount、Role、RoleBinding文件

[root@master gitlab-runner]# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: kube-ops
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io

创建:

[root@master gitlab-runner]# kubectl create -f rbac.yaml

查看:

[root@master gitlab-runner]# kubectl get serviceaccount -n kube-ops
NAME        SECRETS   AGE
default     1         24h
gitlab-ci   1         13m
[root@master gitlab-runner]# kubectl get role -n kube-ops
NAME        CREATED AT
gitlab-ci   2022-05-13T16:42:08Z
[root@master gitlab-runner]# kubectl get rolebinding -n kube-ops
NAME        ROLE             AGE
gitlab-ci   Role/gitlab-ci   14m

4.创建Statefulset

创建statefulset yaml文件

[root@master gitlab-runner]# cat statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitlab-ci-runner
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
spec:
  selector:
    matchLabels:
      app: gitlab-ci-runner
  updateStrategy:
    type: RollingUpdate
  replicas: 1
  serviceName: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec:
      volumes:
      - name: gitlab-ci-runner-scripts
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-scripts
              items:
              - key: run.sh
                path: run.sh
                mode: 0755
      serviceAccountName: gitlab-ci
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        supplementalGroups: [999]
      containers:
      - image: 10.0.0.230/xingdian/gitlab/gitlab-runner:latest
        name: gitlab-ci-runner
        command:
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner-cm
        - secretRef:
            name: gitlab-ci-token
        env:
        - name: RUNNER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true
      restartPolicy: Always

创建:

[root@master gitlab-runner]# kubectl create -f statefulset.yaml

查看:

[root@master gitlab-runner]# kubectl get statefulset -n kube-ops
NAME               READY   AGE
gitlab-ci-runner   1/1     19m
[root@master gitlab-runner]# kubectl get  pod -n kube-ops
NAME                         READY   STATUS    RESTARTS       AGE
gitlab-696d568999-8zg75      1/1     Running   1 			  3h2m
gitlab-ci-runner-0           1/1     Running   0              18m
postgresql-cccb54fff-2gczp   1/1     Running   0              3h3m
redis-7786bc4f96-lxhlj       1/1     Running   0              3h2m

5.gitlab查看注册Runner服务

image-20220514010639780

gitlab-ce使用

1.配置项目启动邮件

Menu---->your project--->admin(其中一个项目)

image-20220514140944925

Configure intergrations---> Add an intergration---> Email on push

image-20220514141021752

添加邮箱(可以先测试连接,后添加)

image-20220514141234577

2.关联harbor仓库

仓库地址:http://10.0.0.230 这是一个本地地址在kubernetes集群外

Menu---->Admin

image-20220514141414315

Settings---->Network

image-20220514141453504

找到Outbound requests

image-20220514141557089

选择允许本地网络

image-20220514141636331

注意:

以上操作就是解决这个问题Import url is blocked: Requests to the local network are not allowed

Menu---->admin(项目)---->configure intergrations

image-20220514141749612

Add an integration---> Harbor

image-20220514141844310

Harbor相关配置仓库地址仓库名称用户名密码

image-20220514142007413

3.CI/CD pipelines

pipelines 管道管道是持续集成、交付和部署的顶级组件作业由runners执行。如果有足够的并发运行者同一阶段的多个作业将并行执行如果一个阶段中的所有作业都成功,则管道将进入下一个阶段;如果一个阶段中的任何作业失败,则(通常)不会执行下一个阶段,并且管道会提前结束;一般来说,管道是自动执行的,一旦创建就不需要干预;但是,有时您也可以手动与管道交互

管道包括:

工作:它定义要做什么。例如,编译或测试代码的作业

阶段:定义何时运行作业。例如,在编译代码的阶段之后运行测试的阶段

四个阶段:

一个build舞台,有一个工作叫做compile

一个test阶段,有两个作业称为test1test2

一个staging舞台,有一个工作叫做deploy-to-stage

一个production舞台,有一个工作叫做deploy-to-prod

4..gitlab-ci.yaml文件:

配置管道行为的全局关键字:

关键词 描述
default 工作关键字的自定义默认值。
include 从其他 YAML 文件导入配置。
stages 流水线阶段的名称和顺序。
variables 为管道中的所有作业定义 CI/CD 变量。
workflow 控制运行什么类型的管道。

使用作业关键字配置的作业:

关键词 描述
after_script 覆盖作业后执行的一组命令。
allow_failure 允许作业失败。失败的作业不会导致管道失败。
artifacts 成功后附加到作业的文件和目录列表。
before_script 覆盖在作业之前执行的一组命令。
cache 应在后续运行之间缓存的文件列表。
coverage 给定作业的代码覆盖率设置。
dast_configuration 在作业级别使用 DAST 配置文件中的配置。
dependencies 通过提供要从中获取工件的作业列表来限制将哪些工件传递给特定作业。
environment 作业部署到的环境的名称。
except 控制何时不创建作业。
extends 此作业继承的配置条目。
image 使用 Docker 镜像。
inherit 选择所有作业继承的全局默认值。
interruptible 定义一个作业是否可以在被较新的运行冗余时取消。
needs 在阶段排序之前执行作业。
only 控制何时创建工作。
pages 上传作业结果以与 GitLab 页面一起使用。
parallel 应并行运行多少个作业实例。
release 指示运行器生成释放对象。
resource_group 限制作业并发。
retry 发生故障时可以自动重试作业的时间和次数。
rules 用于评估和确定作业的选定属性以及是否已创建的条件列表。
script 由运行程序执行的 Shell 脚本。
secrets CI/CD 是工作需要的秘密。
services 使用 Docker 服务镜像。
stage 定义作业阶段。
tags 用于选择跑步者的标签列表。
timeout 定义优先于项目范围设置的自定义作业级超时。
trigger 定义下游管道触发器。
variables 在工作级别定义工作变量。
when 何时运行作业。

5.案例

示例.gitlab-ci.yaml

default:
  image: ruby:3.0

rspec:
  script: bundle exec rspec

rspec 2.7:
  image: ruby:2.7
  script: bundle exec rspec

构建管道:

image-20220514143753535

image-20220514143808724