Spinnaker 安装配置

环境信息

  • Centos-7 3.10.0-1160
  • Docker Engine 19.03.15
  • Kubernetes 1.21.2

本文档中涉及服务器的操作,都是在 Kubernetes 集群的 Master 节点上进行。如果在其他服务器启动 Halyard 容器,需要确保容器中可以访问到 Kubernetes 集群的 API Server

安装部署步骤

Install Halyard on Docker and Choose Cloud Providers - Kubernetes

宿主机准备容器数据目录

宿主机创建 local Halyard config directory [1]

mkdir /root/spinnaker_data
mkdir /root/spinnaker_data/.kube
mkdir /root/spinnaker_data/.hal

Spinnaker 的部署配置会写入容器目录 /home/spinnaker/.hal,将此配置映射到宿主机,可保证配置持久化。

拷贝 Kubernetes 集群管理配置文件到 /root/spinnaker_data/.kube/config [2]

cp /etc/kubernetes/admin.conf /root/spinnaker_data/.kube/config
chmod 777 /root/spinnaker_data/.kube/config

启动 Halyard 容器

启动 Halyard 容器,并挂载 Halyard 本地配置目录

docker run  \
--name halyard --rm \
-v /root/spinnaker_data/.hal:/home/spinnaker/.hal \
-v /root/spinnaker_data/.kube:/home/spinnaker/.kube \
-it \
us-docker.pkg.dev/spinnaker-community/docker/halyard:stable

配置 Spinnaker 关联 Kubernetes 集群

配置并验证 Halyard 容器可以和 kubernetes Api Server 通信

进入 Halyard 容器

docker exec -it halyard bash

修改 /home/spinnaker/.kube/config,将配置中的 Kubernetes API Server 改为正确的 IP

/home/spinnaker/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQk
server: https://10.150.0.21:6443
name: kubernetes
...

进入 Halyard 容器,验证 Halyard 容器可以和 Kubernetes API Server 通信:

$ docker exec -it halyard bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 24h v1.21.2
k8s-work1 Ready <none> 24h v1.21.2
k8s-work2 Ready <none> 24h v1.21.2

启用 Cloud Provider : kubernetes

在 Halyard 容器中,执行以下命令,启用 Cloud Provider : kubernetes [3]

$ hal config provider kubernetes enable
+ Get current deployment
Success
+ Edit the kubernetes provider
Success
Validation in default:
- WARNING You have not yet selected a version of Spinnaker to
deploy.
? Options include:
- 1.29.0
- 1.28.1
- 1.27.1
- 1.26.7

Validation in default.provider.kubernetes:
- WARNING Provider kubernetes is enabled, but no accounts have been
configured.

+ Successfully enabled kubernetes

添加 kubernetes 账号

在 Halyard 容器中,执行以下命令,添加账号

$ CONTEXT=$(kubectl config current-context)

$ hal config provider kubernetes account add my-k8s-account --context $CONTEXT
+ Get current deployment
Success
+ Add the my-k8s-account account
Success
Validation in default:
- WARNING You have not yet selected a version of Spinnaker to
deploy.
? Options include:
- 1.29.0
- 1.28.1
- 1.27.1
- 1.26.7

+ Successfully added account my-k8s-account for provider
kubernetes.

Choose Environment

选择如何安装 Spinnaker,此处选择 Distributed installation on Kubernetes [4]

在 Halyard 容器中,执行以下命令,配置 Distributed installation on Kubernetes,其中 --account-name 的值为 添加 kubernetes 账号 中指定的名称

$ hal config deploy edit --type distributed --account-name my-k8s-account

+ Get current deployment
Success
+ Get the deployment environment
Success
+ Edit the deployment environment
Success
Validation in default:
- WARNING You have not yet selected a version of Spinnaker to
deploy.
? Options include:
- 1.29.0
- 1.28.1
- 1.27.1
- 1.26.7

+ Successfully updated your deployment environment.

External Storage

外部存储用来持久化 Spinnaker 的配置数据 [5]

在 Halyard 容器中,执行以下命令,配置 AWS S3 连接信息

$ hal config storage s3 edit \
--access-key-id $YOUR_ACCESS_KEY_ID \
--secret-access-key \
--region $REGION \
--bucket $BUCKET



Your AWS Secret Key.:
+ Get current deployment
Success
+ Get persistent store
Success
+ Edit persistent store
Success
Validation in default.persistentStorage:
- WARNING Your deployment will most likely fail until you configure
and enable a persistent store.

Validation in default:
- WARNING You have not yet selected a version of Spinnaker to
deploy.
? Options include:
- 1.29.0
- 1.28.1
- 1.27.1
- 1.26.7

+ Successfully edited persistent store "s3".

在 Halyard 容器中,执行以下命令,配置存储源为 AWS S3

$ hal config storage edit --type s3
+ Get current deployment
Success
+ Get persistent storage settings
Success
+ Edit persistent storage settings
Success
Validation in default:
- WARNING You have not yet selected a version of Spinnaker to
deploy.
? Options include:
- 1.29.0
- 1.28.1
- 1.27.1
- 1.26.7

+ Successfully edited persistent storage.

Deploy Spinnaker

部署 Spinnaker,并连接 Deck [6]

Deploy Spinnaker

# 查看可用的版本
$ hal version list

# 指定要部署的版本
$ hal config version edit --version 1.29.0

# 执行部署
$ hal deploy apply

+ Get current deployment
Success
+ Prep deployment
Success
Validation in default.stats:
- INFO Stats are currently ENABLED. Usage statistics are being
collected. Thank you! These stats inform improvements to the product, and that
helps the community. To disable, run `hal config stats disable`. To learn more
about what and how stats data is used, please see
https://spinnaker.io/docs/community/stay-informed/stats.

Validation in default.security:
- WARNING Your UI or API domain does not have override base URLs
set even though your Spinnaker deployment is a Distributed deployment on a
remote cloud provider. As a result, you will need to open SSH tunnels against
that deployment to access Spinnaker.
? We recommend that you instead configure an authentication
mechanism (OAuth2, SAML2, or x509) to make it easier to access Spinnaker
securely, and then register the intended Domain and IP addresses that your
publicly facing services will be using.

+ Preparation complete... deploying Spinnaker
+ Get current deployment
Success
- Apply deployment
_ Apply deployment
^ Apply deployment
* Apply deployment
_ Apply deployment
+ Apply deployment
Success
+ Deploy spin-redis
Success
+ Deploy spin-clouddriver
Success
+ Deploy spin-front50
Success
+ Deploy spin-orca
Success
+ Deploy spin-deck
Success
+ Deploy spin-echo
Success
+ Deploy spin-gate
Success
+ Deploy spin-rosco
Success
+ Run `hal deploy connect` to connect to Spinnaker.

使用 kubernetes 做为 Cloud Provider,选择分布式部署后,会在 kubernetes 集群中创建命名空间:spinnaker,并创建相关的 DeploymentService 等资源

$ kubectl get deployment,services,pods -n spinnaker
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/spin-clouddriver 1/1 1 1 33m
deployment.apps/spin-deck 1/1 1 1 33m
deployment.apps/spin-echo 1/1 1 1 33m
deployment.apps/spin-front50 1/1 1 1 33m
deployment.apps/spin-gate 1/1 1 1 33m
deployment.apps/spin-orca 1/1 1 1 33m
deployment.apps/spin-redis 1/1 1 1 33m
deployment.apps/spin-rosco 1/1 1 1 33m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/spin-clouddriver ClusterIP 10.103.217.2 <none> 7002/TCP 33m
service/spin-deck ClusterIP 10.100.138.37 <none> 9000/TCP 33m
service/spin-echo ClusterIP 10.109.141.10 <none> 8089/TCP 33m
service/spin-front50 ClusterIP 10.97.63.108 <none> 8080/TCP 33m
service/spin-gate ClusterIP 10.107.160.202 <none> 8084/TCP 33m
service/spin-orca ClusterIP 10.101.53.110 <none> 8083/TCP 33m
service/spin-redis ClusterIP 10.97.51.27 <none> 6379/TCP 33m
service/spin-rosco ClusterIP 10.96.45.179 <none> 8087/TCP 33m

NAME READY STATUS RESTARTS AGE
pod/spin-clouddriver-7b8d55598c-cnzbz 1/1 Running 0 33m
pod/spin-deck-64bb7f4b7-mpvpr 1/1 Running 0 33m
pod/spin-echo-645cdd8749-l2krj 1/1 Running 0 33m
pod/spin-front50-5d84b48775-7j462 1/1 Running 0 33m
pod/spin-gate-64589485b-cxpkh 1/1 Running 0 33m
pod/spin-orca-7f665ccdcb-4cnvg 1/1 Running 0 33m
pod/spin-redis-55b7dd654c-q9d8k 1/1 Running 0 33m
pod/spin-rosco-6458964f46-gsdqc 1/1 Running 0 33m

Connect to the Spinnaker UI

使用 kubernetes 做为 Cloud Provider,无需执行以下命令

hal deploy connect

为了连接 UI,需要在 kubernetes 集群中为 spin-deck 创建相应的 Ingress 资源

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: spinnaker-deck
namespace: spinnaker
spec:
ingressClassName: nginx
rules:
- host: deck.my.com
http:
paths:
- backend:
service:
name: spin-deck
port:
number: 9000
path: /
pathType: Prefix

通过 Ingress 绑定的域名,可以访问到 Spinnaker UI,此时 Web UI 中会无法获取到资源,因为默认的 Gate Endpoint 为 http://localhost:8084,浏览器页面会报错 : Error fetching applications. Check that your gate endpoint is accessible. Further information on troubleshooting this error is available here。因此需要执行以下命令配置 UI 和 Gate Api 的域名 [7]

hal config security ui edit --override-base-url http://deck.my.com
hal config security api edit --override-base-url http://spin-gate.my.com

执行以下命令,使配置生效

hal deploy apply

在 kubernetes 集群中为 spin-gate 创建相应的 Ingress 资源

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: spinnaker-gate
namespace: spinnaker
spec:
ingressClassName: nginx
rules:
- host: spin-gate.my.com
http:
paths:
- backend:
service:
name: spin-gate
port:
number: 8084
path: /
pathType: Prefix

重新访问 Web UI,可以看到正确的内容

Spinnaker 其他配置

添加自建的私有 docker 镜像源

启用 docker-registry

$ hal config provider docker-registry enable
+ Get current deployment
Success
+ Edit the dockerRegistry provider
Success
Validation in default.provider.dockerRegistry:
- WARNING Provider dockerRegistry is enabled, but no accounts have
been configured.

+ Successfully enabled dockerRegistry

配置认证信息

$ hal config provider docker-registry account add my-docker-registry \
--address http://54.236.1.2:5000 \
--username $USER \
--password $PASSWD

+ Get current deployment
Success
+ Add the my-docker-registry account
Success
+ Successfully added account my-docker-registry for provider
dockerRegistry.

配置 Jenkins 信息

启用 Jenkins

hal config ci jenkins enable

配置 Jenkins 认证信息

hal config ci jenkins master add my-jenkins-master-01 \ 
--address http://34.150.227.175:8080 \
--username admin \
--password 32366490694c4af4

启用 csrf

hal config ci jenkins master edit my-jenkins-master-01 --csrf true

配置 LDAP 认证

先配置 ldap 服务相关信息,再启用 ldap 认证,否则会报错: ! ERROR LDAP url missing.

hal config security authn ldap edit \
--user-search-base 'dc=example,dc=org' \
--url 'ldap://10.4.7.12:389' \
--user-search-filter 'cn={0}' \
--manager-dn 'cn=admin,dc=example,dc=org' \
--manager-password 'admin'

启用 ldap 认证

hal config security authn ldap enable

重新部署,以使配置生效

hal deploy apply

使用 Nginx 反代 Spinnaker

Spinnaker 部署于 Kubernetes 集群中,为了让集群中的服务不直接接收外部流量,需要使用 Nginx 反代访问 Spinnaker 服务。本示例中 Spinnaker 为配置 https,为 http 服务。使用的 Nginx 配置如下:

upstream ingress-nginx-controller-https {
server 172.31.22.9:443;
server 172.31.16.4:443;
}

upstream ingress-nginx-controller-http {
server 172.31.22.9:80;
server 172.31.16.4:80;
}

# 用来对 用户 ip 进行映射,以便后面做访问控制
map $clientRealIp $k8s_admin_access {
default false;
include k8s_admin_access.list;
}


server {
listen 80;
server_name deck.test.com spin-gate.test.com;
access_log /home/logs/nginx/access/spinnaker.access.log main;
error_log /home/logs/nginx/error/spinnaker.error.log;

location / {
if ($k8s_admin_access = false) {
return 403;
}

proxy_pass http://ingress-nginx-controller-http; # 此处需要使用 http,不能使用 https
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header REMOTE-HOST $remote_addr;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}


常见错误

Spinnaker 无法自动读取到 Kubernetes 中的命名空间

登陆 Spinnaker 后,查看 Applications 中的内容,无任何 Applications 显示,手动添加应用后,pipeline 部署失败,返回以下错误

Exception ( Deploy Manifest )
deployKubernetesManifest.deployKubernetesManifest.deployment.Error reading kind [deployment]. Please check connectivity and access permissions to the cluster

检查 spin-clouddriver 的日志,日志中显示无法连接到 kube-api-server ,解决 spin-clouddriverkube-api-server 的通信异常,问题解决。

参考链接

Install and Configure Spinnaker

Halyard FAQ

脚注