Kubernetes集群调用Ingress

作者:行癫(盗版必究) ------ ## 一:Ingress简介 ​ Ingress 是从 Kubernetes 集群外部访问集群内部服务的入口 ![image-20240818004751371](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240818004751371.png) #### 1.什么是 Ingress ​ 在 Kubernetes 中,Ingress 是一种资源对象,它定义了如何从集群外部访问集群内部服务的规则;Ingress 提供了一种更高级别的抽象,允许用户管理进入集群的 HTTP 和 HTTPS 流量,而无需直接暴露每个服务 Service 主要处理集群内部的服务间通信以及如何从集群外部访问服务 Ingress 处理集群外部对集群内多个服务的高级路由规则,并且可以提供额外的网络功能 ##### Service: ​ Service 主要处理集群内部的服务间通信以及如何从集群外部访问服务 ![image-20240817223023240](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimagesimage-20240817223023240.png) 主要功能: ​ 将流量路由到一组后端 Pod ​ 提供服务发现机制 ​ 支持基于轮询或其他策略的负载均衡 使用场景: ​ 当需要在集群内部访问应用时(例如,一个前端服务调用一个后端服务) ​ 当希望在集群外部通过特定节点端口访问应用时(NodePort 类型) ​ 当需要通过云提供商的负载均衡器公开应用时(LoadBalancer 类型) ##### Ingress: ​ Ingress 控制了进入集群的 HTTP 和 HTTPS 流量,并允许将这些流量路由到不同的 Service;Ingress 通常与反向代理或负载均衡器(如 Nginx 等)一起使用,以实现更高级别的路由规则和特性 ![image-20240817223045739](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240817223045739.png) 主要功能: ​ 基于 URL 路径或主机名的路由 ​ SSL 证书管理 ​ 提供额外的特性,如重写 URL、限流等 使用场景: ​ 当需要通过单一的入口点访问多个 Service 时 ​ 当需要支持基于名称的虚拟主机(多个域名指向同一 IP) ​ 当需要高级的网络功能,如 SSL/TLS 加密、HTTP 重定向等 #### 2.Ingress controller ​ 为了使 Ingress 正常工作,集群中必须运行 Ingress controller ​ Kong Kubernetes Ingress是一个 Kubernetes Ingress 控制器,通过支持Ingress来管理对集群服务的访问 ## 二:基于Kubernetes部署Kong #### 1.集群环境 kubernetes集群正常运行 NFS提供持久化存储 DNS服务器提供域名解析 #### 2.创建命名空间kong ```shell [root@xingdiancloud-master kong]# kubectl create ns kong ``` #### 3.创建CRD的RBAC ​ CRD:CustomResourceDefinition(自定义资源定义)是 Kubernetes 用来扩展其 API 和资源模型的重要特性,允许用户定义自己的资源类型以适应特定的应用场景或需求,通过自定义资源定义,可以让 Kubernetes 管理任何类型的资源,而不仅仅是标准的容器化应用;这为 Kubernetes 带来了极大的灵活性和可扩展性 ​ RBAC:Role-Based Access Control(基于角色的访问控制)是一种访问控制机制,用于管理对资源的访问权限,在 Kubernetes 中,RBAC 是一种核心机制,用于授予用户、服务账户或其他身份验证主体对 Kubernetes API 的访问权限 ​ 官方地址:https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.9.3/deploy/single/all-in-one-dbless.yaml ```yaml [root@xingdiancloud-master kong]# cat crd.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: ingressclassparameterses.configuration.konghq.com spec: group: configuration.konghq.com names: kind: IngressClassParameters listKind: IngressClassParametersList plural: ingressclassparameterses singular: ingressclassparameters scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: IngressClassParameters is the Schema for the IngressClassParameters API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec is the IngressClassParameters specification. properties: enableLegacyRegexDetection: default: false description: EnableLegacyRegexDetection automatically detects if ImplementationSpecific Ingress paths are regular expression paths using the legacy 2.x heuristic. The controller adds the "~" prefix to those paths if the Kong version is 3.0 or higher. type: boolean serviceUpstream: default: false description: Offload load-balancing to kube-proxy or sidecar. type: boolean type: object type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: kongclusterplugins.configuration.konghq.com spec: group: configuration.konghq.com names: categories: - kong-ingress-controller kind: KongClusterPlugin listKind: KongClusterPluginList plural: kongclusterplugins shortNames: - kcp singular: kongclusterplugin scope: Cluster versions: - additionalPrinterColumns: - description: Name of the plugin jsonPath: .plugin name: Plugin-Type type: string - description: Age jsonPath: .metadata.creationTimestamp name: Age type: date - description: Indicates if the plugin is disabled jsonPath: .disabled name: Disabled priority: 1 type: boolean - description: Configuration of the plugin jsonPath: .config name: Config priority: 1 type: string name: v1 schema: openAPIV3Schema: description: KongClusterPlugin is the Schema for the kongclusterplugins API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string config: description: Config contains the plugin configuration. It's a list of keys and values required to configure the plugin. Please read the documentation of the plugin being configured to set values in here. For any plugin in Kong, anything that goes in the `config` JSON key in the Admin API request, goes into this property. Only one of `config` or `configFrom` may be used in a KongClusterPlugin, not both at once. type: object x-kubernetes-preserve-unknown-fields: true configFrom: description: ConfigFrom references a secret containing the plugin configuration. This should be used when the plugin configuration contains sensitive information, such as AWS credentials in the Lambda plugin or the client secret in the OIDC plugin. Only one of `config` or `configFrom` may be used in a KongClusterPlugin, not both at once. properties: secretKeyRef: description: Specifies a name, a namespace, and a key of a secret to refer to. properties: key: description: The key containing the value. type: string name: description: The secret containing the key. type: string namespace: description: The namespace containing the secret. type: string required: - key - name - namespace type: object type: object consumerRef: description: ConsumerRef is a reference to a particular consumer. type: string disabled: description: Disabled set if the plugin is disabled or not. type: boolean kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object ordering: description: 'Ordering overrides the normal plugin execution order. It''s only available on Kong Enterprise. `` is a request processing phase (for example, `access` or `body_filter`) and `` is the name of the plugin that will run before or after the KongPlugin. For example, a KongPlugin with `plugin: rate-limiting` and `before.access: ["key-auth"]` will create a rate limiting plugin that limits requests _before_ they are authenticated.' properties: after: additionalProperties: items: type: string type: array description: PluginOrderingPhase indicates which plugins in a phase should affect the target plugin's order type: object before: additionalProperties: items: type: string type: array description: PluginOrderingPhase indicates which plugins in a phase should affect the target plugin's order type: object type: object plugin: description: PluginName is the name of the plugin to which to apply the config. type: string protocols: description: Protocols configures plugin to run on requests received on specific protocols. items: description: KongProtocol is a valid Kong protocol. This alias is necessary to deal with https://github.com/kubernetes-sigs/controller-tools/issues/342 enum: - http - https - grpc - grpcs - tcp - tls - udp type: string type: array run_on: description: RunOn configures the plugin to run on the first or the second or both nodes in case of a service mesh deployment. enum: - first - second - all type: string required: - plugin type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: kongconsumers.configuration.konghq.com spec: group: configuration.konghq.com names: categories: - kong-ingress-controller kind: KongConsumer listKind: KongConsumerList plural: kongconsumers shortNames: - kc singular: kongconsumer scope: Namespaced versions: - additionalPrinterColumns: - description: Username of a Kong Consumer jsonPath: .username name: Username type: string - description: Age jsonPath: .metadata.creationTimestamp name: Age type: date name: v1 schema: openAPIV3Schema: description: KongConsumer is the Schema for the kongconsumers API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string credentials: description: Credentials are references to secrets containing a credential to be provisioned in Kong. items: type: string type: array custom_id: description: CustomID is a Kong cluster-unique existing ID for the consumer - useful for mapping Kong with users in your existing database. type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object username: description: Username is a Kong cluster-unique username of the consumer. type: string type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: kongingresses.configuration.konghq.com spec: group: configuration.konghq.com names: categories: - kong-ingress-controller kind: KongIngress listKind: KongIngressList plural: kongingresses shortNames: - ki singular: kongingress scope: Namespaced versions: - name: v1 schema: openAPIV3Schema: description: KongIngress is the Schema for the kongingresses API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object proxy: description: Proxy defines additional connection options for the routes to be configured in the Kong Gateway, e.g. `connection_timeout`, `retries`, etc. properties: connect_timeout: description: "The timeout in milliseconds for\testablishing a connection to the upstream server. Deprecated: use Service's \"konghq.com/connect-timeout\" annotation instead." minimum: 0 type: integer path: description: '(optional) The path to be used in requests to the upstream server. Deprecated: use Service''s "konghq.com/path" annotation instead.' pattern: ^/.*$ type: string protocol: description: 'The protocol used to communicate with the upstream. Deprecated: use Service''s "konghq.com/protocol" annotation instead.' enum: - http - https - grpc - grpcs - tcp - tls - udp type: string read_timeout: description: 'The timeout in milliseconds between two successive read operations for transmitting a request to the upstream server. Deprecated: use Service''s "konghq.com/read-timeout" annotation instead.' minimum: 0 type: integer retries: description: 'The number of retries to execute upon failure to proxy. Deprecated: use Service''s "konghq.com/retries" annotation instead.' minimum: 0 type: integer write_timeout: description: 'The timeout in milliseconds between two successive write operations for transmitting a request to the upstream server. Deprecated: use Service''s "konghq.com/write-timeout" annotation instead.' minimum: 0 type: integer type: object route: description: Route define rules to match client requests. Each Route is associated with a Service, and a Service may have multiple Routes associated to it. properties: headers: additionalProperties: items: type: string type: array description: 'Headers contains one or more lists of values indexed by header name that will cause this Route to match if present in the request. The Host header cannot be used with this attribute. Deprecated: use Ingress'' "konghq.com/headers" annotation instead.' type: object https_redirect_status_code: description: 'HTTPSRedirectStatusCode is the status code Kong responds with when all properties of a Route match except the protocol. Deprecated: use Ingress'' "ingress.kubernetes.io/force-ssl-redirect" or "konghq.com/https-redirect-status-code" annotations instead.' type: integer methods: description: 'Methods is a list of HTTP methods that match this Route. Deprecated: use Ingress'' "konghq.com/methods" annotation instead.' items: type: string type: array path_handling: description: 'PathHandling controls how the Service path, Route path and requested path are combined when sending a request to the upstream. Deprecated: use Ingress'' "konghq.com/path-handling" annotation instead.' enum: - v0 - v1 type: string preserve_host: description: 'PreserveHost sets When matching a Route via one of the hosts domain names, use the request Host header in the upstream request headers. If set to false, the upstream Host header will be that of the Service’s host. Deprecated: use Ingress'' "konghq.com/preserve-host" annotation instead.' type: boolean protocols: description: 'Protocols is an array of the protocols this Route should allow. Deprecated: use Ingress'' "konghq.com/protocols" annotation instead.' items: description: KongProtocol is a valid Kong protocol. This alias is necessary to deal with https://github.com/kubernetes-sigs/controller-tools/issues/342 enum: - http - https - grpc - grpcs - tcp - tls - udp type: string type: array regex_priority: description: 'RegexPriority is a number used to choose which route resolves a given request when several routes match it using regexes simultaneously. Deprecated: use Ingress'' "konghq.com/regex-priority" annotation instead.' type: integer request_buffering: description: 'RequestBuffering sets whether to enable request body buffering or not. Deprecated: use Ingress'' "konghq.com/request-buffering" annotation instead.' type: boolean response_buffering: description: 'ResponseBuffering sets whether to enable response body buffering or not. Deprecated: use Ingress'' "konghq.com/response-buffering" annotation instead.' type: boolean snis: description: 'SNIs is a list of SNIs that match this Route when using stream routing. Deprecated: use Ingress'' "konghq.com/snis" annotation instead.' items: type: string type: array strip_path: description: 'StripPath sets When matching a Route via one of the paths strip the matching prefix from the upstream request URL. Deprecated: use Ingress'' "konghq.com/strip-path" annotation instead.' type: boolean type: object upstream: description: Upstream represents a virtual hostname and can be used to loadbalance incoming requests over multiple targets (e.g. Kubernetes `Services` can be a target, OR `Endpoints` can be targets). properties: algorithm: description: Algorithm is the load balancing algorithm to use. enum: - round-robin - consistent-hashing - least-connections type: string hash_fallback: description: 'HashFallback defines What to use as hashing input if the primary hash_on does not return a hash. Accepted values are: "none", "consumer", "ip", "header", "cookie".' type: string hash_fallback_header: description: HashFallbackHeader is the header name to take the value from as hash input. Only required when "hash_fallback" is set to "header". type: string hash_fallback_query_arg: description: HashFallbackQueryArg is the "hash_fallback" version of HashOnQueryArg. type: string hash_fallback_uri_capture: description: HashFallbackURICapture is the "hash_fallback" version of HashOnURICapture. type: string hash_on: description: 'HashOn defines what to use as hashing input. Accepted values are: "none", "consumer", "ip", "header", "cookie", "path", "query_arg", "uri_capture".' type: string hash_on_cookie: description: The cookie name to take the value from as hash input. Only required when "hash_on" or "hash_fallback" is set to "cookie". type: string hash_on_cookie_path: description: The cookie path to set in the response headers. Only required when "hash_on" or "hash_fallback" is set to "cookie". type: string hash_on_header: description: HashOnHeader defines the header name to take the value from as hash input. Only required when "hash_on" is set to "header". type: string hash_on_query_arg: description: HashOnQueryArg is the query string parameter whose value is the hash input when "hash_on" is set to "query_arg". type: string hash_on_uri_capture: description: HashOnURICapture is the name of the capture group whose value is the hash input when "hash_on" is set to "uri_capture". type: string healthchecks: description: Healthchecks defines the health check configurations in Kong. properties: active: description: ActiveHealthcheck configures active health check probing. properties: concurrency: minimum: 1 type: integer healthy: description: Healthy configures thresholds and HTTP status codes to mark targets healthy for an upstream. properties: http_statuses: items: type: integer type: array interval: minimum: 0 type: integer successes: minimum: 0 type: integer type: object http_path: pattern: ^/.*$ type: string https_sni: type: string https_verify_certificate: type: boolean timeout: minimum: 0 type: integer type: type: string unhealthy: description: Unhealthy configures thresholds and HTTP status codes to mark targets unhealthy. properties: http_failures: minimum: 0 type: integer http_statuses: items: type: integer type: array interval: minimum: 0 type: integer tcp_failures: minimum: 0 type: integer timeouts: minimum: 0 type: integer type: object type: object passive: description: PassiveHealthcheck configures passive checks around passive health checks. properties: healthy: description: Healthy configures thresholds and HTTP status codes to mark targets healthy for an upstream. properties: http_statuses: items: type: integer type: array interval: minimum: 0 type: integer successes: minimum: 0 type: integer type: object type: type: string unhealthy: description: Unhealthy configures thresholds and HTTP status codes to mark targets unhealthy. properties: http_failures: minimum: 0 type: integer http_statuses: items: type: integer type: array interval: minimum: 0 type: integer tcp_failures: minimum: 0 type: integer timeouts: minimum: 0 type: integer type: object type: object threshold: type: number type: object host_header: description: HostHeader is The hostname to be used as Host header when proxying requests through Kong. type: string slots: description: Slots is the number of slots in the load balancer algorithm. minimum: 10 type: integer type: object type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: kongplugins.configuration.konghq.com spec: group: configuration.konghq.com names: categories: - kong-ingress-controller kind: KongPlugin listKind: KongPluginList plural: kongplugins shortNames: - kp singular: kongplugin scope: Namespaced versions: - additionalPrinterColumns: - description: Name of the plugin jsonPath: .plugin name: Plugin-Type type: string - description: Age jsonPath: .metadata.creationTimestamp name: Age type: date - description: Indicates if the plugin is disabled jsonPath: .disabled name: Disabled priority: 1 type: boolean - description: Configuration of the plugin jsonPath: .config name: Config priority: 1 type: string name: v1 schema: openAPIV3Schema: description: KongPlugin is the Schema for the kongplugins API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string config: description: Config contains the plugin configuration. It's a list of keys and values required to configure the plugin. Please read the documentation of the plugin being configured to set values in here. For any plugin in Kong, anything that goes in the `config` JSON key in the Admin API request, goes into this property. Only one of `config` or `configFrom` may be used in a KongPlugin, not both at once. type: object x-kubernetes-preserve-unknown-fields: true configFrom: description: ConfigFrom references a secret containing the plugin configuration. This should be used when the plugin configuration contains sensitive information, such as AWS credentials in the Lambda plugin or the client secret in the OIDC plugin. Only one of `config` or `configFrom` may be used in a KongPlugin, not both at once. properties: secretKeyRef: description: Specifies a name and a key of a secret to refer to. The namespace is implicitly set to the one of referring object. properties: key: description: The key containing the value. type: string name: description: The secret containing the key. type: string required: - key - name type: object type: object consumerRef: description: ConsumerRef is a reference to a particular consumer. type: string disabled: description: Disabled set if the plugin is disabled or not. type: boolean kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object ordering: description: 'Ordering overrides the normal plugin execution order. It''s only available on Kong Enterprise. `` is a request processing phase (for example, `access` or `body_filter`) and `` is the name of the plugin that will run before or after the KongPlugin. For example, a KongPlugin with `plugin: rate-limiting` and `before.access: ["key-auth"]` will create a rate limiting plugin that limits requests _before_ they are authenticated.' properties: after: additionalProperties: items: type: string type: array description: PluginOrderingPhase indicates which plugins in a phase should affect the target plugin's order type: object before: additionalProperties: items: type: string type: array description: PluginOrderingPhase indicates which plugins in a phase should affect the target plugin's order type: object type: object plugin: description: PluginName is the name of the plugin to which to apply the config. type: string protocols: description: Protocols configures plugin to run on requests received on specific protocols. items: description: KongProtocol is a valid Kong protocol. This alias is necessary to deal with https://github.com/kubernetes-sigs/controller-tools/issues/342 enum: - http - https - grpc - grpcs - tcp - tls - udp type: string type: array run_on: description: RunOn configures the plugin to run on the first or the second or both nodes in case of a service mesh deployment. enum: - first - second - all type: string required: - plugin type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: tcpingresses.configuration.konghq.com spec: group: configuration.konghq.com names: categories: - kong-ingress-controller kind: TCPIngress listKind: TCPIngressList plural: tcpingresses singular: tcpingress scope: Namespaced versions: - additionalPrinterColumns: - description: Address of the load balancer jsonPath: .status.loadBalancer.ingress[*].ip name: Address type: string - description: Age jsonPath: .metadata.creationTimestamp name: Age type: date name: v1beta1 schema: openAPIV3Schema: description: TCPIngress is the Schema for the tcpingresses API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec is the TCPIngress specification. properties: rules: description: A list of rules used to configure the Ingress. items: description: IngressRule represents a rule to apply against incoming requests. Matching is performed based on an (optional) SNI and port. properties: backend: description: Backend defines the referenced service endpoint to which the traffic will be forwarded to. properties: serviceName: description: Specifies the name of the referenced service. minLength: 1 type: string servicePort: description: Specifies the port of the referenced service. format: int32 maximum: 65535 minimum: 1 type: integer required: - serviceName - servicePort type: object host: description: Host is the fully qualified domain name of a network host, as defined by RFC 3986. If a Host is not specified, then port-based TCP routing is performed. Kong doesn't care about the content of the TCP stream in this case. If a Host is specified, the protocol must be TLS over TCP. A plain-text TCP request cannot be routed based on Host. It can only be routed based on Port. type: string port: description: Port is the port on which to accept TCP or TLS over TCP sessions and route. It is a required field. If a Host is not specified, the requested are routed based only on Port. format: int32 maximum: 65535 minimum: 1 type: integer required: - backend - port type: object type: array tls: description: TLS configuration. This is similar to the `tls` section in the Ingress resource in networking.v1beta1 group. The mapping of SNIs to TLS cert-key pair defined here will be used for HTTP Ingress rules as well. Once can define the mapping in this resource or the original Ingress resource, both have the same effect. items: description: IngressTLS describes the transport layer security. properties: hosts: description: Hosts are a list of hosts included in the TLS certificate. The values in this list must match the name/s used in the tlsSecret. Defaults to the wildcard host setting for the loadbalancer controller fulfilling this Ingress, if left unspecified. items: type: string type: array secretName: description: SecretName is the name of the secret used to terminate SSL traffic. type: string type: object type: array type: object status: description: TCPIngressStatus defines the observed state of TCPIngress. properties: loadBalancer: description: LoadBalancer contains the current status of the load-balancer. properties: ingress: description: Ingress is a list containing ingress points for the load-balancer. Traffic intended for the service should be sent to these ingress points. items: description: 'LoadBalancerIngress represents the status of a load-balancer ingress point: traffic intended for the service should be sent to an ingress point.' properties: hostname: description: Hostname is set for load-balancer ingress points that are DNS based (typically AWS load-balancers) type: string ip: description: IP is set for load-balancer ingress points that are IP based (typically GCE or OpenStack load-balancers) type: string ports: description: Ports is a list of records of service ports If used, every port defined in the service should have an entry in it items: properties: error: description: 'Error is to record the problem with the service port The format of the error shall comply with the following rules: - built-in error values shall be specified in this file and those shall use CamelCase names - cloud provider specific error values must have names that comply with the format foo.example.com/CamelCase. --- The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)' maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string port: description: Port is the port number of the service port of which status is recorded here format: int32 type: integer protocol: default: TCP description: 'Protocol is the protocol of the service port of which status is recorded here The supported values are: "TCP", "UDP", "SCTP"' type: string required: - port - protocol type: object type: array x-kubernetes-list-type: atomic type: object type: array type: object type: object type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: udpingresses.configuration.konghq.com spec: group: configuration.konghq.com names: categories: - kong-ingress-controller kind: UDPIngress listKind: UDPIngressList plural: udpingresses singular: udpingress scope: Namespaced versions: - additionalPrinterColumns: - description: Address of the load balancer jsonPath: .status.loadBalancer.ingress[*].ip name: Address type: string - description: Age jsonPath: .metadata.creationTimestamp name: Age type: date name: v1beta1 schema: openAPIV3Schema: description: UDPIngress is the Schema for the udpingresses API. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec is the UDPIngress specification. properties: rules: description: A list of rules used to configure the Ingress. items: description: UDPIngressRule represents a rule to apply against incoming requests wherein no Host matching is available for request routing, only the port is used to match requests. properties: backend: description: Backend defines the Kubernetes service which accepts traffic from the listening Port defined above. properties: serviceName: description: Specifies the name of the referenced service. minLength: 1 type: string servicePort: description: Specifies the port of the referenced service. format: int32 maximum: 65535 minimum: 1 type: integer required: - serviceName - servicePort type: object port: description: Port indicates the port for the Kong proxy to accept incoming traffic on, which will then be routed to the service Backend. format: int32 maximum: 65535 minimum: 1 type: integer required: - backend - port type: object type: array type: object status: description: UDPIngressStatus defines the observed state of UDPIngress. properties: loadBalancer: description: LoadBalancer contains the current status of the load-balancer. properties: ingress: description: Ingress is a list containing ingress points for the load-balancer. Traffic intended for the service should be sent to these ingress points. items: description: 'LoadBalancerIngress represents the status of a load-balancer ingress point: traffic intended for the service should be sent to an ingress point.' properties: hostname: description: Hostname is set for load-balancer ingress points that are DNS based (typically AWS load-balancers) type: string ip: description: IP is set for load-balancer ingress points that are IP based (typically GCE or OpenStack load-balancers) type: string ports: description: Ports is a list of records of service ports If used, every port defined in the service should have an entry in it items: properties: error: description: 'Error is to record the problem with the service port The format of the error shall comply with the following rules: - built-in error values shall be specified in this file and those shall use CamelCase names - cloud provider specific error values must have names that comply with the format foo.example.com/CamelCase. --- The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)' maxLength: 316 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ type: string port: description: Port is the port number of the service port of which status is recorded here format: int32 type: integer protocol: default: TCP description: 'Protocol is the protocol of the service port of which status is recorded here The supported values are: "TCP", "UDP", "SCTP"' type: string required: - port - protocol type: object type: array x-kubernetes-list-type: atomic type: object type: array type: object type: object type: object served: true storage: true subresources: status: {} --- apiVersion: v1 kind: ServiceAccount metadata: name: kong-serviceaccount namespace: kong --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: kong-leader-election namespace: kong rules: - apiGroups: - "" - coordination.k8s.io resources: - configmaps - leases verbs: - get - list - watch - create - update - patch - delete - apiGroups: - "" resources: - events verbs: - create - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null name: kong-ingress rules: - apiGroups: - "" resources: - endpoints verbs: - list - watch - apiGroups: - "" resources: - endpoints/status verbs: - get - patch - update - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - pods verbs: - get - list - watch - apiGroups: - "" resources: - secrets verbs: - list - watch - apiGroups: - "" resources: - secrets/status verbs: - get - patch - update - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - services/status verbs: - get - patch - update - apiGroups: - configuration.konghq.com resources: - ingressclassparameterses verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - kongclusterplugins verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - kongclusterplugins/status verbs: - get - patch - update - apiGroups: - configuration.konghq.com resources: - kongconsumers verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - kongconsumers/status verbs: - get - patch - update - apiGroups: - configuration.konghq.com resources: - kongingresses verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - kongingresses/status verbs: - get - patch - update - apiGroups: - configuration.konghq.com resources: - kongplugins verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - kongplugins/status verbs: - get - patch - update - apiGroups: - configuration.konghq.com resources: - tcpingresses verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - tcpingresses/status verbs: - get - patch - update - apiGroups: - configuration.konghq.com resources: - udpingresses verbs: - get - list - watch - apiGroups: - configuration.konghq.com resources: - udpingresses/status verbs: - get - patch - update - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - get - patch - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - get - patch - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null name: kong-ingress-gateway rules: - apiGroups: - gateway.networking.k8s.io resources: - gatewayclasses verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - gatewayclasses/status verbs: - get - update - apiGroups: - gateway.networking.k8s.io resources: - gateways verbs: - get - list - update - watch - apiGroups: - gateway.networking.k8s.io resources: - gateways/status verbs: - get - update - apiGroups: - gateway.networking.k8s.io resources: - grpcroutes verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - grpcroutes/status verbs: - get - patch - update - apiGroups: - gateway.networking.k8s.io resources: - httproutes verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - httproutes/status verbs: - get - update - apiGroups: - gateway.networking.k8s.io resources: - referencegrants verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - referencegrants/status verbs: - get - apiGroups: - gateway.networking.k8s.io resources: - tcproutes verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - tcproutes/status verbs: - get - update - apiGroups: - gateway.networking.k8s.io resources: - tlsroutes verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - tlsroutes/status verbs: - get - update - apiGroups: - gateway.networking.k8s.io resources: - udproutes verbs: - get - list - watch - apiGroups: - gateway.networking.k8s.io resources: - udproutes/status verbs: - get - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null name: kong-ingress-knative rules: - apiGroups: - networking.internal.knative.dev resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.internal.knative.dev resources: - ingresses/status verbs: - get - patch - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kong-leader-election namespace: kong roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kong-leader-election subjects: - kind: ServiceAccount name: kong-serviceaccount namespace: kong --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kong-ingress roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kong-ingress subjects: - kind: ServiceAccount name: kong-serviceaccount namespace: kong --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kong-ingress-gateway roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kong-ingress-gateway subjects: - kind: ServiceAccount name: kong-serviceaccount namespace: kong --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kong-ingress-knative roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kong-ingress-knative subjects: - kind: ServiceAccount name: kong-serviceaccount namespace: kong [root@xingdiancloud-master kong]# kubectl apply -f crd.yaml ``` #### 4.部署数据库PostgreSql 创建持久卷PV 提前在NFS服务器上创建共享目录 ```yaml [root@xingdiancloud-master kong]# cat postgres-pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: postgrespv01 labels: name: postgrespv01 function: postgres spec: nfs: path: /data/xingdiancloud/master/postgresql/ server: 10.9.12.250 accessModes: ["ReadWriteMany","ReadWriteOnce"] capacity: storage: 10Gi [root@xingdiancloud-master kong]# kubectl apply -f postgres-pv.yaml ``` 创建对应的StatefulSet控制器运行PostgreSql 创建对应的SVC ```yaml [root@xingdiancloud-master kong]# cat postgres-sts.yaml --- apiVersion: v1 kind: Service metadata: name: postgres namespace: kong spec: ports: - name: pgql port: 5432 protocol: TCP targetPort: 5432 selector: app: postgres --- apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres namespace: kong spec: replicas: 1 selector: matchLabels: app: postgres serviceName: postgres template: metadata: labels: app: postgres spec: containers: - env: - name: POSTGRES_USER value: kong - name: POSTGRES_PASSWORD value: kong - name: POSTGRES_DB value: kong - name: PGDATA value: /var/lib/postgresql/data/pgdata image: 10.9.12.201/kong/postgres:9.5 name: postgres ports: - containerPort: 5432 volumeMounts: - mountPath: /var/lib/postgresql/data name: postgres-pvc subPath: pgdata terminationGracePeriodSeconds: 60 volumeClaimTemplates: - metadata: name: postgres-pvc spec: selector: matchLabels: function: postgres accessModes: ["ReadWriteOnce"] resources: requests: storage: 9Gi [root@xingdiancloud-master kong]# kubectl apply -f postgres-sts.yaml ``` 数据导入 ```yaml [root@xingdiancloud-master kong]# cat kong-postgresql.yaml --- apiVersion: batch/v1 kind: Job metadata: name: kong-migrations namespace: kong spec: template: metadata: name: kong-migrations spec: containers: - command: - /bin/sh - -c - kong migrations bootstrap env: - name: KONG_PG_PASSWORD value: kong - name: KONG_PG_HOST value: postgres - name: KONG_PG_PORT value: "5432" image: 10.9.12.201/kong/kong:3.2 name: kong-migrations initContainers: - command: - /bin/sh - -c - until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo 'waiting for db'; sleep 1; done env: - name: KONG_PG_HOST value: postgres - name: KONG_PG_PORT value: "5432" image: 10.9.12.201/xingdian/busybox name: wait-for-postgres restartPolicy: OnFailure ``` #### 5.创建配置ConfigMap ```yaml [root@xingdiancloud-master kong]# cat configmap.yaml --- kind: ConfigMap apiVersion: v1 metadata: name: kong-server-blocks namespace: kong data: servers.conf: | # Prometheus metrics server server { server_name kong_prometheus_exporter; listen 0.0.0.0:9542; # can be any other port as well access_log off; location /metrics { default_type text/plain; content_by_lua_block { local prometheus = require "kong.plugins.prometheus.exporter" prometheus:collect() } } location /nginx_status { internal; stub_status; } } # Health check server server { server_name kong_health_check; listen 0.0.0.0:9001; # can be any other port as well access_log off; location /health { return 200; } } [root@xingdiancloud-master kong]# kubectl apply -f configmap.yaml ``` #### 6.部署Kong Ingress 创建SVC 使用Deployment创建kong ingress 创建IngressClass ```yaml [root@xingdiancloud-master kong]# cat kong-ingress.yaml apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp service.beta.kubernetes.io/aws-load-balancer-type: nlb name: kong-proxy namespace: kong spec: ports: - name: proxy port: 80 protocol: TCP targetPort: 8000 - name: proxy-ssl port: 443 protocol: TCP targetPort: 8443 - name: kong-admin port: 8001 protocol: TCP targetPort: 8001 - name: kong-admin-ssl port: 8444 protocol: TCP targetPort: 8444 selector: app: ingress-kong type: NodePort --- apiVersion: v1 kind: Service metadata: name: kong-validation-webhook namespace: kong spec: ports: - name: webhook port: 443 protocol: TCP targetPort: 8080 selector: app: ingress-kong --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: ingress-kong name: ingress-kong namespace: kong spec: replicas: 3 selector: matchLabels: app: ingress-kong template: metadata: annotations: kuma.io/gateway: enabled prometheus.io/port: "8100" prometheus.io/scrape: "true" traffic.sidecar.istio.io/includeInboundPorts: "" labels: app: ingress-kong spec: containers: - env: - name: KONG_PROXY_LISTEN value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2 - name: KONG_ADMIN_LISTEN value: 0.0.0.0:8001, 0.0.0.0:8444 ssl - name: KONG_STATUS_LISTEN value: 0.0.0.0:8100 - name: KONG_DATABASE value: postgres - name: KONG_PG_HOST value: postgres - name: KONG_PG_PASSWORD value: kong - name: KONG_NGINX_WORKER_PROCESSES value: "1" - name: KONG_ADMIN_ACCESS_LOG value: /dev/stdout - name: KONG_ADMIN_ERROR_LOG value: /dev/stderr - name: KONG_PROXY_ERROR_LOG value: /dev/stderr image: 10.9.12.201/kong/kong:3.2 lifecycle: preStop: exec: command: - /bin/sh - -c - kong quit livenessProbe: failureThreshold: 3 httpGet: path: /status port: 8100 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: proxy ports: - containerPort: 8000 name: proxy protocol: TCP - containerPort: 8443 name: proxy-ssl protocol: TCP - containerPort: 8100 name: metrics protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /status port: 8100 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 securityContext: runAsUser: 1000 - env: - name: CONTROLLER_KONG_ADMIN_URL value: https://127.0.0.1:8444 - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY value: "true" - name: CONTROLLER_PUBLISH_SERVICE value: kong/kong-proxy - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace image: 10.9.12.201/kong/kubernetes-ingress-controller:2.9.3 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: ingress-controller ports: - containerPort: 8080 name: webhook protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 serviceAccountName: kong-serviceaccount --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: kong spec: controller: ingress-controllers.konghq.com/kong [root@xingdiancloud-master kong]# kubectl apply -f kong-ingress.yaml ``` #### 7.部署Konga 数据导入 ```yaml [root@xingdiancloud-master kong]# cat magrations.yaml --- apiVersion: batch/v1 kind: Job metadata: name: konga-migrations namespace: kong spec: template: metadata: name: konga-migrations spec: imagePullSecrets: - name: harbor-secret containers: - command: - /bin/sh - -c - /app/start.sh -c prepare -a postgres -u postgresql://kong:kong@postgres:5432/konga env: - name: KONG_PG_PASSWORD value: kong - name: KONG_PG_HOST value: postgres - name: KONG_PG_PORT value: "5432" image: 10.9.12.201/kong/konga:latest name: kong-migrations initContainers: - command: - /bin/sh - -c - until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo 'waiting for db'; sleep 1; done env: - name: KONG_PG_HOST value: postgres - name: KONG_PG_PORT value: "5432" image: 10.9.12.201/xingdian/busybox name: wait-for-postgres restartPolicy: OnFailure [root@xingdiancloud-master kong]# kubectl apply -f magrations.yaml ``` 部署Konga ```yaml [root@xingdiancloud-master kong]# cat konga.yaml --- apiVersion: v1 kind: Service metadata: name: konga-proxy namespace: kong spec: type: NodePort ports: - name: konga-proxy port: 1337 targetPort: 1337 nodePort: 1337 protocol: TCP selector: app: dashboard-konga --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: dashboard-konga name: konga namespace: kong spec: replicas: 1 selector: matchLabels: app: dashboard-konga template: metadata: labels: app: dashboard-konga spec: nodeSelector: ingress: proxy containers: - env: - name: NODE_ENV value: production - name: DB_ADAPTER value: postgres - name: DB_URI value: postgresql://kong:kong@postgres:5432/konga image: 10.9.12.201/kong/konga:latest name: konga ports: - containerPort: 1337 name: konga-port protocol: TCP tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Equal" value: "" effect: "NoSchedule" [root@xingdiancloud-master kong]# kubectl apply -f konga.yaml ``` #### 8.验证 ```shell [root@xingdiancloud-master kong]# kubectl get pod -n kong ``` ![](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901212658198.png) #### 9.浏览器访问 需要先按照要求创建管理员账户 使用创建的管理员账户登录konga ![image-20240901212814805](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901212814805.png) #### 10.Konga连接kong Name:自定义 Kong Admin URL:kong-proxy 这个是 kong的svc的名字;kong 这个是svc对应的命名空间;svc固定 ![image-20240901213045693](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213045693.png) ![image-20240901213259271](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213259271.png) ![image-20240901213315103](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213315103.png) ## 三.使用kong ingress #### 1.创建upsteams 只需要起个名字,其他的默认 ![image-20240901213518443](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213518443.png) 配置 Targets Target:访问项目的地址,konga-proxy:项目对应svc的名字;kong:项目的命名空间;svc固定;1337:端口 ![image-20240901214734062](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901214734062.png) #### 2.创建Services ![image-20240901213613036](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213613036.png) Name:自定义 Protocol:http和https均可 没有证书的情况下使用http Host:关联Upstreams,写对应的upstreams的名字 Port:项目对应svc访问端口 ![image-20240901213724283](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213724283.png) #### 3.创建Route ![image-20240901213935923](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901213935923.png) Name:自定义 Host:指定对应项目访问域名 kong.xingdian.com 该域名需要跟访问IP地址做域名解析 Paths:请求路径 使用默认的 / ![image-20240901214014769](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901214014769.png) #### 4.配置域名解析 如果使用DNS服务器,将域名与IP加入配置zone即可 如果没有使用DNS服务器,在客户端访问时添加本地域名解析(实验环境) #### 5.浏览器访问 如果成功使用域名访问到,说明使用kong ingress引流成功 ![image-20240901214326146](https://xingdian-home.oss-cn-beijing.aliyuncs.com/imagesimage-20240901214326146.png) #### 6.应用场景 ​ 未来在kubernetes集群中发布的任何项目,如果使用Kong Ingress进行引流,均可采用上述流程