4.1.1.1 Architecture > Overview > Infrastructure Components > Kubernetes Infrastructure

Viewing Master Service Logs

# master-logs api api # master-logs controllers controllers # master-logs etcd etcd

Restart Master Services: API

# master-restart api

Restart Master Services: controllers

# master-restart controllers

Restart Master Services: etcd

# master-restart etcd

Node Bootstrapping

A node's configuration is ______________ from the master, which means nodes pull their pre-defined configuration and client and server certificates from the master. This allows faster node start-up by reducing the differences between nodes, as well as centralizing more configuration and letting the cluster converge on the desired state. Certificate rotation and centralized certificate management are enabled by default.

Master components

API Server etcd Controller Manager Server HAProxy

Control plane components

API server, controller manager server, and etcd.

How Is OpenShift Container Platform Secured?

Authenticate users who present credentials, and then authorize them based on their role. Both developers and administrators can be authenticated via a number of means, primarily OAuth tokens and X.509 client certificates. OAuth tokens are signed with JSON Web Algorithm RS256, which is RSA signature algorithm PKCS#1 v1.5 with SHA-256.

Availability Matrix with HAProxy: HAProxy

Balances load between API master endpoints.

Hyperkube

Binary that contains all of Kubernetes (kube-apiserver, controller-manager, scheduler, proxy, and kubelet)

Controllers

Bring system into sync.

Kubernetes cluster

Consists of one or more masters and a set of nodes.

API Server (Kubernetes)

Control plane component: Validates and configures the data for pods, services, and replication controllers. It also assigns pods to nodes and synchronizes pod information with service configuration.

etcd (Kubernetes)

Control plane component: stores the persistent master state while other components watch _______ for changes to bring themselves into the desired state. _______ can be optionally configured for high availability, typically deployed with 2n+1 peer services.

Service Proxy

Each node also runs a simple network proxy that reflects the services defined in the API on that node. This allows the node to do simple TCP and UDP stream forwarding across a set of back ends.

Kubelet

Each node has a __________ that updates the node as specified by a container manifest, which is a YAML file that describes a pod. Uses a set of manifests to ensure that its containers are started and that they continue to run.

Openshift 3.11 uses which versions of Kubernetes and Docker?

Kubernetes 1.11 and Docker 1.13.1

OpenShift

Layered system designed to expose underlying Docker-formatted container image and Kubernetes concepts as accurately as possible, with a focus on easy composition of applications by a developer.

What is the Masters job?

Manage nodes in its Kubernetes cluster and schedule pods to run on those nodes.

Control Plane Static pods

Managed directly by the kubelet daemon on a specific node, without the API server observing them. Managed by the control plane, kubelet watches each and restarts it if it fails. Always bound to one Kubelet on a specific node. API server Controller manager components

Kubernetes

Manages containerized applications across a set of containers or hosts and provides mechanisms for deployment, maintenance, and application-scaling.

OpenShift Container Platform Architecture Overview

Microservices-based architecture of smaller, decoupled units that work together. Runs on top of a Kubernetes cluster Data about the objects stored in etcd, a reliable clustered key-value store. Those services are broken down by function: -REST APIs, which expose each of the core objects. -Controllers, which read those APIs, apply changes to other objects, and report status or write back to the object.

Modifying Node Configurations

Modified by editing the appropriate ConfigMap in the openshift-node project. The /etc/origin/node/node-config.yaml must not be modified directly. For example, for a node that is in the node-config-compute group, edit the ConfigMap using: $ oc edit cm node-config-compute -n openshift-node

Nodes required for master services when collocating with etcd

No quorum or requirement that they have an odd number of hosts. However, since you need at least two _______ _______ for HA, it is common to maintain a uniform odd number of hosts when collocating _______ ________and ______.

OpenShift Container Platform policy engine

OpenShift authorization is handled by which component?

OpenShift Container Platform adds these capabilities

Source code management, builds, and deployments for developers Managing and promoting images at scale as they flow through your system Application management at scale Team and user tracking for organizing a large developer organization Networking infrastructure that supports the cluster

OpenShift Container Platform creates nodes from?

cloud provider, physical systems, or virtual systems

Container runtime

Packages, instantiates, and runs containerized applications

Transport Layer Security (TLS) 1.2

Provides strong encryption, data integrity, and authentication of servers with X.509 server certificates and public key infrastructure. By default, a new internal PKI is created for each deployment of OpenShift Container Platform. The internal PKI uses 2048 bit RSA keys and SHA-256 signatures. Custom certificates for public hosts are supported as well. All communication channels with the REST API, as well as between master components such as etcd and the API server, are secured with _______?

OpenShift Layers: Docker service

Provides the abstraction for packaging and creating Linux-based, lightweight container images.

OpenShift Layers: Kubernetes

Provides the cluster management and orchestrates containers on multiple hosts.

Nodes

Provides the runtime environments for containers. Each in a Kubernetes cluster has the required services to be managed by the master. Have the required services to run pods, including the container runtime, a kubelet, and a service proxy. Managed by the CLI

Node Bootstrap Workflow

1. By default during cluster installation, a set of clusterrole, clusterrolebinding and serviceaccount objects are created for use in node bootstrapping 2. Also by default during cluster installation, the openshift-ansible installer creates a OpenShift Container Platform certificate authority and various other certificates, keys, and kubeconfig files in the /etc/origin/master directory. 3. The kubelet is first started with the supplied /etc/origin/node/bootstrap.kubeconfig file. After initial connection internally, the kubelet creates certificate signing requests (CSRs) and sends them to the master. 4. The CSRs are verified and approved via the controller manager (specifically the certificate signing controller). If approved, the kubelet client and server certificates are created in the /etc/origin/node/ceritificates directory. 5. After the CSR approval, the node.kubeconfig file is created at /etc/origin/node/node.kubeconfig. 6. The kubelet is restarted with the /etc/origin/node/node.kubeconfig file and the certificates in the /etc/origin/node/certificates/ directory, after which point it is ready to join the cluster.

Node Configuration Workflow

1. Initially the node's kubelet is started with the bootstrap configuration file, bootstrap-node-config.yaml in the /etc/origin/node/ directory, created at the time of node provisioning. 2. On each node, the node service file uses the local script openshift-node in the /usr/local/bin/ directory to start the kubelet with the supplied bootstrap-node-config.yaml. 3. On each master, the directory /etc/origin/node/pods contains pod manifests for apiserver, controller and etcd which are created as static pods on masters. 4. During cluster installation, a sync DaemonSet is created which creates a sync pod on each node. The sync pod monitors changes in the file /etc/sysconfig/atomic-openshift-node. It specifically watches for BOOTSTRAP_CONFIG_NAME to be set. BOOTSTRAP_CONFIG_NAME is set by the openshift-ansible installer and is the name of the ConfigMap based on the node configuration group the node belongs to. 5. The sync pod extracts the appropriate ConfigMap based on the value set in BOOTSTRAP_CONFIG_NAME. 6. The sync pod converts the ConfigMap data into kubelet configurations and creates a /etc/origin/node/node-config.yaml for that node host. If a change is made to this file (or it is the file's initial creation), the kubelet is restarted.

Container manifest can be provided by a kubelet

A file path on the command line that is checked every 20 seconds. An HTTP endpoint passed on the command line that is checked every 20 seconds. The kubelet watching an etcd server, such as /registry/hosts/$(hostname -f), and acting on any changes. The kubelet listening for HTTP and responding to a simple API to submit a new manifest.

HAProxy (Kubernetes)

Control plane component: optional, used when configuring highly-available masters with the native method to balance load between API master endpoints. The cluster installation process can configure _______ for you with the native method. Alternatively, you can use the native method but pre-configure your own load balancer of choice.

Controller Manager Server (Kubernetes)

Control plane component: watches etcd for changes to replication controller objects and then uses the API to enforce the desired state. Several such processes create a cluster with one active leader at a time.

Masters

Host or hosts that contain the control plane components. API server, controller manager server, and etcd

What type of token does a developer use on OpenShift?

OAuth bearer tokens Developers (clients of the system) typically make REST API calls from a client program like oc or to the web console via their browser, and use OAuth bearer tokens for most communications. Infrastructure components (like nodes) use client certificates generated by the system that contain their identities. Infrastructure components that run in containers use a token associated with their service account to connect to the API.

Number of Nodes required by etcd

While clustered _____ requires an odd number of hosts for quorum.

Availability Matrix with HAProxy: etcd

active-active Fully redundant deployment with load balancing. Can be installed on separate hosts or collocated on master hosts.

Availability Matrix with HAProxy: API Server

active-active Managed by HAProxy

Availability Matrix with HAProxy: Controller Manager Server

active-passive One instance is elected as a cluster leader at a time.

Startup Sequence Overview

kubelet creates the kubepods.slice kublet creates the QoS-level slices burstable.slice and best-effort.slice inside the kubepods.slice. When a pod starts, the kubelet creates a pod-level slice with the format pod<UUID-of-pod>.slice and passes that path to the runtime on the other side of the Container Runtime Interface (CRI). Docker or CRI-O then creates the container-level slices inside the pod-level slice.

Mirror Pods

kubelet on master nodes automatically creates _____ _____ on the API server for each of the control plane static pods so that they are visible in the cluster in the kube-system project. Static installed by default by the openshift-ansible installer

Control Plane hosts: Systemd and DaemonSets

systemd -docker.service -crio.service atomic-openshift-node.service(hyperhube) DaemonSets -openshift-node/sync -openshift-sdn/sdn -openshift-sdn/ovs DaemonSets -master-api(apiserver.yaml) -master-controllers(controllers.yaml) -etcd(etcd.yaml)

Control plan host architecture

systemd - docker.service, crio-service, atomic-openshift-node.service (hyperkube) DaemonSets - openshift-node/sync, openshift-sdn/sdn, openshift-sdn/ovs Static Pods - master-api, master-controller, etcd