Notes on Kubernetes Cloud Provider

Posted on Edited on

Terminology

cloudprovider.Interface: https://github.com/kubernetes/cloud-provider/blob/master/cloud.go#L43

type loadbalancer interface: https://github.com/kubernetes/cloud-provider/blob/master/cloud.go#L106

  1. GetLoadbalancer
  2. GetLoadBalancerName
  3. EnsureLoadBalancer
  4. UpdateLoadBalancer
  5. EnsureLoadBalancerDeleted

Components

Cloud controller manager

k8s.io/kubernetes/cmd/cloud-controller-manager
one package: app, which contains the core logic of cloud-controller-manager.
Starts:

  1. Node controller
  2. Node lifecycle controller
  3. Service controller
  4. Route controller

Ref: here

Service Controller

Service controller will be managing LoadBalancer type of service, see here for the initialization.

  1. Run() starts a group of workers here
  2. Worker() calls processNextWorkItem();
  3. processNextWorkItem() calls syncService();
  4. syncService() by default calls processServiceCreateOrUpdate();
  5. processServiceCreateOrUpdate() calls syncLoadBalancerIfNeeded() if needed;
  6. syncLoadBalancerIfNeeded() calls ensureLoadBalancer() if loadbalancer needs to be created;
  7. Calls cloud provider’s EnsureLoadBalancer through cloudprovider.Interface with a group of nodes;

Examples

vSphere out-of-tree Cloud Provider

  1. Vsphere-cloud-controller-manager is a cobra command which invokes app.Run;
  2. Run starts above mentioned four controllers and here;

As a result, vsphere-cloud-provider-manager is a concrete “implementation” of cloud-controller-manager upstream in the sense that:

  1. Leverages the same app package to Run;
  2. Implements cloudprovider.Interface;

Azure out-of-tree Cloud provider

  1. Azure cloud provider does the same thing here;

Aws out-of-tree Cloud Provider

  1. Aws cloud provider does the same thing here;

Openstack out-of-tree Cloud Provider

  1. Openstack cloud provider does the same thing here;

AliCloud Cloud Provider

AliCloud Cloud Provider is an exception, seems like it “copies” upstream cloud-controller-manager and other controllers’ code, and made its own modifications.

  1. Run calls MainLoop;
  2. MainLoop calls RunControllers;
  3. RunControllers calls runControllerPV and runControllerService, which in turn starts its own controllers;

Summary

To summarize, to implement an out-of-tree cloud provider:

  1. the entry point is app.Run;

  2. Have your own implementation of cloudprovider.Interface;

  3. Register your own cluster provider by calling cloudprovider.RegisterCloudProvider, because cloud-controller-manager calls InitCloudProvider to initialize cloud provider instance;

  4. InitCloudProvider uses configured name to get corresponding registered initialization function in an internal map, which can be updated by RegisterCloudProvider;

  5. See example here in vsphere cloud provider;

Here is an example I created: https://github.com/maplain/dummy-k8s-cloud-provider

References

Cloud Controller Manager: https://kubernetes.io/docs/concepts/architecture/cloud-controller/
Develop an out-of-tree Cloud Provider: https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/#out-of-tree
KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/0002-cloud-controller-manager.md#remove-cloud-provider-code-from-kubernetes-core

Out-of-tree cloud providers

Vsphere: https://github.com/kubernetes/cloud-provider-vsphere
Alibaba: https://github.com/kubernetes/cloud-provider-alibaba-cloud
Azure: https://github.com/kubernetes/cloud-provider-azure
Openstack: https://github.com/kubernetes/cloud-provider-openstack
Aws: https://github.com/kubernetes/cloud-provider-aws

Common Packages

Klog for logging
Gcfg for config file, which has the same syntax as git