It’s my belief that history and nature tend to repeat themselves in different scales. Therefore, it’s essential that we have a good understanding, on every level and scale, of the primitives and the way they’re constructed to form the architecture or the pattern of the structures.

This will be the focus of this blog.

Terminology

cloudprovider.Interface: https://github.com/kubernetes/cloud-provider/blob/master/cloud.go#L43

type loadbalancer interface: https://github.com/kubernetes/cloud-provider/blob/master/cloud.go#L106

1. GetLoadbalancer
2. GetLoadBalancerName
3. EnsureLoadBalancer
4. UpdateLoadBalancer
5. EnsureLoadBalancerDeleted

Components

Cloud controller manager

k8s.io/kubernetes/cmd/cloud-controller-manager
one package: app, which contains the core logic of cloud-controller-manager.
Starts:

1. Node controller
2. Node lifecycle controller
3. Service controller
4. Route controller

Ref: here

Service Controller

Service controller will be managing LoadBalancer type of service, see here for the initialization.

1. Run() starts a group of workers here
2. Worker() calls processNextWorkItem();
3. processNextWorkItem() calls syncService();
4. syncService() by default calls processServiceCreateOrUpdate();
5. processServiceCreateOrUpdate() calls syncLoadBalancerIfNeeded() if needed;
6. syncLoadBalancerIfNeeded() calls ensureLoadBalancer() if loadbalancer needs to be created;
7. Calls cloud provider’s EnsureLoadBalancer through cloudprovider.Interface with a group of nodes;

Examples

vSphere out-of-tree Cloud Provider

1. Vsphere-cloud-controller-manager is a cobra command which invokes app.Run;
2. Run starts above mentioned four controllers and here;

As a result, vsphere-cloud-provider-manager is a concrete “implementation” of cloud-controller-manager upstream in the sense that:

1. Leverages the same app package to Run;
2. Implements cloudprovider.Interface;

Azure out-of-tree Cloud provider

1. Azure cloud provider does the same thing here;

Aws out-of-tree Cloud Provider

1. Aws cloud provider does the same thing here;

Openstack out-of-tree Cloud Provider

1. Openstack cloud provider does the same thing here;

AliCloud Cloud Provider

AliCloud Cloud Provider is an exception, seems like it “copies” upstream cloud-controller-manager and other controllers’ code, and made its own modifications.

1. Run calls MainLoop;
2. MainLoop calls RunControllers;
3. RunControllers calls runControllerPV and runControllerService, which in turn starts its own controllers;

Summary

To summarize, to implement an out-of-tree cloud provider:

1. the entry point is app.Run;

2. Have your own implementation of cloudprovider.Interface;

3. Register your own cluster provider by calling cloudprovider.RegisterCloudProvider, because cloud-controller-manager calls InitCloudProvider to initialize cloud provider instance;

4. InitCloudProvider uses configured name to get corresponding registered initialization function in an internal map, which can be updated by RegisterCloudProvider;

5. See example here in vsphere cloud provider;

Here is an example I created: https://github.com/maplain/dummy-k8s-cloud-provider

References

Cloud Controller Manager: https://kubernetes.io/docs/concepts/architecture/cloud-controller/
Develop an out-of-tree Cloud Provider: https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/#out-of-tree
KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/0002-cloud-controller-manager.md#remove-cloud-provider-code-from-kubernetes-core

Out-of-tree cloud providers

Vsphere: https://github.com/kubernetes/cloud-provider-vsphere
Alibaba: https://github.com/kubernetes/cloud-provider-alibaba-cloud
Azure: https://github.com/kubernetes/cloud-provider-azure
Openstack: https://github.com/kubernetes/cloud-provider-openstack
Aws: https://github.com/kubernetes/cloud-provider-aws

Common Packages

Klog for logging
Gcfg for config file, which has the same syntax as git