Docker Storage Options and why ??

Did you ever come across an error like below ever?

/usr/bin/docker-current: Error response from daemon: devmapper: Thin Pool has 1026 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.

Then you are looking at the right place.  Few things that I have to point out before telling you how to resolve this and how to go with your implementation to begin with:

1. If you are seeing this on your local, don’t stress, re-install docker and everything should be good. (Remember: This will remove all existing images / containers / volumes).  Basically it’s a clear slate for you to do a fresh start 🙂
2. If you are seeing this in your dev environment then I guess you can just follow 1 and no one cares
3. But if you see this any where else like, your test / integration / staging / production environments then be sure that you took a considerably big risk and didn’t follow docker documentation to the dot.

But it is always better to understand this as a design.

Docker needs some storage to store all that you are creating with-in it’s realm.  Like when you create a container it will use some storage from this space and create a container for you… or when you pull an image… it should be able to store this somewhere, correct?  These are called storage drivers.

Now, there are different storage drivers for different version on different operating systems.  Believe it or not, every thing is very very well documented.  You can find all that information for Docker CE, here and for Docker EE, here.  And the underlying file system if required for each of these storage drivers, here.

Out of all these, overlay2 and devicemapper will suite most of the use-cases we generally find.  Above error is when you are using devicemapper.  Most commonly with loopback-device.  There are ways to get out of issues, but I was not able to recover lost space even after cleaning up the whole device.

Loopback-device will use your existing filesystem, ext3 or ext4 and devicemapper being efficient on block storage will have obvious performance hits.  To avoid this completely start using direct-lvm mode where you mount a 200G (or some according to your load) volume on your Server and use according to the load and if you ever run out of space then add another volume and extend your logical volume or lvm.

So let’s take an example, I have a CentOS server on my Oracle VirtualBox.  Now I create a new Volume on OVB and attach it to my CentOS server:

Before:

After:

Now you see a 10G sdb in there.  Create a Physical Volume

pvcreate /dev/sdb

Create a Volume Group called docker

vgcreate docker /dev/sdb

Now create a Logical Volume (lvm) [instead of thinpool you can give any name of your choice, like thinp / tpool / thpool]

lvcreate --wipesignatures y -n thinpool docker -l 50%VG

This will create a logical volume thinpool with 50% of space from your VG which is 10G.  So it will allocate 5G for this thinpool volume.  Now you will see a link (docker-thinpool) in your /dev/mapper which will be linking to a block under /dev.

Now create some space for metadata

lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG

This will create another directory docker-thinpoolmeta as well in your /dev/mapper.  Convert these to a thinpool using lvconvert

lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta

Congratulations, you now have your thinpool with configured with direct-lvm mode.  Just one small thing that is still left to do is to make sure your thinpool can scale automatically.  Like if we fill it up by 80% (by pulling images, creating containers etc) then instead of giving us an error it will extends from 50% which 5 GB by 20% and will become 6 GB.  Put below configuration in /etc/lvm/profile/docker-thinpool.profile [VGName-ThinpoolName.profile].

activation {
  thin_pool_autoextend_threshold=80
  thin_pool_autoextend_percent=20
}

Now update your LV with this profile with lvchange command

lvchange --metadataprofile docker-thinpool docker/thinpool

Now add a monitor that will trigger auto extension of your thinkpool.

lvs -o+seg_monitor

Now create a file daemon.json in /etc/docker and below content

{
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
    ]
}

That’s it just start docker service and check docker info.  You have now devicemapper storage driver with direct-lvm mode setup.

Now when your thinpool fills up by 70% you can setup an alert and you can either cleanup by removing unwanted artifacts that are still lingering around or add another block to the VG Group and your auto-extender will keeping extending the thinpool.

Cleaning up Unwanted artifacts on docker:

$ docker ps -aq -f status=exited | xargs docker rm
$ docker ps -aq -f status=created | xargs docker rm
$ docker images -q -f dangling=true | xargs docker rmi
$ docker volume ls -q -f dangling=true | xargs docker volume rm

Extending a Thinpool Volume

It is just like extending any other logical volume:

1. pvcreate a create a new physical volume on your new /dev/sdd
2. Add this newly created volume to VG by vgextend docker /dev/sdd

That’s it your auto extend should be able to extend now.

Please note that you need to have permissions to execute all the commands that were given in this post.

 

Calico and Network Policies

Calico is CNI plugin on Kubernetes enable networking and network policy enforcement.  More details about calico can be found @ docs.projectcalico.org. Calico version used here in this demo is 2.6.2.  Please refer to calico documentation if you are trying out with a different version.

Make sure we have Kubernetes cluster with Calico CNI installed.  If not please refer to my earlier post on how to do that using Oracle VirtualBox.

Now Create two services, nginx and apache and see how we can control the traffic to these two services and also to the DNS service (kube-dns) here.

nginx.yaml used here is as below:

apiVersion: v1
kind: Namespace
metadata:
  name: www

---
apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  namespace: www
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
apiVersion: v1
metadata:
  namespace: www
  name: nginx
spec:
  selector:
    app: nginx
  clusterIP: 10.96.0.80
  ports:
  - protocol: TCP
    name: nginx-80
    port: 80
    targetPort: 80

apache2.yaml used here is as below:

apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  namespace: www
  name: apache2-deployment
spec:
  selector:
    matchLabels:
      app: apache2
  replicas: 2 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: apache2
    spec:
      containers:
      - name: apache2
        image: httpd
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
apiVersion: v1
metadata:
  namespace: www
  name: apache2
spec:
  selector:
    app: apache2
  clusterIP: 10.96.0.81
  ports:
  - protocol: TCP
    name: apache2
    port: 80
    targetPort: 80

Now we will have three busybox pods one in each namespace {kube-system, www, default}:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: www
spec:
  containers:
  - name: busybox
    image: busybox
    command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]

---
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox-default
    image: busybox
    command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: kube-system
spec:
  containers:
  - name: busybox
    image: busybox
    command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]

Apart from setting up your services and busybox pods lets also install calicoctl as a pod so that we can apply our network policies.  Below is the yaml that I used to create this pod:

apiVersion: v1
kind: Pod
metadata:
  name: calicoctl
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: calicoctl
    image: quay.io/calico/ctl:v1.6.1
    command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
    env:
    - name: ETCD_ENDPOINTS
      valueFrom:
        configMapKeyRef:
          name: calico-config
          key: etcd_endpoints

Now after applying all three yaml files above out cluster would look something like this:

NAMESPACE   NAME             STATUS    AGE
            ns/default       Active    2h
            ns/kube-public   Active    2h
            ns/kube-system   Active    2h
            ns/www           Active    2h

NAMESPACE     NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE       SELECTOR
default       svc/kubernetes    ClusterIP   10.96.0.1          <none>     443/TCP         2h        <none>
kube-system   svc/calico-etcd   ClusterIP   10.96.232.136      <none>     6666/TCP        2h        k8s-app=calico-etcd
kube-system   svc/kube-dns      ClusterIP   10.96.0.10         <none>     53/UDP,53/TCP   2h        k8s-app=kube-dns
www           svc/apache2       ClusterIP   10.96.0.81         <none>     80/TCP          2h        app=apache2
www           svc/nginx         ClusterIP   10.96.0.80         <none>     80/TCP          2h        app=nginx

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE       IP                NODE
default       po/busybox                                    1/1       Running   0          1h        192.168.104.22    node2
kube-system   po/busybox                                    1/1       Running   0          1h        192.168.166.157   node1
kube-system   po/calico-etcd-mcrrx                          1/1       Running   0          2h        10.0.2.4          master
kube-system   po/calico-kube-controllers-55449f8d88-w6w26   1/1       Running   0          2h        10.0.2.4          master
kube-system   po/calico-node-689qv                          2/2       Running   0          2h        10.0.2.4          master
kube-system   po/calico-node-csgdh                          2/2       Running   1          2h        172.16.0.4        node1
kube-system   po/calico-node-nc5xf                          2/2       Running   1          2h        172.16.0.5        node2
kube-system   po/calicoctl                                  1/1       Running   0          2h        172.16.0.5        node2
kube-system   po/etcd-master                                1/1       Running   0          2h        10.0.2.4          master
kube-system   po/kube-apiserver-master                      1/1       Running   0          2h        10.0.2.4          master
kube-system   po/kube-controller-manager-master             1/1       Running   0          2h        10.0.2.4          master
kube-system   po/kube-dns-545bc4bfd4-pb8vs                  3/3       Running   7          2h        192.168.166.154   node1
kube-system   po/kube-proxy-4ggld                           1/1       Running   0          2h        172.16.0.4        node1
kube-system   po/kube-proxy-4l84w                           1/1       Running   0          2h        10.0.2.4          master
kube-system   po/kube-proxy-7kqf9                           1/1       Running   0          2h        172.16.0.5        node2
kube-system   po/kube-scheduler-master                      1/1       Running   0          2h        10.0.2.4          master
www           po/apache2-deployment-b856fc995-h557f         1/1       Running   0          2h        192.168.104.21    node2
www           po/apache2-deployment-b856fc995-nct2c         1/1       Running   0          2h        192.168.166.156   node1
www           po/busybox                                    1/1       Running   0          2h        192.168.104.18    node2
www           po/nginx-deployment-75f4785b7-l29kt           1/1       Running   0          2h        192.168.104.20    node2
www           po/nginx-deployment-75f4785b7-qd48b           1/1       Running   0          2h        192.168.166.155   node1

Now we have

1. Three busybox pods running in each namespace
2. Two nginx and two apache2 servers running with a service each.
3. One Calicoctl pod that we will use to apply the network policies

Below are the network policies that we are tying to implement:

First is the deny-all policy for www namespace.  It looks like this:

- apiVersion: v1
  kind: policy
  metadata:
    name: www.default-deny
  spec:
    order: 1000
    selector: calico/k8s_ns == 'www'
    types:
    - ingress
    - egress

What this policy is going to do is to block all the traffic to and from the www namespace, that means there can be no communication going out of any pods or into any pods or services that belong to this namespace.

$ kubectl exec -it -n kube-system calicoctl -- /calicoctl apply -f deny-all.yaml

Now let’s test the traffic:

$ kubectl exec -it -n kube-system calicoctl -- /calicoctl apply -f deny-all.yaml
Successfully applied 1 'policy' resource(s)
rt251j@master:~/calico$ kubectl exec -it -n www busybox -- nslookup nginx.www
Server: 10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'nginx.www'
command terminated with exit code 1

Well there you go… no communication to anyone from anywhere :).   Now let’s just enable anyone from www to access the dns service for service discovery:

- apiVersion: v1
  kind: policy
  metadata:
    name: www.allow-dns-access
  spec:
    egress:
    - action: allow
      destination:
        ports:
        - 53
        selector: has(calico/k8s_ns)
      protocol: tcp
      source: {}
    - action: allow
      destination:
        ports:
        - 53
        selector: has(calico/k8s_ns)
      protocol: udp
      source: {}
    order: 990
    selector: calico/k8s_ns == 'www'
    types:
    - egress

Now run the same test once again:

$ kubectl exec -it -n kube-system calicoctl -- /calicoctl apply -f dns.yaml
Successfully applied 1 'policy' resource(s)
$ kubectl exec -it -n kube-system busybox -- nslookup nginx.www
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: nginx.www
Address 1: 10.96.0.80 nginx.www.svc.cluster.local
$ kubectl exec -it busybox -- nslookup nginx.www
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: nginx.www
Address 1: 10.96.0.80 nginx.www.svc.cluster.local
$ kubectl exec -it -n www busybox -- nslookup nginx.www
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: nginx.www
Address 1: 10.96.0.80 nginx.www.svc.cluster.local
$ kubectl exec -it -n www busybox -- wget -qO - -T 5 http://nginx.www
wget: download timed out
command terminated with exit code 1

Now we can see that dns queries are being resolved.  But still the nginx service is not giving out any output.  You can run the same test with apache2 service as well.  Now let’s open the gates to nginx service itself.  Below is the yaml that we are going to use:

- apiVersion: v1
 kind: policy
 metadata:
 name: www.nginx-egress
 spec:
 egress:
 - action: allow
 destination:
 selector: calico/k8s_ns == 'www' && app == 'nginx'
 source: {}
 order: 500
 selector: calico/k8s_ns == 'www'
 types:
 - egress
---
- apiVersion: v1
 kind: policy
 metadata:
 name: www.nginx-ingress
 spec:
 ingress:
 - action: allow
 destination: {}
 source:
 selector: calico/k8s_ns in {'www'}
 order: 500
 selector: calico/k8s_ns == 'www' && app == 'nginx'
 types:
 - ingress

Now let’s verify

$ kubectl exec -it -n kube-system calicoctl -- /calicoctl apply -f nginx.yaml
Successfully applied 2 'policy' resource(s)
rt251j@master:~/calico$ kubectl exec -it -n www busybox -- wget -qO - -T 5 http://nginx.www
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body {
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$ kubectl exec -it -n kube-system busybox -- wget -qO - -T 5 http://nginx.www
wget: download timed out
command terminated with exit code 1
$ kubectl exec -it busybox -- wget -qO - -T 5 http://nginx.www
wget: download timed out
command terminated with exit code 1

Now nginx service is visble to everyother component in www namespace, like busybox and apache pods in www can reach out to nginx service but no one from outside www, interesting?  Let’s make this more interesting, let’s open up nginx service to default namespace and not the kube-system namespace.

Just add ‘default’ to ingress policy source selector.  That’s it.  Apply the yaml file after the change and test it.

Now if you test apache servie then it will still be inaccessible from anywhere in our cluster.  All this while we didnot talk about the importance of order value in the network policy yaml files.  Just change the order in deny-all.yaml to 100, apply it and see how all the previous tests we have done perform.

Please find all the network policy yamls and the service yamls attached here.

Setting up Kubernetes Cluster with Oracle VirtualBox

Assumptions

Networking

SERVICE_CLUSTER_IP_RANGE = 10.96.0.0/16
MASTER_IP=172.168.0.3
POD_CIDR = 192.168.0.0/16

If you happend to be behind proxy, then please follow the insturctions given @ Docker Docs to setup your docker so that it can run in a proxy environment.

Setting up VMs in Oracle Virtual Box

Setup a Host Only Network

Create VMs

Create Three VMs:

1. Master
   1.2. CPU: 2
   1.3. Memory: 2GB
   1.4. Networks Adapaters: 2
   1.4.1 NAT / NAT Network
   1.4.2 Host Only Adapter with above Host only network
2. Node1
   2.1 CPU: 2
   2.2 Memory: 4GB
   2.3 Networks Adapaters: 2
   2.3.1 NAT / NAT Network
   2.3.2 Host Only Adapter with above Host only network
3. Node2
   3.1 CPU: 2
   3.2 Memory: 4GB
   3.3 Networks Adapaters: 2
   3.3.1 NAT / NAT Network
   3.3.2 Host Only Adapter with above Host only network

OS: Ubuntu 16.04 Server

Disable Swap

sudo swapoff -a

Now comment out the lines related to swap space in “/etc/fstab”.  Do this on all three nodes.

Install Pre-requisites

This step is to setup all thre pre-requisites for setting up kubernetes cluster using Kubeadm. In this you will install Docker, Kubelet, Kubeadm and Kubectl on your master and nodes.

Execute:

sudo apt-get install -y docker.io
sudo apt-get update
sudo apt-get install -y
sudo install -y apt-transport-https
sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
sudo cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

Setting Up Kubenetes Cluster with Kubeadm

On Master

Start creating your cluster on master node by executing the below command

$ sudo -E kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=172.16.0.3 --service-cidr=10.96.0.0/16
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.8.1
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kube-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.0.3]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] This often takes around a minute; or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 29.509714 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node kube-master as master by adding a label and a taint
[markmaster] Master kube-master tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 620033.73faad95c5f4a5f4
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node as root:

  kubeadm join --token 620033.73faad95c5f4a5f4 172.16.0.3:6443 --discovery-token-ca-cert-hash sha256:0ee1125e3783df4923021b4c7eeb4a2bb0038bb2620906e35bd2a64f00e00609


rtalari@kube-master:~$ sudo kubectl get nodes;
NAME          STATUS    ROLES     AGE       VERSION
kube-master   Ready     master    1m        v1.8.0

Node 1

Adding Node1 to your cluster

$ sudo -E kubeadm join --token 620033.73faad95c5f4a5f4 172.16.0.3:6443 --discovery-token-ca-cert-hash sha256:0ee1125e3783df4923021b4c7eeb4a2bb0038bb2620906e35bd2a64f00e00609
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "172.16.0.3:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.16.0.3:6443"
[discovery] Requesting info from "https://172.16.0.3:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.16.0.3:6443"
[discovery] Successfully established connection with API Server "172.16.0.3:6443"
[bootstrap] Detected server version: v1.8.1
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

Node 2

Add node2 to your cluster

$ kubeadm join --token 620033.73faad95c5f4a5f4 172.16.0.3:6443 --discovery-token-ca-cert-hash sha256:0ee1125e3783df4923021b4c7eeb4a2bb0038bb2620906e35bd2a64f00e00609
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "172.16.0.3:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.16.0.3:6443"
[discovery] Requesting info from "https://172.16.0.3:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.16.0.3:6443"
[discovery] Successfully established connection with API Server "172.16.0.3:6443"
[bootstrap] Detected server version: v1.8.1
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

On Master

See what all PODs are running on your cluster

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                  READY     STATUS    RESTARTS   AGE
kube-system   etcd-kube-master                      1/1       Running   0          25s
kube-system   kube-apiserver-kube-master            1/1       Running   0          29s
kube-system   kube-controller-manager-kube-master   1/1       Running   0          35s
kube-system   kube-dns-545bc4bfd4-f8z6w             3/3       Running   0          1m
kube-system   kube-proxy-28926                      1/1       Running   0          1m
kube-system   kube-scheduler-kube-master            1/1       Running   0          41s

See all the nodes that are deployed in your cluster

$ kubectl get nodes
NAME          STATUS     ROLES     AGE       VERSION
kube-master   Ready      master     2m        v1.8.0
kube-node1    NotReady        10s       v1.8.1
kube-node2    NotReady        4s        v1.8.1

Apply the Calico Plugin or Any CNI plugin of your choice in here

$ kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
configmap "calico-config" created
daemonset "calico-etcd" created
service "calico-etcd" created
daemonset "calico-node" created
deployment "calico-kube-controllers" created
deployment "calico-policy-controller" created
clusterrolebinding "calico-cni-plugin" created
clusterrole "calico-cni-plugin" created
serviceaccount "calico-cni-plugin" created
clusterrolebinding "calico-kube-controllers" created
clusterrole "calico-kube-controllers" created
serviceaccount "calico-kube-controllers" created

To get the pods and their status running in your cluster

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS              RESTARTS   AGE
kube-system   calico-etcd-hpkzt                          1/1       Running             0          5s
kube-system   calico-kube-controllers-6ff88bf6d4-nhzxn   1/1       Running             0          4s
kube-system   calico-node-j55n7                          0/2       ContainerCreating   0          4s
kube-system   calico-node-m6nlg                          1/2       Running             0          4s
kube-system   etcd-kube-master                           1/1       Running             0          1m
kube-system   kube-apiserver-kube-master                 1/1       Running             0          1m
kube-system   kube-controller-manager-kube-master        1/1       Running             0          1m
kube-system   kube-dns-545bc4bfd4-ccdxf                  3/3       Running             0          2m
kube-system   kube-proxy-gncxl                           1/1       Running             0          1m
kube-system   kube-proxy-pjxxp                           1/1       Running             0          2m
kube-system   kube-proxy-qm726                           1/1       Running             0          1m
kube-system   kube-scheduler-kube-master		 1/1       Running             0          1m

Note that some of them are still in ContainerCreating status.  Wait for some time and check if all pods or up and running.  For the first time it takes a while, so be patient nothings wrong with your cluster 🙂

$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE       IP           NODE
kube-system   calico-etcd-hpkzt                          1/1       Running   0          14m       10.0.3.7     kube-master
kube-system   calico-kube-controllers-6ff88bf6d4-nhzxn   1/1       Running   0          14m       10.0.3.7     kube-master
kube-system   calico-node-j55n7                          2/2       Running   0          14m       10.0.3.8     kube-node1
kube-system   calico-node-m6nlg                          2/2       Running   0          14m       10.0.3.9     kube-node2
kube-system   etcd-kube-master                           1/1       Running   0          15m       10.0.3.7     kube-master
kube-system   kube-apiserver-kube-master                 1/1       Running   0          15m       10.0.3.7     kube-master
kube-system   kube-controller-manager-kube-master        1/1       Running   0          15m       10.0.3.7     kube-master
kube-system   kube-dns-545bc4bfd4-ccdxf                  3/3       Running   0          16m       172.17.0.2   kube-master
kube-system   kube-proxy-gncxl                           1/1       Running   0          15m       10.0.3.9     kube-node2
kube-system   kube-proxy-pjxxp                           1/1       Running   0          16m       10.0.3.7     kube-master
kube-system   kube-proxy-qm726                           1/1       Running   0          15m       10.0.3.8     kube-node1
kube-system   kube-scheduler-kube-master                 1/1       Running   0          15m       10.0.3.7     kube-master

Below will give you the list of Services that are running in your kuebernetes cluster:

$ kubectl get services --all-namespaces
NAMESPACE     NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes    ClusterIP   10.96.0.1               443/TCP         17h
kube-system   calico-etcd   ClusterIP   10.96.232.136           6666/TCP        11h
kube-system   kube-dns      ClusterIP   10.96.0.10              53/UDP,53/TCP   17h

Removing Taints your master (optional)

$ kubectl taint nodes --all node-role.kubernetes.io/master-
node master untainted

Now just join your master back into the cluster with (–skip-preflight-checks)

kubeadm join --token 620033.73faad95c5f4a5f4 172.16.0.3:6443 --discovery-token-ca-cert-hash sha256:0ee1125e3783df4923021b4c7eeb4a2bb0038bb2620906e35bd2a64f00e00609