Tuesday, March 29, 2016

Deploy All-In-One Kubernetes and Docker from Scratch with Flannel Networking

first_kubernetes

Deploy Kubernetes and Docker from Scratch with Flannel Networking

來解釋一下如何安裝Kubernetes，並launch一個Container，最後我們將Networking改成Flannel。

套件安裝與環境設定

首先先安裝Docker等，相關的套件。

$ apt-get update
$ apt-get install ssh
$ apt-get install docker.io
$ apt-get install curl

打通ssh。

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub 127.0.0.1
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
$ ssh root@127.0.0.1
exit

下載Kubernetes v1.0.1版。

$ wget https://github.com/GoogleCloudPlatform/kubernetes/releases/download/v1.0.1/kubernetes.tar.gz
tar -xvf kubernetes.tar.gz

All-in-one 套件執行

開始進行安裝了。

cd kubernetes/cluster/ubuntu
./build.sh

設定Config檔。

cd
vi kubernetes/cluster/ubuntu/config-default.sh
export nodes="root@127.0.0.1"
export roles="ai"
export NUM_MINIONS=${NUM_MINIONS:-1}

將所有執行檔叫起來。

$ cd kubernetes/cluster
$ KUBERNETES_PROVIDER=ubuntu ./kube-up.sh

將執行檔export給系統，可再任意地方執行kubernetes的執行檔，並用kubectl測試一下。

export PATH=$PATH:~/kubernetes/cluster/ubuntu/binaries
kubectl --help

看一下，目前安裝的狀態

kubectl get nodes

kube-UI的安裝，也可以不用。

$ kubectl create -f addons/kube-ui/kube-ui-rc.yaml --namespace=kube-system
$ kubectl create -f addons/kube-ui/kube-ui-svc.yaml --namespace=kube-system

目前單機已經安裝完了，因此，可以執行以下命令將wordpress container叫起來。
如果你想換成Flannel的網路架構，可以先跳過接下來的步驟，直接去Flannel的安裝。

kubectl run wordpress --image=tutum/wordpress --port=80 --hostport=81

用kubectl查看，Container是否啟動，需要等個一分鐘左右，因為要從dockerhub下載檔案。

kubectl get pods
kubectl get rc

也可以用docker指令查看。

docker ps

看一下網路架構，與Docker本身的網路架構是一樣的。

#ifconfig
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1072 (1.0 KB)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:ad:f5:61
          inet addr:172.16.235.128  Bcast:172.16.235.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fead:f561/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:300420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:127648 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:437238373 (437.2 MB)  TX bytes:8566239 (8.5 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:522030 errors:0 dropped:0 overruns:0 frame:0
          TX packets:522030 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:207725326 (207.7 MB)  TX bytes:207725326 (207.7 MB)

veth0ef8f9d Link encap:Ethernet  HWaddr 26:f0:39:f9:65:3a
          inet6 addr: fe80::24f0:39ff:fef9:653a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1944 (1.9 KB)

vetha2a43fc Link encap:Ethernet  HWaddr d2:ad:ea:e1:ff:7f
          inet6 addr: fe80::d0ad:eaff:fee1:ff7f/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

連到Container看一下網路IP。

root@wordpress-j0f8t:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:02
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ETCD在Kubernetes有著非常重要的位置，看一下etcdctl內容。

root@kubernetes:~# /opt/bin/etcdctl ls
/registry
root@kubernetes:~# ps aux|grep kube

root       4763  0.0  0.1   8392  1972 ?        Ssl  13:20   0:00 /kube-ui
root      10628  3.2  3.6 248140 36140 ?        Ssl  13:35   0:03 /opt/bin/kubelet --address=0.0.0.0 --port=10250 --hostname_override=127.0.0.1 --api_servers=http://127.0.0.1:8080 --logtostderr=true --cluster_dns=192.168.3.10 --cluster_domain=cluster.local
root      10629  0.4  1.2  16276 12500 ?        Ssl  13:35   0:00 /opt/bin/kube-scheduler --logtostderr=true --master=127.0.0.1:8080
root      10631  1.2  3.9  48896 39936 ?        Ssl  13:35   0:01 /opt/bin/kube-apiserver --address=0.0.0.0 --port=8080 --etcd_servers=http://127.0.0.1:4001 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ServiceAccount,ResourceQuota --service_account_key_file=/tmp/kube-serviceaccount.key --service_account_lookup=false
root      10632  0.6  1.9  25556 19248 ?        Ssl  13:35   0:00 /opt/bin/kube-controller-manager --master=127.0.0.1:8080 --service_account_private_key_file=/tmp/kube-serviceaccount.key --logtostderr=true
root      10633  0.2  1.4 203044 14244 ?        Ssl  13:35   0:00 /opt/bin/kube-proxy --master=http://127.0.0.1:8080 --logtostderr=true

Flannel的安裝

Let's change it to Flannel network

vim /etc/default/flanneld
FLANNEL_OPTS="--etcd-endpoints=http://127.0.0.1:2379"
service flanneld stop 
service docker stop
service kubelet stop
service kube-proxy stop
apt-get install -y bridge-utils
ip link set dev docker0 down
brctl delbr docker0
/opt/bin/etcdctl rm --recursive /coreos.com/network
/opt/bin/etcdctl set /coreos.com/network/config '{ "Network": "172.17.0.0/16" }'
service flanneld start

You will see /run/flannel/subnet.env

ls /run/flannel/subnet.env

and modify /etc/default/docker

vim /etc/default/docker
. /run/flannel/subnet.env 
DOCKER_OPTS="--bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}"

where the /run/flannel/subnet.env file shown as

vim /run/flannel/subnet.env
FLANNEL_SUBNET=172.17.16.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=false
service docker start
service kubelet start
service kube-proxy start
kubectl run wordpress --image=tutum/wordpress --port=80 --hostport=81
kubectl get pods
kubectl get rc
docker ps -a

After 1 min, you will see the result that the VM's been launched.

root@kubernetes:~/kubernetes/cluster# kubectl get pods
NAME              READY     REASON    RESTARTS   AGE
wordpress-gp4cl   1/1       Running   0          7m

You wil see the Flannel Network here. 

root@kubernetes:~/kubernetes/cluster# ifconfig
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.16.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1472  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:ad:f5:61
          inet addr:172.16.235.128  Bcast:172.16.235.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fead:f561/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:297104 errors:0 dropped:0 overruns:0 frame:0
          TX packets:118915 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:435040989 (435.0 MB)  TX bytes:7818950 (7.8 MB)

flannel0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:172.17.16.0  P-t-P:172.17.16.0  Mask:255.255.0.0
          UP POINTOPOINT RUNNING  MTU:1472  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:87012 errors:0 dropped:0 overruns:0 frame:0
          TX packets:87012 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:184668449 (184.6 MB)  TX bytes:184668449 (184.6 MB)

vethbd09c72 Link encap:Ethernet  HWaddr 96:0a:11:72:07:77
          inet6 addr: fe80::940a:11ff:fe72:777/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1296 (1.2 KB)

From Official Site, you wiil see the result is right.

However, we only have one node here.

Also see the container's IP, and login to the container.

root@wordpress-gp4cl:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:10:02
          inet addr:172.17.16.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:acff:fe11:1002/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1296 (1.2 KB)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

網路問題

在Container內，ping www.google.com，我發現不通，查了一下container內的nameserver

vim /etc/resolv.conf
nameserver 192.168.3.10
nameserver 172.16.235.2
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

delete 192.168.3.10 nameserver

vim /etc/resolv.conf
nameserver 172.16.235.2
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

再ping一下通了

oot@wordpress-gp4cl:/# ping www.google.com
PING www.google.com (64.233.188.103) 56(84) bytes of data.
64 bytes from tk-in-f103.1e100.net (64.233.188.103): icmp_seq=1 ttl=127 time=10.2 ms
64 bytes from tk-in-f103.1e100.net (64.233.188.103): icmp_seq=2 ttl=127 time=9.81 ms

發現了deploy config有問題，下次可以修改一下。

root@kubernetes:~/kubernetes/cluster# grep -R  192.168.3 *
ubuntu/config-default.sh:export SERVICE_CLUSTER_IP_RANGE=${SERVICE_CLUSTER_IP_RANGE:-192.168.3.0/24}  # formerly PORTAL_NET
ubuntu/config-default.sh:DNS_SERVER_IP=${DNS_SERVER_IP:-"192.168.3.10"}

改成8.8.8.8試試看，這部分是手動在container中改的，下次部署可以改上述的檔案ubuntu/config-default.sh中DNSSERVERIP.

nameserver 8.8.8.8
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

ping google again, it works well.

root@wordpress-gp4cl:/# ping www.google.com
PING www.google.com (74.125.203.106) 56(84) bytes of data.
64 bytes from th-in-f106.1e100.net (74.125.203.106): icmp_seq=1 ttl=127 time=11.4 ms
64 bytes from th-in-f106.1e100.net (74.125.203.106): icmp_seq=2 ttl=127 time=12.5 ms
64 bytes from th-in-f106.1e100.net (74.125.203.106): icmp_seq=3 ttl=127 time=15.0 ms

To check the detailed infomation via curl. 

root@kubernetes:/# curl -s http://localhost:4001/v2/keys/coreos.com/network/subnets | python -mjson.tool
{
    "action": "get",
    "node": {
        "createdIndex": 37,
        "dir": true,
        "key": "/coreos.com/network/subnets",
        "modifiedIndex": 37,
        "nodes": [
            {
                "createdIndex": 37,
                "expiration": "2016-03-30T07:37:28.947871304Z",
                "key": "/coreos.com/network/subnets/172.17.16.0-24",
                "modifiedIndex": 37,
                "ttl": 83666,
                "value": "{\"PublicIP\":\"172.16.235.128\"}"
            }
        ]
    }
}

下一篇我們可以來看看VXlan的模式如何開啟。

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)