Saturday, April 16, 2016

Using Devstack to install OpenStack Mitaka

mitaka

Using Devstack to install OpenStack Mitaka

不會忘記精中求細這句話,因此又回到OpenStack。

一開始,我們用Kilo的local.conf安裝Mitaka,我們發現Neutron的部分安裝失敗,因此沿用Kilo的Nuetron的設定有些問題。 會變成nova-network的安裝,因此很久不見的br100又出現了。
後來發現local.conf的 [[local|localrc]] 寫成 [local|localrc]] 大家別犯這錯誤,devstack有檢查header的。

經過一段時間努力,終於裝起來了,因此跟大家分享。

VM Information:

hostname: jonah2 
ip: 192.168.121.180
 
账号/密码 : root / xxxxx
 
里面有两张网卡 :
eth0 : 192.168.120.0/22 , GW : 192.168.120.254
eth1 : 10.0.0.0/24

install pacakge and download mitaka from git source

apt-get install git
git clone https://github.com/openstack-dev/devstack.git -b stable/mitaka

測試一下repository是不是指向mitaka。

stack@jonah1:~/devstack$ grep -R mitaka *
stackrc:CINDER_BRANCH=${CINDER_BRANCH:-stable/mitaka}
stackrc:GLANCE_BRANCH=${GLANCE_BRANCH:-stable/mitaka}
stackrc:HEAT_BRANCH=${HEAT_BRANCH:-stable/mitaka}
stackrc:HORIZON_BRANCH=${HORIZON_BRANCH:-stable/mitaka}
stackrc:KEYSTONE_BRANCH=${KEYSTONE_BRANCH:-stable/mitaka}
stackrc:NEUTRON_BRANCH=${NEUTRON_BRANCH:-stable/mitaka}
stackrc:NEUTRON_FWAAS_BRANCH=${NEUTRON_FWAAS_BRANCH:-stable/mitaka}
stackrc:NEUTRON_LBAAS_BRANCH=${NEUTRON_LBAAS_BRANCH:-stable/mitaka}
stackrc:NEUTRON_VPNAAS_BRANCH=${NEUTRON_VPNAAS_BRANCH:-stable/mitaka}
stackrc:NOVA_BRANCH=${NOVA_BRANCH:-stable/mitaka}
stackrc:SWIFT_BRANCH=${SWIFT_BRANCH:-stable/mitaka}
stackrc:REQUIREMENTS_BRANCH=${REQUIREMENTS_BRANCH:-stable/mitaka}
tests/test_refs.sh:REFS=`grep BRANCH stackrc | grep -v -- '-master' | grep -v -- '-stable/mitaka'`

start to install mitaka

cd devstack/tools

產生stack user,devstack需要用stack user。經過第二次測試,以下安裝方法是成立的,包含產生stack user到執行./stack.sh。

root@jonah1:~/devstack/tools# ./create-stack-user.sh
Creating a group called stack
Creating a user called stack
Giving stack user passwordless sudo privileges

root user

cd ../..
cp devstack /opt/stack/. -rf
chown -R stack:stack /opt/stack/

stack user

su stack
cd
cd devstack

edit local.conf

[[local|localrc]]
DEST=/opt/stack

ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken


ENABLED_SERVICES=key,rabbit,mysql,horizon
ENABLED_SERVICES+=,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,g-api,g-reg

# Enabling Neutron (network) Service
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-metering
enable_service neutron

Q_PLUGIN=ml2
#Q_USE_DEBUG_COMMAND=True
if [ "$Q_PLUGIN" = "ml2" ]; then
  #Q_ML2_TENANT_NETWORK_TYPE=gre
  Q_ML2_TENANT_NETWORK_TYPE=vxlan
  :
fi
## Neutron options
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
NETWORK_GATEWAY=10.0.0.1
PRIVATE_SUBNET_NAME=privateA

PUBLIC_SUBNET_NAME=public-subnet
FLOATING_RANGE=192.168.120.0/24
PUBLIC_NETWORK_GATEWAY=192.168.120.254
##Q_FLOATING_ALLOCATION_POOL=start=192.168.27.102,end=192.168.27.110
PUBLIC_INTERFACE=eth0
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex

LIBVIRT_TYPE=qemu

IMAGE_URLS="http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-uec.tar.gz"

SCREEN_LOGDIR=/opt/stack/screen-logs
SYSLOG=True
LOGFILE=~/devstack/stack.sh.log


Q_USE_DEBUG_COMMAND=True

# RECLONE=No
RECLONE=yes
OFFLINE=False

why the local.conf like that ?

where PUBLICNETWORKGATEWAY=192.168.120.254 obtained by

stack@jonah2:~/devstack$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.120.254 0.0.0.0         UG    0      0        0 br-ex
10.0.0.0        192.168.120.1   255.255.255.0   UG    0      0        0 br-ex
192.168.120.0   *               255.255.252.0   U     0      0        0 br-ex

to check ip and netmask 192.168.121.180/22 obtained by

stack@jonah2:~/devstack$ ip a
. 
.
.

6: br-ex: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 00:50:56:85:05:dd brd ff:ff:ff:ff:ff:ff
    inet 192.168.121.180/22 brd 192.168.123.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 2001:db8::2/64 scope global tentative dadfailed
       valid_lft forever preferred_lft forever
    inet6 fe80::e04f:28ff:fe31:c282/64 scope link
       valid_lft forever preferred_lft forever
7: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 2e:08:1a:43:bd:4c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7063:6eff:fe42:5fe8/64 scope link
       valid_lft forever preferred_lft forever
       

run devstack

./stack.sh

result

經過22分鐘左右,我們得到了結果。 透過devstack返回的資訊,現在可以連到dashbaoard了。

This is your host IP address: 192.168.121.180
This is your host IPv6 address: ::1
Horizon is now available at http://192.168.121.180/dashboard
Keystone is serving at http://192.168.121.180:5000/
The default users are: admin and demo
The password: password

看一下網路狀況

stack@jonah2:~/devstack$ ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:50:56:85:05:dd
          inet addr:192.168.121.180  Bcast:192.168.123.255  Mask:255.255.252.0
          inet6 addr: fe80::e04f:28ff:fe31:c282/64 Scope:Link
          inet6 addr: 2001:db8::2/64 Scope:Global
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:7491 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5702 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1064105 (1.0 MB)  TX bytes:6426734 (6.4 MB)

br-int    Link encap:Ethernet  HWaddr 1e:e9:1f:ec:8c:4c
          inet6 addr: fe80::2cbb:c7ff:feab:f0b0/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:833 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:375685 (375.6 KB)  TX bytes:828 (828.0 B)

br-tun    Link encap:Ethernet  HWaddr 2e:08:1a:43:bd:4c
          inet6 addr: fe80::7063:6eff:fe42:5fe8/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:828 (828.0 B)

eth0      Link encap:Ethernet  HWaddr 00:50:56:85:05:dd
          inet6 addr: fe80::250:56ff:fe85:5dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:104433 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20999 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:28740156 (28.7 MB)  TX bytes:9126144 (9.1 MB)

eth1      Link encap:Ethernet  HWaddr 00:50:56:85:49:9d
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe85:499d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:22 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2847 (2.8 KB)  TX bytes:908 (908.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:151886 errors:0 dropped:0 overruns:0 frame:0
          TX packets:151886 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:107699755 (107.6 MB)  TX bytes:107699755 (107.6 MB)

看一下process是否正常運作

20014 stack     20   0  245564  98828  10088 S   2.3  1.2   0:36.31 neutron-server
20614 stack     20   0  204460  64040  10056 S   2.3  0.8   0:43.61 neutron-openvsw
17583 stack     20   0  186944  92504   5052 S   2.0  1.1   0:30.38 glance-api
20185 stack     20   0  250260  98636   4932 S   2.0  1.2   0:37.13 neutron-server
20187 stack     20   0  248812  97020   4868 S   2.0  1.2   0:37.87 neutron-server
20188 stack     20   0  232748  80804   4700 S   2.0  1.0   0:31.44 neutron-server
21080 stack     20   0  197632  57120  10044 S   2.0  0.7   0:32.41 neutron-dhcp-ag
21960 stack     20   0  195112  54332  10032 S   2.0  0.7   0:32.20 neutron-metadat
22466 stack     20   0  194940  54508   9968 S   2.0  0.7   0:32.17 neutron-meterin
26969 stack     20   0  255324 105492   6808 S   2.0  1.3   0:32.21 nova-conductor
20186 stack     20   0  250252  98400   4932 S   1.7  1.2   0:36.92 neutron-server
21607 stack     20   0  198104  57624  10068 S   1.7  0.7   0:33.05 neutron-l3-agen
26968 stack     20   0  256844 106692   6744 S   1.7  1.3   0:32.45 nova-conductor
29282 stack     20   0 1885800 125076  16692 S   1.7  1.5   0:38.43 nova-compute
17584 stack     20   0  187712  93168   5052 S   1.3  1.1   0:30.90 glance-api
19317 stack     20   0  294084 136332  10268 S   0.7  1.7   0:15.06 nova-api
26397 stack     20   0  240860  96428  10104 S   0.7  1.2   0:13.23 nova-conductor
14349 rabbitmq  20   0 2304244  58072   4188 S   0.3  0.7   0:14.72 beam.smp
17458 stack     20   0  180976  90196   8808 S   0.3  1.1   0:12.34 glance-api
18881 root      10 -10  243364  32180   6412 S   0.3  0.4   0:03.65 ovs-vswitchd
27549 stack     20   0  251648 105052  10216 S   0.3  1.3   0:05.42 nova-scheduler
    1 root      20   0   33636   4124   2656 S   0.0  0.1   0:01.72 init

基本上neutron有起來。

Tuesday, March 29, 2016

Deploy All-In-One Kubernetes and Docker from Scratch with Flannel Networking

first_kubernetes

Deploy Kubernetes and Docker from Scratch with Flannel Networking

來解釋一下如何安裝Kubernetes,並launch一個Container,最後我們將Networking改成Flannel。

套件安裝與環境設定

首先先安裝Docker等,相關的套件。

$ apt-get update
$ apt-get install ssh
$ apt-get install docker.io
$ apt-get install curl

打通ssh。

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub 127.0.0.1
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
$ ssh root@127.0.0.1
exit

下載Kubernetes v1.0.1版。

$ wget https://github.com/GoogleCloudPlatform/kubernetes/releases/download/v1.0.1/kubernetes.tar.gz
tar -xvf kubernetes.tar.gz

All-in-one 套件執行

開始進行安裝了。

cd kubernetes/cluster/ubuntu
./build.sh

設定Config檔。

cd
vi kubernetes/cluster/ubuntu/config-default.sh
export nodes="root@127.0.0.1"
export roles="ai"
export NUM_MINIONS=${NUM_MINIONS:-1}

將所有執行檔叫起來。

$ cd kubernetes/cluster
$ KUBERNETES_PROVIDER=ubuntu ./kube-up.sh

將執行檔export給系統,可再任意地方執行kubernetes的執行檔,並用kubectl測試一下。

export PATH=$PATH:~/kubernetes/cluster/ubuntu/binaries
kubectl --help

看一下,目前安裝的狀態

kubectl get nodes

kube-UI的安裝,也可以不用。

$ kubectl create -f addons/kube-ui/kube-ui-rc.yaml --namespace=kube-system
$ kubectl create -f addons/kube-ui/kube-ui-svc.yaml --namespace=kube-system

目前單機已經安裝完了,因此,可以執行以下命令將wordpress container叫起來。
如果你想換成Flannel的網路架構,可以先跳過接下來的步驟,直接去Flannel的安裝

kubectl run wordpress --image=tutum/wordpress --port=80 --hostport=81

kubectl查看,Container是否啟動,需要等個一分鐘左右,因為要從dockerhub下載檔案。

kubectl get pods
kubectl get rc

也可以用docker指令查看。

docker ps

看一下網路架構,與Docker本身的網路架構是一樣的。

#ifconfig
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1072 (1.0 KB)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:ad:f5:61
          inet addr:172.16.235.128  Bcast:172.16.235.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fead:f561/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:300420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:127648 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:437238373 (437.2 MB)  TX bytes:8566239 (8.5 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:522030 errors:0 dropped:0 overruns:0 frame:0
          TX packets:522030 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:207725326 (207.7 MB)  TX bytes:207725326 (207.7 MB)

veth0ef8f9d Link encap:Ethernet  HWaddr 26:f0:39:f9:65:3a
          inet6 addr: fe80::24f0:39ff:fef9:653a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1944 (1.9 KB)

vetha2a43fc Link encap:Ethernet  HWaddr d2:ad:ea:e1:ff:7f
          inet6 addr: fe80::d0ad:eaff:fee1:ff7f/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)
          

連到Container看一下網路IP。

root@wordpress-j0f8t:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:02
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          

ETCD在Kubernetes有著非常重要的位置,看一下etcdctl內容。

root@kubernetes:~# /opt/bin/etcdctl ls
/registry
root@kubernetes:~# ps aux|grep kube

root       4763  0.0  0.1   8392  1972 ?        Ssl  13:20   0:00 /kube-ui
root      10628  3.2  3.6 248140 36140 ?        Ssl  13:35   0:03 /opt/bin/kubelet --address=0.0.0.0 --port=10250 --hostname_override=127.0.0.1 --api_servers=http://127.0.0.1:8080 --logtostderr=true --cluster_dns=192.168.3.10 --cluster_domain=cluster.local
root      10629  0.4  1.2  16276 12500 ?        Ssl  13:35   0:00 /opt/bin/kube-scheduler --logtostderr=true --master=127.0.0.1:8080
root      10631  1.2  3.9  48896 39936 ?        Ssl  13:35   0:01 /opt/bin/kube-apiserver --address=0.0.0.0 --port=8080 --etcd_servers=http://127.0.0.1:4001 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ServiceAccount,ResourceQuota --service_account_key_file=/tmp/kube-serviceaccount.key --service_account_lookup=false
root      10632  0.6  1.9  25556 19248 ?        Ssl  13:35   0:00 /opt/bin/kube-controller-manager --master=127.0.0.1:8080 --service_account_private_key_file=/tmp/kube-serviceaccount.key --logtostderr=true
root      10633  0.2  1.4 203044 14244 ?        Ssl  13:35   0:00 /opt/bin/kube-proxy --master=http://127.0.0.1:8080 --logtostderr=true

Flannel的安裝

Let's change it to Flannel network

vim /etc/default/flanneld
FLANNEL_OPTS="--etcd-endpoints=http://127.0.0.1:2379"
service flanneld stop 
service docker stop
service kubelet stop
service kube-proxy stop
apt-get install -y bridge-utils
ip link set dev docker0 down
brctl delbr docker0
/opt/bin/etcdctl rm --recursive /coreos.com/network
/opt/bin/etcdctl set /coreos.com/network/config '{ "Network": "172.17.0.0/16" }'
service flanneld start

You will see /run/flannel/subnet.env

ls /run/flannel/subnet.env

and modify /etc/default/docker

vim /etc/default/docker
. /run/flannel/subnet.env 
DOCKER_OPTS="--bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}"

where the /run/flannel/subnet.env file shown as

vim /run/flannel/subnet.env
FLANNEL_SUBNET=172.17.16.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=false
service docker start
service kubelet start
service kube-proxy start
kubectl run wordpress --image=tutum/wordpress --port=80 --hostport=81
kubectl get pods
kubectl get rc
docker ps -a

After 1 min, you will see the result that the VM's been launched.

root@kubernetes:~/kubernetes/cluster# kubectl get pods
NAME              READY     REASON    RESTARTS   AGE
wordpress-gp4cl   1/1       Running   0          7m

You wil see the Flannel Network here.

root@kubernetes:~/kubernetes/cluster# ifconfig
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.16.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1472  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:ad:f5:61
          inet addr:172.16.235.128  Bcast:172.16.235.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fead:f561/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:297104 errors:0 dropped:0 overruns:0 frame:0
          TX packets:118915 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:435040989 (435.0 MB)  TX bytes:7818950 (7.8 MB)

flannel0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:172.17.16.0  P-t-P:172.17.16.0  Mask:255.255.0.0
          UP POINTOPOINT RUNNING  MTU:1472  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:87012 errors:0 dropped:0 overruns:0 frame:0
          TX packets:87012 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:184668449 (184.6 MB)  TX bytes:184668449 (184.6 MB)

vethbd09c72 Link encap:Ethernet  HWaddr 96:0a:11:72:07:77
          inet6 addr: fe80::940a:11ff:fe72:777/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1296 (1.2 KB)
          
          

From Official Site, you wiil see the result is right.

However, we only have one node here.

Also see the container's IP, and login to the container.

root@wordpress-gp4cl:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:10:02
          inet addr:172.17.16.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:acff:fe11:1002/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1296 (1.2 KB)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

網路問題

在Container內,ping www.google.com,我發現不通,查了一下container內的nameserver

vim /etc/resolv.conf
nameserver 192.168.3.10
nameserver 172.16.235.2
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

delete 192.168.3.10 nameserver

vim /etc/resolv.conf
nameserver 172.16.235.2
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

再ping一下通了

oot@wordpress-gp4cl:/# ping www.google.com
PING www.google.com (64.233.188.103) 56(84) bytes of data.
64 bytes from tk-in-f103.1e100.net (64.233.188.103): icmp_seq=1 ttl=127 time=10.2 ms
64 bytes from tk-in-f103.1e100.net (64.233.188.103): icmp_seq=2 ttl=127 time=9.81 ms

發現了deploy config有問題,下次可以修改一下。

root@kubernetes:~/kubernetes/cluster# grep -R  192.168.3 *
ubuntu/config-default.sh:export SERVICE_CLUSTER_IP_RANGE=${SERVICE_CLUSTER_IP_RANGE:-192.168.3.0/24}  # formerly PORTAL_NET
ubuntu/config-default.sh:DNS_SERVER_IP=${DNS_SERVER_IP:-"192.168.3.10"}

改成8.8.8.8試試看,這部分是手動在container中改的,下次部署可以改上述的檔案ubuntu/config-default.shDNSSERVERIP.

nameserver 8.8.8.8
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

ping google again, it works well.

root@wordpress-gp4cl:/# ping www.google.com
PING www.google.com (74.125.203.106) 56(84) bytes of data.
64 bytes from th-in-f106.1e100.net (74.125.203.106): icmp_seq=1 ttl=127 time=11.4 ms
64 bytes from th-in-f106.1e100.net (74.125.203.106): icmp_seq=2 ttl=127 time=12.5 ms
64 bytes from th-in-f106.1e100.net (74.125.203.106): icmp_seq=3 ttl=127 time=15.0 ms

To check the detailed infomation via curl.

root@kubernetes:/# curl -s http://localhost:4001/v2/keys/coreos.com/network/subnets | python -mjson.tool
{
    "action": "get",
    "node": {
        "createdIndex": 37,
        "dir": true,
        "key": "/coreos.com/network/subnets",
        "modifiedIndex": 37,
        "nodes": [
            {
                "createdIndex": 37,
                "expiration": "2016-03-30T07:37:28.947871304Z",
                "key": "/coreos.com/network/subnets/172.17.16.0-24",
                "modifiedIndex": 37,
                "ttl": 83666,
                "value": "{\"PublicIP\":\"172.16.235.128\"}"
            }
        ]
    }
}

下一篇我們可以來看看VXlan的模式如何開啟。

Saturday, March 19, 2016

Implement ETCD Watch by using Golang

etcdWatch

一直很喜歡etcd這套key-value store。透過http的方式存取,COREOS也提供強大的Client供使用。
Watch找了很久,終於知道該怎麼寫了,這中間,還依賴了之前安裝了auto-completion (見我之前的Blog),c-x x-o實在太強大了。
不止Watch,還可對不同的操作Watch,以下為代碼。

package main

import (
    "github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
    "github.com/coreos/etcd/client"
    "log"
    "time"
)

func main() {
    cfg := client.Config{
        Endpoints: []string{"http://127.0.0.1:2379"},
        Transport: client.DefaultTransport,
        // set timeout per request to fail fast when the target endpoint is unavailable
        HeaderTimeoutPerRequest: time.Second,
    }
    c, err := client.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    kapi := client.NewKeysAPI(c)

    watcher := kapi.Watcher("/cred/", &client.WatcherOptions{
        Recursive: true,
    })
    log.Println("redyto run")
    for {
        detail, _ := watcher.Next(context.Background())
        log.Println("redyto run")
        if detail.Action == "expire" {
            log.Println("expire", detail.Node.Key, detail.Node.Value)
        }
        if detail.Action == "set" {
            log.Println("set", detail.Node.Key, detail.Node.Value)
        }
        if detail.Action == "update" {
            log.Println("update")
        }
        if detail.Action == "expired" {
            log.Println("expired")
        }
        if detail.Action == "delete" {
            log.Println("delete")
        }
       }

}

很可惜ETC取消了Lock的功能,相信以後還會開啟的。

Monday, March 14, 2016

Docker Container 初探

dockerInstall

Docker Install

最近事情有點多,所以只好中英文交雜著,也懶得管文法了。

只是初探而已,我想我就直接用Ubuntu14.04的Docker,不做升級了。

apt-get update 
apt-get -y install docker.io

測試安裝結果

裝完了,拉個Ubuntu Container檔案看看。 接下來連進這台Container中。

docker pull ubuntu
docker run -i -t ubuntu /bin/bash

An test image

download a webapp from docker hub.

指下載image不啟動Container。

docker pull training/webapp

runing the webapp for testing

docker run -d -p 80:5000 training/webapp python app.py

提取container的5000 port maps to host 80 port. So one can use

curl http://hostip:80

to test it,其中hostip為host的IP。

commit a modified image

You can commit a running container and will not affect the running status(UP).

docker commit -m "Added new webapp" -a "Docker Newbee for webapp" 89f4c0576be3 training/webapp2:v2

where 89xx is id, traning/webapp repository name, and v2 is a TAG.
The default TAG is latest so if you want to run a image with v2 tag, you can use

docker run -d -p 81:5001 training/webapp2:v2 python app1.py

why show Exited

If you have no job running, docker will mark it as Exited(0) while type docker ps -a. You can have a test on it, after 10 sec it will show Exited(0).

docker run --name cont6 ubuntu sleep 10 &

if then command with -t, TTY, it will not show Exited, the status will always show "UP".

docker run -t --name cont6 ubuntu /bin/bash 

Time Namespace Problem

If you are running container with the following commands, changing date is not permission. The admin permission is being limited.

docker run -t --name intermode2 ubuntu

but use the following commands will release the admin permission.

docker run -t --privileged --name intermode2 ubuntu

When you change the date with Date -s " 2000 / 01 / 01 00 : 00 : 00 command in the container.
The host's date will be changed, since kernel not support time namespace.
When disable --privileged will make the user as a guest user and cannot be admin user. Some command also will be disable, such as mount command. If you want to be a admin user, you can not avoid the lack of namespace supported problem.

tty Connection

tty connect to container by using docker command

docker exec -i -t id bash

Docker Link

Docker link 只能發生在同一台Server。

Running db first.

docker run -d --name db training/postgres

Running web then.

docker run -d -P --name web --link db:db training/webapp python app.py

連到web server檢查/etc/hosts發現如下,link只是更改了/etc/hosts/加入了db information。

72.17.0.4      8eb308a6f375
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.3      db a768185b48d7

The most cool thing is:
If the db' IP changed due to DHCP or ..., the /etc/hosts in web server will automatically update.

One can check the detail connection information through:

docker inspect -f "{{ .HostConfig.Links }}" web

It's in the section of HostConfig with variable Links using command docker inspect.

Dockerfile

建立一個目錄放入Dockerfile,名為Dockerfile

# A basic apache server. To use either add or bind mount content under /var/www
FROM ubuntu

MAINTAINER Kimbro Staken version: 0.1

RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/*

ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2

EXPOSE 80

CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]

其中FROM指下載的image位置。
RUN 表示在內部執行的命令,如果image存在,可透過此命令做為init的方法執行啟動程式。發生在build階段。
CMD 表示docker -it ubuntu python webapp.py。其中CMD就是指這段最後的指令。發生在啟動階段。

No init

這是一件很難想像的事情,而大部分的人好似也接受了。官網希望一個Container只跑一個特定的服務。 但我總覺得這會讓事情沒有更容易。這也是個奇想,在任何地方,任何老闆,大概沒有辦法接受有這樣的情況。
這也證明了創意才是重點,那些小問題一點都不重要。 這到目前為止還是個大問題,SIGKILL捕捉的到嗎?sub-process清得乾淨嗎?

s6 seems the best application for solving this problem.

ambassador

這是一個好想法,我應該再往這裡面專研一下。
首先我們先測試一下。

在server1,啟動redis-server and abassador connector to redis-server

docker run -d --name redis redis
docker run -d --link redis:redis --name ambassador1 -p 6379:6379 svendowideit/ambassador

在server2,啟動redis-client and abassador connector to redis-cli

docker run -d --name ambassador2 --expose 6379 -e REDIS_PORT_6379_TCP=tcp://172.16.235.128:6379 svendowideit/ambassador
docker run -d -t  --link ambassador2:redis relateiq/redis-cli

其中,172.16.235.128為server1的IP。

我們進去redis-cli看看

root@7882118ba9aa:/# env
REDIS_PORT_6379_TCP_PROTO=tcp
HOSTNAME=7882118ba9aa
TERM=xterm
REDIS_NAME=/mad_stallman/redis
REDIS_PORT_6379_TCP_ADDR=172.17.0.2
REDIS_PORT_6379_TCP_PORT=6379
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
REDIS_ENV_REDIS_PORT_6379_TCP=tcp://172.16.235.128:6379
REDIS_PORT_6379_TCP=tcp://172.17.0.2:6379
SHLVL=1
HOME=/
REDIS_PORT=tcp://172.17.0.2:6379
_=/usr/bin/env

redis-cli的script為

root@7882118ba9aa:/# cat start.sh
#!/bin/bash

if [ $# -eq 0 ]; then
    /redis/src/redis-cli -h $REDIS_PORT_6379_TCP_ADDR -p $REDIS_PORT_6379_TCP_PORT
else
    /redis/src/redis-cli "$@"
fi

其中REDISPORT6379TCPADDRREDISPORT6379TCPPORT在環境變數中已經設立。 啟動start.sh後執行

root@7882118ba9aa:/# ./start.sh
redis 172.17.0.2:6379> ping
PONG

有反應了。

這裏解釋一下,為何是--link ambassador2:redis。 首先我們看一下envstart.sh間用什麼變數溝通,答案是REDIS_這個prefix。
假設我們用--link ambassador2:ambassador2會怎樣?

root@f0a8b07680bc:/# env
AMBASSADOR2_PORT_6379_TCP=tcp://172.17.0.2:6379
HOSTNAME=f0a8b07680bc
TERM=xterm
AMBASSADOR2_PORT_6379_TCP_PROTO=tcp
AMBASSADOR2_PORT_6379_TCP_PORT=6379
AMBASSADOR2_PORT=tcp://172.17.0.2:6379
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
AMBASSADOR2_NAME=/furious_einstein/ambassador2
AMBASSADOR2_ENV_REDIS_PORT_6379_TCP=tcp://172.16.235.128:6379
SHLVL=1
HOME=/
AMBASSADOR2_PORT_6379_TCP_ADDR=172.17.0.2
_=/usr/bin/env

env被改成AMBASSADOR2的env了,使得,script對應的參數是錯的。

另外也影響了/etc/hosts

root@f0a8b07680bc:/# cat /etc/hosts
172.17.0.10 f0a8b07680bc
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2  ambassador2 5fb7c99e8b6c

因此link可解讀為--link destination container name:local alias name
local alias name表示source container(啟動的這台)所隱含的參數相對於desination container。

command

list all container
docker ps -a

delete a container 
docker rm id

show all images
docker images

run a container from an image: with -t to keep inteactive, and alway show UP.  
if not contains -t will show Exited(0) while no job running.  
docker run -t  --name intermode ubuntu

to run it on the background
docker run -d rethinkdb

remove an image
docker rmi id

show the detailed container infomation
docker inspect id

trouble shooting

already being pulled by another client. Waiting.

service restart docker

check list of container and see the status that should be shown "UP".

docker start id