Friday, May 13, 2016

Deploy Kilo OpenStack Multiple Regions using DevStack

multipleregion

OpenStack Kilo Multiple Region Deployment via DevStack

Environment

node1 eth0:192.168.1.161 eth1:192.168.141.161

node2 eth0:192.168.1.162 eth1:192.168.141.162

gateway:

stack@jonah2:~/devstack$ route
Kernel IP routing tableceilometer-alarm-not   22$(L) ceilometer-alarm-evaluator*
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.140.254 0.0.0.0         UG    0      0        0 eth1
192.168.1.0     *               255.255.255.0   U     0      0        0 br-ex
192.168.140.0   *               255.255.252.0   U     0      0        0 eth1

Devstack Deploment

Based on our previous works, we can start to deploy our system

http://gogosatellite.blogspot.tw/2016/05/deploy-openstack-kilo-with-ceilometer.html

Let's start to deploy two region OpenStack.

In Region1 (node1)

[[local|localrc]]
DEST=/opt/stack
#change github to trystack
HOST_IP=192.168.1.161
GIT_BASE=${GIT_BASE:-http://git.trystack.cn}
#NOVNC_REPO=${NOVNC_REPO:-http://git.trystack.cn/kanaka/noVNC.git}

ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken


ENABLED_SERVICES=key,rabbit,mysql,horizon
ENABLED_SERVICES+=,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,g-api,g-reg

# Enabling Neutron (network) Service
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service q-vpn


# Ceilometer
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector ceilometer-api
enable_service ceilometer-alarm-notifier ceilometer-alarm-evaluator
CEILOMETER_BACKEND=mongodb

### Enable Heat
##enable_service heat h-api h-api-cfn h-api-cw h-eng


## Neutron options
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
NETWORK_GATEWAY=10.0.0.1
PRIVATE_SUBNET_NAME=privateA

PUBLIC_SUBNET_NAME=public-subnet
FLOATING_RANGE=192.168.140.0/24
PUBLIC_NETWORK_GATEWAY=192.168.140.254
#Q_FLOATING_ALLOCATION_POOL="start=172.24.4.11,end=172.24.4.29"
PUBLIC_INTERFACE=eth0
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex



LIBVIRT_TYPE=qemu

IMAGE_URLS="http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-uec.tar.gz"

SCREEN_LOGDIR=/opt/stack/screen-logs
SYSLOG=True
LOGFILE=~/devstack/stack.sh.log


Q_USE_DEBUG_COMMAND=True

# RECLONE=No
RECLONE=yes
OFFLINE=False

In Region2 (node2)

[[local|localrc]]
DEST=/opt/stack
#change github to trystack
HOST_IP=192.168.1.162
GIT_BASE=${GIT_BASE:-http://git.trystack.cn}
#NOVNC_REPO=${NOVNC_REPO:-http://git.trystack.cn/kanaka/noVNC.git}
REGION_NAME=RegioneTwo
KEYSTONE_SERVICE_HOST=192.168.141.161
KEYSTONE_AUTH_HOST=192.168.141.161

ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken


ENABLED_SERVICES=key,rabbit,mysql,horizon
ENABLED_SERVICES+=,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,g-api,g-reg

# Enabling Neutron (network) Service
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service q-vpn


# Ceilometer
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector ceilometer-api
enable_service ceilometer-alarm-notifier ceilometer-alarm-evaluator
CEILOMETER_BACKEND=mongodb

### Enable Heat
##enable_service heat h-api h-api-cfn h-api-cw h-eng


## Neutron options
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
NETWORK_GATEWAY=10.0.0.1
PRIVATE_SUBNET_NAME=privateA

PUBLIC_SUBNET_NAME=public-subnet
FLOATING_RANGE=192.168.140.0/24
PUBLIC_NETWORK_GATEWAY=192.168.140.254
#Q_FLOATING_ALLOCATION_POOL="start=172.24.4.11,end=172.24.4.29"
PUBLIC_INTERFACE=eth0
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex



LIBVIRT_TYPE=qemu

IMAGE_URLS="http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-uec.tar.gz"

SCREEN_LOGDIR=/opt/stack/screen-logs
SYSLOG=True
LOGFILE=~/devstack/stack.sh.log


Q_USE_DEBUG_COMMAND=True

# RECLONE=No
RECLONE=yes
OFFLINE=False

We mentions that we have set addtional parameters in Region2's local.conf shown as followed.

REGION_NAME 
KEYSTONE_SERVICE_HOST
KEYSTONE_AUTH_HOST 

Result

A simple way to realize the result we deployed is to connect to Horizon.

After login to the Horizon, we can see two Regions as above figure. And we can switch to different region to see different resources.

How to Get Quota Information

edit openrc_admin

export OS_SERVICE_TOKEN=tokentoken
export OS_SERVICE_ENDPOINT=http://localhost:35357/v2.0
source openrc_admin
keystone --debug tenant-list
.
.
.
+----------------------------------+--------------------+---------+
|                id                |        name        | enabled |
+----------------------------------+--------------------+---------+
| 03daba5a7f73432d8edecb881c1e381a |       admin        |   True  |
| 11d655a1358546ae86ad6d7ba61de6f4 |        demo        |   True  |
| 5d609499001c43ecb4137aedf3416995 | invisible_to_admin |   True  |
| 199467ffaa4a43408799ea6afb186862 |      service       |   True  |
+----------------------------------+--------------------+---------+

To get the tenant_id you want, demo, and then

nova-manage --debug project quota 11d655a1358546ae86ad6d7ba61de6f4
.
.
.
Quota                                Limit      In Use     Reserved
2016-05-13 08:45:24.542 DEBUG nova.quota [req-5cf3fcff-8462-43bf-9361-333ad13907f4 None None] Getting all quota usages for project: 11d655a1358546ae86ad6d7ba61de6f4 get_project_quotas /opt/stack/nova/nova/quota.py:290
metadata_items                       128        0          0
injected_file_content_bytes          10240      0          0
server_group_members                 10         0          0
server_groups                        10         0          0
ram                                  51200      0          0
floating_ips                         10         0          0
security_group_rules                 20         0          0
instances                            10         0          0
key_pairs                            100        0          0
injected_files                       5          0          0
cores                                20         0          0
fixed_ips                            unlimited  0          0
injected_file_path_bytes             255        0          0
security_groups                      10         0          0

Since the quota controled from individual modules, such as nova, neutron, and not in Keystone. One can update quota by check this http://docs.openstack.org/openstack-ops/content/projects_users.html.

The problem is Glance not support tenant quota in Kilo, but it can setup global quota of image storage.

Via the nova-manager, we have to mention that the quota must be set to regions individually. There are no way to set global quota for a tenant.

Compare Database

Since our keystone is set to Region1 while deploying Region2. So there are no keystone information in Region2 database but in Region1.

In Region1

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| glance             |
| keystone           |
| mysql              |
| neutron            |
| nova               |
| performance_schema |
+--------------------+
7 rows in set (0.00 sec)

In Region2

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| glance             |
| mysql              |
| neutron            |
| nova               |
| performance_schema |
+--------------------+
6 rows in set (0.00 sec)

We can see the keystone is not existed in Region2's Database.

What Changed During Deployment

To address what changed in openstack configuration. We can grip the IP address of Region1 to make sure changes shown as followsed.

root@devr2:/etc# grep -R "192.168.141.161" *

ceilometer/ceilometer.conf:os_auth_url = http://192.168.141.161:5000/v2.0
ceilometer/ceilometer.conf:auth_uri = http://192.168.141.161:5000
ceilometer/ceilometer.conf:auth_url = http://192.168.141.161:35357
glance/glance-cache.conf:auth_url = http://192.168.141.161:35357/v2.0
glance/glance-api.conf:auth_uri = http://192.168.141.161:5000
glance/glance-api.conf:auth_url = http://192.168.141.161:35357
glance/glance-registry.conf:auth_uri = http://192.168.141.161:5000
glance/glance-registry.conf:auth_url = http://192.168.141.161:35357
neutron/neutron.conf:auth_url = http://192.168.141.161:35357
neutron/neutron.conf:auth_uri = http://192.168.141.161:5000
neutron/neutron.conf:auth_url = http://192.168.141.161:35357
neutron/metadata_agent.ini:auth_uri = http://192.168.141.161:5000
neutron/metadata_agent.ini:auth_url = http://192.168.141.161:35357
nova/nova-dhcpbridge.conf:keystone_ec2_url = http://192.168.141.161:5000/v2.0/ec2tokens
nova/nova-dhcpbridge.conf:auth_uri = http://192.168.141.161:5000
nova/nova-dhcpbridge.conf:auth_url = http://192.168.141.161:35357
nova/nova.conf:keystone_ec2_url = http://192.168.141.161:5000/v2.0/ec2tokens
nova/nova.conf:auth_uri = http://192.168.141.161:5000
nova/nova.conf:auth_url = http://192.168.141.161:35357
nova/nova.conf:admin_auth_url = http://192.168.141.161:35357/v2.0

That means if we want switch back to Region2's keystone, we need to modify the above configuration and restart the daemon shown as above.

確定不同services都有不同的endpoint分別對應不同region。不同region的位置也都是對的。

keystone --debug endpoint-list
.
.
.
+----------------------------------+------------+------------------------------------------------+------------------------------------------------+------------------------------------------------+----------------------------------+
|                id                |   region   |                   publicurl                    |                  internalurl                   |                    adminurl                    |            service_id            |
+----------------------------------+------------+------------------------------------------------+------------------------------------------------+------------------------------------------------+----------------------------------+
| 030a7da536d24ecb9f59953ad2905833 | RegioneTwo |          http://192.168.141.162:8773/          |          http://192.168.141.162:8773/          |          http://192.168.141.162:8773/          | fbb89d73112248a98fd7e67af6dbfb8e |
| 0425acb1931c4409bda1bf9c6c935baf | RegionOne  |          http://192.168.141.161:8773/          |          http://192.168.141.161:8773/          |          http://192.168.141.161:8773/          | fbb89d73112248a98fd7e67af6dbfb8e |
| 0ad4766387b545a7b234627982317fa6 | RegionOne  |          http://192.168.141.161:8777/          |          http://192.168.141.161:8777/          |          http://192.168.141.161:8777/          | 275a2c49545d426a9166185da1c31a16 |
| 0f3c9d33607744f1a2e1a7811162f2ba | RegionOne  |          http://192.168.141.161:9696/          |          http://192.168.141.161:9696/          |          http://192.168.141.161:9696/          | 7cd1067bfb3a4485b0f90c76ae4bd419 |
| 206b8531d7fd49e493382041741d4088 | RegioneTwo |        http://192.168.141.161:5000/v2.0        |        http://192.168.141.161:5000/v2.0        |       http://192.168.141.161:35357/v2.0        | 42f34398300648e7b702bdbb40ddb193 |
| 394ce1871229498eb8a68c3750fdafd1 | RegioneTwo |  http://192.168.141.162:8774/v2/$(tenant_id)s  |  http://192.168.141.162:8774/v2/$(tenant_id)s  |  http://192.168.141.162:8774/v2/$(tenant_id)s  | 87fafd630bad47df9c34f4bb51cd5ca1 |
| 397bacb64bb047128a6b8403ea609aa5 | RegioneTwo |          http://192.168.141.162:9292           |          http://192.168.141.162:9292           |          http://192.168.141.162:9292           | 46fa35450c80421eb59faf4f88bfc039 |
| 4ad1c79eb2e844c99020374d54ed561e | RegioneTwo |          http://192.168.141.162:8777/          |          http://192.168.141.162:8777/          |          http://192.168.141.162:8777/          | 275a2c49545d426a9166185da1c31a16 |
| 4d88dc1c94a94555aa516181105f0cd7 | RegioneTwo |          http://192.168.141.162:9696/          |          http://192.168.141.162:9696/          |          http://192.168.141.162:9696/          | 7cd1067bfb3a4485b0f90c76ae4bd419 |
| 9c6a6fe1f6924249a6da8e90ea23bd8a | RegionOne  |          http://192.168.141.161:9292           |          http://192.168.141.161:9292           |          http://192.168.141.161:9292           | 46fa35450c80421eb59faf4f88bfc039 |
| c78f7d8bfbb044e68297e0076011b64c | RegionOne  |  http://192.168.141.161:8774/v2/$(tenant_id)s  |  http://192.168.141.161:8774/v2/$(tenant_id)s  |  http://192.168.141.161:8774/v2/$(tenant_id)s  | 87fafd630bad47df9c34f4bb51cd5ca1 |
| e00afb4d18484e2684124f6b98b88baa | RegionOne  |        http://192.168.141.161:5000/v2.0        |        http://192.168.141.161:5000/v2.0        |       http://192.168.141.161:35357/v2.0        | 42f34398300648e7b702bdbb40ddb193 |
| e4e0e9050b644de19b874b34e8b6fb5b | RegionOne  | http://192.168.141.161:8774/v2.1/$(tenant_id)s | http://192.168.141.161:8774/v2.1/$(tenant_id)s | http://192.168.141.161:8774/v2.1/$(tenant_id)s | 8a616bcaee344a9382ecade08c2f4dbe |
| f479712d18f64883928d220a0c756757 | RegioneTwo | http://192.168.141.162:8774/v2.1/$(tenant_id)s | http://192.168.141.162:8774/v2.1/$(tenant_id)s | http://192.168.141.162:8774/v2.1/$(tenant_id)s | 8a616bcaee344a9382ecade08c2f4dbe |
+----------------------------------+------------+------------------------------------------------+------------------------------------------------+------------------------------------------------+----------------------------------+

No comments:

Post a Comment