Keystone: Setup Fernet Token and Analysis Source Code

Introduce Keystone Fernet Token

We want to introduce how to setup Fernet token in Keystone, and trace keystone source code (kilo) to understand how get token and validate token works.

Installation

Assume you have installed OpenStack or Keystone already. You can see my Blog How to install Keystone.

http://gogosatellite.blogspot.tw/2016/03/keystone-domain-endpoint.html

Then, you can change the setting based on a normal deployment of Keystone.

edit openrc

export OS_TOKEN=iamadmin
export OS_SERVICE_ENDPOINT=http://controller:35357/v2.0

Set up Environment

source openrc

mkdir /etc/keystone/fernet-keys/
chown -R keystone:keystone /etc/keystone/fernet-keys
keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone

edit /etc/keystone/keystone.conf

[token]
provider = keystone.token.providers.fernet.Provider
#provider = keystone.token.providers.uuid.Provider
driver = keystone.token.persistence.backends.sql.Token

We use fernet token instead of uuid shown as above. We keeyp the setting of driver but it takes no effect.

Restart keystone service

service apache2 restart

to get token from Keystone.

curl -si -d @token-request2.json -H "Content-type: application/json" http://172.16.235.128:35357/v3/auth/tokens

where token-request2.json

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "name": "Default"
                    },
                    "name": "newuser",
                    "password": "newuser"
                }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "name": "Default"
                },
                "name": "testtenant"
            }
        }
    }
}

Result

You can see the X-Subject-Token, we got the Fernet Token. Where getToken.sh is just a curl command we discussed above.

junmeindeMacBook-Pro:openstack_api junmein$ bash getToken.sh
HTTP/1.1 201 Created
Date: Thu, 26 May 2016 11:10:16 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Subject-Token: gAAAAABXRtmaD0t0r3n0suqWsEOu4Yp76JaLgOJYLdzRJDNx0FwTb5oBVn429dMmMhiQFVGXltQVvlrhaVBaB9SkYnuTqxq06LnjoHkTmZd-afF5fP-g0YX9n2pG6ESNFgNepgmQANQqEl0oVBrC_S8SrKP9f8QcJkuxBDSwQfKIi3mylbvugK4%3D
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-58735ed6-aeed-4fcb-99b2-bf03512fc45f
Content-Length: 2065
Content-Type: application/json

{"token": {"methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}], "expires_at": "2016-05-26T12:10:17.283270Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "258b879e4df748caa1bac3416d38a819", "name": "testtenant"}, "catalog": [{"endpoints": [{"region_id": "RegionOne", "url": "http://controller:35357/v2.0", "region": "RegionOne", "interface": "admin", "id": "02f744638b2f44

To make sure the token table in the database won't be increased, since fernet token is not persistent.

mysql -uroot -pshark -e 'use keystone; select * from token;'|wc -l

How to Create Token using Fernet Token Algorithm

def create_token(self, user_id, expires_at, audit_ids, methods=None,
                 domain_id=None, project_id=None, trust_id=None,
                 federated_info=None):
    """Given a set of payload attributes, generate a Fernet token."""
    
    .
    .
    elif project_id:
    version = ProjectScopedPayload.version
    payload = ProjectScopedPayload.assemble(
        user_id,
        methods,
        project_id,
        expires_at,
        audit_ids)
        
        
class ProjectScopedPayload(BasePayload):
    version = 2

    @classmethod
    def assemble(cls, user_id, methods, project_id, expires_at, audit_ids):
        """Assemble the payload of a project-scoped token.

        :param user_id: ID of the user in the token request
        :param methods: list of authentication methods used
        :param project_id: ID of the project to scope to
        :param expires_at: datetime of the token's expiration
        :param audit_ids: list of the token's audit IDs
        :returns: the payload of a project-scoped token

        """
        b_user_id = cls.attempt_convert_uuid_hex_to_bytes(user_id)
        methods = auth_plugins.convert_method_list_to_integer(methods)
        b_project_id = cls.attempt_convert_uuid_hex_to_bytes(project_id)
        expires_at_int = cls._convert_time_string_to_int(expires_at)
        b_audit_ids = list(map(provider.random_urlsafe_str_to_bytes,
                           audit_ids))
        return (b_user_id, methods, b_project_id, expires_at_int, b_audit_ids)    

So the paidload is a list that one can access it by using paidload[0]. It's quite a beautiful skill that we will discuss it later.

and then it will form the token as follows.

versioned_payload = (version,) + payload
serialized_payload = msgpack.packb(versioned_payload)
token = self.pack(serialized_payload)

return token

so the serial encryption packs are

• version
• user_id
• method
• project_id
• expire_at
• audit_ids

We can see what b_user_id = cls.attempt_convert_uuid_hex_to_bytes(user_id) doing.

@classmethod
def convert_uuid_hex_to_bytes(cls, uuid_string):
    """Compress UUID formatted strings to bytes.

    :param uuid_string: uuid string to compress to bytes
    :returns: a byte representation of the uuid

    """
    # TODO(lbragstad): Wrap this in an exception. Not sure what the case
    # would be where we couldn't handle what we've been given but incase
    # the integrity of the token has been compromised.
    uuid_obj = uuid.UUID(uuid_string)
    return uuid_obj.bytes

Just ，transfer uuid hex to bytes or int。

What is Serialize Paidload !! Godd Stuff!!

It's a beautiful way to pack a accessable attibute, list, to a string and send it to socket or mq. And we then unpack it and access it as a list directly. That's msgpack doing.

This is an example, to realize what is msgpack doing. This is called serialize paidload.

ori = ('ddd', 'aaa', 'bbb', 'ccc')
pack = ��ddd�aaa�bbb�ccc 
unpack = ['ddd', 'aaa', 'bbb', 'ccc'] 

In keystone we pack it and encrypt it. After decrypt it we can access a list directly.

encrypt key

def crypto(self):
    """Return a cryptography instance.

    You can extend this class with a custom crypto @property to provide
    your own token encoding / decoding. For example, using a different
    cryptography library (e.g. ``python-keyczar``) or to meet arbitrary
    security requirements.

    This @property just needs to return an object that implements
    ``encrypt(plaintext)`` and ``decrypt(ciphertext)``.

    """
    keys = utils.load_keys()
  .
  .
  .

that contains primary, secondary, ..... and rotation. and finally encrypt it by pack function.

def load_keys():
    """Load keys from disk into a list.

    The first key in the list is the primary key used for encryption. All
    other keys are active secondary keys that can be used for decrypting
    tokens.

    """
    if not validate_key_repository():
        return []

    # build a dictionary of key_number:encryption_key pairs
    keys = dict()
    for filename in os.listdir(CONF.fernet_tokens.key_repository):
        path = os.path.join(CONF.fernet_tokens.key_repository, str(filename))
        if os.path.isfile(path):
            with open(path, 'r') as key_file:
                try:
                    key_id = int(filename)
                except ValueError:
                    pass
                else:
                    keys[key_id] = key_file.read()

    if len(keys) != CONF.fernet_tokens.max_active_keys:
        # If there haven't been enough key rotations to reach max_active_keys,
        # or if the configured value of max_active_keys has changed since the
        # last rotation, then reporting the discrepancy might be useful. Once
        # the number of keys matches max_active_keys, this log entry is too
        # repetitive to be useful.
        LOG.info(_LI(
            'Loaded %(count)s encryption keys from: %(dir)s'), {
                'count': len(keys),
                'dir': CONF.fernet_tokens.key_repository})

    # return the encryption_keys, sorted by key number, descending
    return [keys[x] for x in sorted(keys.keys(), reverse=True)]

Now we want to solve the fernet key problem.
Before obtaining the fernet token service, we must setup by keystone-manage. As the steps we done before.

 mkdir /etc/keystone/fernet-keys/
$ keystone-manage fernet_setup

and the configuration

[fernet_tokens]
        # key repository where the fernet keys are stored
    key_repository = /etc/keystone/fernet-keys/
        # maximum number of keys in key repository
    max_active_keys =  # default is 3

$ ls /etc/keystone/fernet-keys
0 1 2 3 4

An example for fernet.MultiFernet

https://cryptography.io/en/latest/fernet/
https://media.readthedocs.org/pdf/cryptography/latest/cryptography.pdf

>>> from cryptography.fernet import Fernet, MultiFernet
>>> key1 = Fernet(Fernet.generate_key())
>>> key2 = Fernet(Fernet.generate_key())
>>> f = MultiFernet([key1, key2])
>>> token = f.encrypt(b"Secret message!")
.
.
>>> f.decrypt(token)
'Secret message!'

After load serveral keys, and pass to fernet encrypt function to get encrpt result of token. and where the key we get is from the directory of /etc/keystone/fernet-keys/.

Key Rotation

What is MultiFernet doing, according to the offcial site's explanation of MultiFernet:

MultiFernet performs all encryption options using the first key in the list provided. MultiFernet attempts to decrypt tokens with each key in turn. A cryptography.fernet.InvalidToken exception is raised if the correct key is not found in the list provided. Key rotation makes it easy to replace old keys. You can add your new key at the front of the list to start encrypting new messages, and remove old keys as they are no longer needed.

It will use the first key to encrypt the data. Then decrypt by each key in turn, since the key will rotate for security.

where is the key location

root@controller:~# ls /etc/keystone/fernet-keys/
0  1

to see content of the key

root@controller:~# cat /etc/keystone/fernet-keys/0
BIoWWHPzDcoNRhFsg3TFzrRHoYVlL1MECDreHBREqJo=

Conclusion: Generate Token and Validate Token

使用project based Fernet Token包含以下物件: Fernet algorithsm uses the bellow parameters.

• version
• user_id
• method
• project_id
• expire_at
• audit_ids

The process of getting a token:

transfer the above hex string object to **bytes** or **int** -> to form a list -> serialize pack(msgpack.pack) -> Rotated MutiFernet Encryption (Fernet.MultiFernet(key1, key2..)) -> Token.

The process of validating a token (inverse process):

Token -> Rotated MultiFernet Decryption -> serialize unpack(msgpack.unpack) -> list -> access list -> bytes int object transfers to  hex string -> to check expire time .. -> validate or invalidate token.

Now we realize how is the keystone to get token and validate token.

And next chapter, we want to talk more about validate token.

How To Validate Token

def validate_token(self, token):
    """Validates a Fernet token and returns the payload attributes."""
    # Convert v2 unicode token to a string
    if not isinstance(token, six.binary_type):
        token = token.encode('ascii')
    
    serialized_payload = self.unpack(token)
    versioned_payload = msgpack.unpackb(serialized_payload)
    version, payload = versioned_payload[0], versioned_payload[1:]
    
    # depending on the formatter, these may or may not be defined
    domain_id = None
    project_id = None
    trust_id = None
    federated_info = None
    
    if version == UnscopedPayload.version:
        (user_id, methods, expires_at, audit_ids) = (
            UnscopedPayload.disassemble(payload))
    elif version == DomainScopedPayload.version:
        (user_id, methods, domain_id, expires_at, audit_ids) = (
            DomainScopedPayload.disassemble(payload))
    elif version == ProjectScopedPayload.version:
        (user_id, methods, project_id, expires_at, audit_ids) = (
            ProjectScopedPayload.disassemble(payload))
    elif version == TrustScopedPayload.version:
        (user_id, methods, project_id, expires_at, audit_ids, trust_id) = (
            TrustScopedPayload.disassemble(payload))
    elif version == FederatedPayload.version:
        (user_id, methods, expires_at, audit_ids, federated_info) = (
            FederatedPayload.disassemble(payload))
    else:
        # If the token_format is not recognized, raise ValidationError.
        raise exception.ValidationError(_(
            'This is not a recognized Fernet payload version: %s') %
            version)
    # rather than appearing in the payload, the creation time is encoded
    # into the token format itself
    created_at = TokenFormatter.creation_time(token)
    created_at = timeutils.isotime(at=created_at, subsecond=True)
    expires_at = timeutils.parse_isotime(expires_at)
    expires_at = timeutils.isotime(at=expires_at, subsecond=True)
    
    return (user_id, methods, audit_ids, domain_id, project_id, trust_id,
            federated_info, created_at, expires_at)

def disassemble(cls, payload):
    """Disassemble a payload into the component data.

    :param payload: the payload of a token
    :return: a tuple containing the user_id, auth methods, project_id,
             expires_at_str, and audit_ids

    """
    (is_stored_as_bytes, user_id) = payload[0]
    if is_stored_as_bytes:
        user_id = cls.attempt_convert_uuid_bytes_to_hex(user_id)
    methods = auth_plugins.convert_integer_to_method_list(payload[1])
    (is_stored_as_bytes, project_id) = payload[2]
    if is_stored_as_bytes:
        project_id = cls.attempt_convert_uuid_bytes_to_hex(project_id)
    expires_at_str = cls._convert_int_to_time_string(payload[3])
    audit_ids = list(map(provider.base64_encode, payload[4]))

    return (user_id, methods, project_id, expires_at_str, audit_ids)

The process is very similar to generate Token.