URI in config: transport type (SSL, plain), parameters (keys, certs)

[Gerold103]

The related issue is #325. The discussion starts with a description of how the task looks in my understanding. Then I provide my vision of API, frequent questions, alternatives. Note, nothing about behaviour or internals - this is just some new API to a feature which already exists in core.

Problems with how it works now

VShard config requires to specify URI for each replica / instance / node / storage in there. For example:

{
    sharding = {
        ['storage_1_uuid'] = {
            replicas = {
                ['storage_1_a_uuid'] = {
                    uri = 'storage:[email protected]:3301',
                    name = 'storage_1_a',
                },
            },
        },
    },
}

This is fine as long as the storage accepts plain TCP connections. But gets complicated when want to use SSL which exists in Tarantool Enterprise.

To use SSL the user needs to set the URI in a form like below:

{
    'storage:[email protected]:3301',
    params = {
        transport = 'ssl',
        ssl_cert_file = cert_file,
        ssl_key_file = key_file,
    }
}

instead of just a string. But that is not a problem really - vshard can allow to specify URI as a table. The actual issue is that the URI needs to be different for listening and for connecting. For the storage and for those who connect to it. Server needs cert and key files. The client either needs nothing or needs CA cert to validate the server. Or the server might also need a CA cert to validate the client. Or any other combination is possible.

Example. I want vshard.storage.cfg() on storage_1_a to make:

box.cfg{
    listen = {
        'storage:[email protected]:3301',
        params = {
            transport = 'ssl',
            ssl_cert_file = cert_file,
            ssl_key_file = key_file,
        }
    }
}

while vshard.router.cfg() on any router and vshard.storage.cfg() on other storages should do this to connect to storage_1_a:

conn = netbox.connect({
    'storage:[email protected]:3301',
    params = {
        transport = 'ssl',
        ssl_ca_file = ca_file,
    }
})

Different parameters for the same storage depending on where the config is applied.

How it should work

Need a way to specify different options for the same URI to use for listening and for connecting.

API and behaviour

Option with clean config, but config can't be same everywhere

The most clean way is not do anything with the config except relax a bit the rules for uri parameter. Currently it can only be a string. Now it will be allowed to use a string, a number (port), a table. All is additionally validated to be parse-able by parse() function from uri built-in module.

To give special parameters for listening to each storage the user will need to specify the listen argument in the root of the config on this specific storage. It worked before as well, for any box.cfg-parameter, not just for listen. But for SSL it will be the only way to set it up.

For example, if you want SSL, then all routers will have to use this config:

{
    sharding = {
        ['storage_1_uuid'] = {
            replicas = {
                ['storage_1_a_uuid'] = {
                    uri = {'storage:[email protected]:3301',
                        params = {transport = 'ssl', ssl_ca_file = ca_file}
                    },
                    name = 'storage_1_a',
                },
                ['storage_1_b_uuid'] = {
                    uri = {'storage:[email protected]:3302',
                        params = {transport = 'ssl', ssl_ca_file = ca_file}
                    },
                    name = 'storage_1_b',
                },
            },
        },
    },
}

storage_1_a will have to use this config:

{
    sharding = {...}, -- Same.
    listen = {
        'storage:[email protected]:3301',
        params = {
            transport = 'ssl',
            ssl_cert_file = cert_1_a_file,
            ssl_key_file = key_1_a_file,
        }
    }
}

storage_1_b will have to use this config:

{
    sharding = {...}, -- Same.
    listen = {
        'storage:[email protected]:3302',
        params = {
            transport = 'ssl',
            ssl_cert_file = cert_1_b_file,
            ssl_key_file = key_1_b_file,
        }
    }
}

As you can see, with SSL it won't be possible to have a single config for all instances in the cluster.

Here routers have no listen option at all. storage_1_a has listen with own URI and has its own cert and key signed by a CA. storage_1_b has other listen URI and has its own different cert and key signed by the same CA.

This is fully backward compatible. User will need to specify listen explicitly on each storage only if he wants to enable SSL properly.

Option with config being same everywhere

This option allows to continue having the same config on all instances in the cluster. But the config becomes notably bigger if one wants to use SSL.

The idea is to allow to specify listen in the sharding section of the config, like below:

{
    sharding = {
        ['storage_1_uuid'] = {
            replicas = {
                ['storage_1_a_uuid'] = {
                    uri = {
                        'storage:[email protected]:3301',
                        params = {
                            transport = 'plain',
                            ssl_ca_file = ca_file,
                        }
                    },
                    listen = {
                        'storage:[email protected]:3301',
                        params = {
                            transport = 'plain',
                            ssl_cert_file = cert_1_a_file,
                            ssl_key_file = key_1_a_file,
                        }
                    },
                    name = 'storage_1_a',
                },
                ['storage_1_b_uuid'] = {
                    uri = {
                        'storage:[email protected]:3302',
                        params = {
                            transport = 'plain',
                            ssl_ca_file = ca_file,
                        }
                    },
                    listen = {
                        'storage:[email protected]:3302',
                        params = {
                            transport = 'plain',
                            ssl_cert_file = cert_1_b_file,
                            ssl_key_file = key_1_b_file,
                        }
                    },
                    name = 'storage_1_b',
                },
            },
        },
    },
}

The same config can be applied on all routers and all storages. The storages will extract the listen parameter related to them and use it for box.cfg{listen=...}. For connecting to other instances a storage will use uri parameters from the config. Listen is optional. When it is not specified, it is = option uri.

This is fully backward compatible. The bad side is that the config gets notably bigger when user wants SSL.

Another bad side is that if eventually it will be allowed to set key password in the URI options as open text, then it is probably not a good idea to have this config cluster-wide. But I don't know.

The main question

The question of the RFC is whether the Option 2 is needed at all. Option 1 will work regardless of what is decided here. It always worked, like I said. Any box.cfg option even now can be specified in the root of the config, next to sharding, and the instance will apply it above everything else. It allows, for instance, to override on the storages box.cfg.replication and even now allows to override box.cfg.listen.

FAQ

Non yet

Alternatives

None yet

[Gerold103]

@olegrok, @R-omk. Don't know who else to summon.

[Totktonada]

I would think into the 'same config on all instances' direction. Different configs will make things complex to insane.

It would be good if tarantool's URI format would support listening options and connecting options as part of one table, so a code can use one subset to listen and another subset to connect.

The problem, as I see, is that if we set ssl_cert_file and ssl_key_file on the server, the client will use it for the client certificate and vice versa: if the client sets ssl_ca_file, the server will authorize clients using this CA certificate.

So the question is whether we really want to issue per-instance certificates or a shared secret in a cluster is completely okay. I can't answer with a good level of confidence.

However I think that generation of all those certificates (especially if we'll going to support some kind of autoscaling) with something like --auto-tls from etcd is so cumbersome that a usual setup will happy with sharded secret.

[olegrok]

The same config is a good idea. But seems it's not anymore relevant. Not only in SSL scope. Currently it's possible to specify several listen uri (tarantool/tarantool@531e229).

Potentially I should be able to:

• Specify several listening sockets for storage
• Connect to only one socket from router (there is no guarantee that sockets will be the same for all routers in cluster. E.g. if storage and router live on single host it's reasonable to use unix sockets).

Probably if we could specify some instance-related options we can do it with env variables. Currently it's possible to specify TT_LISTEN as uri. I don't know is it possible to specify ssl options in the same way.

[Gerold103]

The multilisten is actually a good point. It means we already have a case when listen != connect. At least because listen can be an array of URIs. However with my same-config approach it will work anyway. You just specify replica.uri = <how to connect> and replica.listen = <array of uris>. Inside of the sharding config section. It will get even bigger though if listens here will be arrays.

As for the per-instance certificate - why not? If they are all signed with the same CA, then the clients will be able to validate all the servers with just one cert. Anyway, it does not matter much. In this RFC it isn't relevant whether servers have same certs or not. Only relevant part is that listen != connect. As Oleg noticed, listen even be multiple URIs, not just one.

[olegrok]

Moreover "connect" to storage could be different for different instances.
If router and storage live on single machine should be possible to connect them using unix sockets.