How to implement a master-slave service

[sepcity]

My requirement is to have only one endpoint for external service, but also have high availability

My current approach is create an application with replica group(using ordered load balancing type) of two nodes(with different priority)

Under normal circumstances, it works right and all clients connect to master services(with minimal priority)

But when i closed the master services, clients will connect to slave service. And then i startup the master service, clients still connected with the slave service, but new client will connect to master service, this is not what I expected.

I Already set "Ice.Default.LocatorCacheTimeout" and "Ice.BackgroundLocatorCacheUpdates", but it has no effect.

In my case, the client keeps sending requests to the service, and I don't think the connection will be disconnected in this situation

what can i do?

[pepone]

I Already set "Ice.Default.LocatorCacheTimeout" and "Ice.BackgroundLocatorCacheUpdates", but it has no effect.

As you've noticed, these settings do not affect existing connections. If the clients are actively using a connection, it will remain open, and the clients will not switch back to the master service.

If a connection becomes idle, ACM will close it. When the clients reconnect, they will resolve the endpoint from the locator service again.

What is your desired behavior? Do you want all clients to send requests to the master once it's back online?

[sepcity]

Yes, I do want all clients connect to the master node once it's online

[pepone]

If you disable the proxy connection caching, then the proxy will not keep the existing connection for subsequent requests at the expense of repeating the connection establishment process for each request.

Please refer to the Ice documentation for more details on connection establishment: Connection Establishment in Ice

[sepcity]

I have read this article, but it seems that I cannot directly implement my requirements(all clients send requests to the master once it's back online)

auto locatorPrx = Ice::uncheckedCast<Ice::LocatorPrx>(ic->stringToProxy("IceGrid/Locator"));
auto adapter = locatorPrx->findAdapterById("my_replica_group");
I have an idea. I manually check the adapter information(adapter->ice_toString()) on the client at regular intervals, and if there are any changes, I will manually close the existing connection(how?) and initiate the establishment of a new connection

What do you think, or is there a better approach

[pepone]

Did you read the section about connection caching? You have to disable the proxy connection cache using ice_connectionCached then the new request will not use cached connections. See Connection Caching section in the linked doc.

[sepcity]

The new proxy returned by ice_connectionCached repeats the connection-establishment procedure before each request

This behavior is not what I expected just like it says " the expense of potentially higher latency"。

In fact, what I mentioned earlier was not rigorous. Once the master service is online, all clients need to be able to connect to the master service as soon as possible within an acceptable time.

[pepone]

Disabling the connection cache seems like the simpler solution in this context. As to whether the impact of the connection-establishment procedure will be noticeable for your use case, that's something you'll need to measure.

An alternative would be to gracefully shut down the Slave once the Master is back online. However, this introduces a new problem: the moment you take the Slave offline, you eliminate the possibility of a failover, and if something goes wrong with the Master, the requests would fail.

To mitigate this, you could start a 'New Slave' before shutting down the old one, but this adds more complexity to the setup. Without knowing the specifics of your system, it is difficult to advise on the best approach. Generally, it is best to begin with the simplest possible solution.

[sepcity]

In my case, availability is more important than consistency, and each connection establishment may encounter unexpected connection exceptions and takes higher latency.

Is there any mechanism for gracefully shut down a service in the world of ICE, or notification mechanism when serivce online?

[pepone]

Is there any mechanism for gracefully shut down a service in the world of ICE, or notification mechanism when serivce online?

If IceGrid manages your server you can use the IceGrid Admin object to manage your servers, there are a number of observers that you can implement and register to get notified about events with the deployment. The IceGrid GUI application uses these interfaces, to manage the IceGrid deployments.