Troubleshooting

Cronicle is very sensitive about server hostnames, and uses them to determine who is a member of the cluster, and who can and should become a master server. If server's hostname has changed, then Cronicle can no longer find it in one of its internal lists.

Here is how to fix this. First, stop Cronicle on all your servers, SSH to your master server, become root, and run this command:

/opt/cronicle/bin/storage-cli.js get global/servers/0

This will dump out a JSON document listing all the servers Cronicle knows about. For example, here is what it looks like on my test cluster:

{
	"type": "list_page",
	"items": [
		{
			"hostname": "dev01.dev.ca.test.net",
			"ip": "192.168.66.167"
		},
		{
			"hostname": "dev02.dev.ca.test.net",
			"ip": "192.168.66.199"
		}
	]
}

What you need to do is figure out exactly what Node.js thinks your current server hostname is, find the old server in that JSON document, and correct the hostname (and possibly the IP address too).

To determine what Node.js thinks your current server hostname is, type this command:

node -e 'console.log(require("os").hostname().toLowerCase());'

Do you see that hostname in the JSON document above? If so, then skip down to the next section. If not, then read on.

To correct the server hostname, the Cronicle storage-cli.js script can spawn your local text editor (usually vi, whatever is in your EDITOR environment variable) and allow you to edit the storage record right in your Terminal. Please be very careful doing this, as you can permanently damage Cronicle. Here's the command:

/opt/cronicle/bin/storage-cli.js edit global/servers/0

Once you have corrected the hostname, save and exit your editor, and the JSON document will be rewritten back to storage (either local disk, AWS S3 or Couchbase, whatever you have configured).

Please Note: It is very important that you only correct the hostname (and maybe the IP) of the server in the list. Do not remove or add any servers, as the number of items in the array is stored in a separate document. Make sure the hostname is lower-case!

Now try starting cronicle, waiting 60 seconds, and seeing if the Web UI is happy again:

/opt/cronicle/bin/control.sh start

Please note that it takes a full minute (60 seconds) for Cronicle to decide to become master, so please be patient here.

Did that do the trick? If not, keep reading...

Master Server Groups

So, your server hostname is now correct and in the JSON document, but the Web UI is still stuck on "Waiting for master server"? The reason is probably that your server's new hostname is no longer part of a master server group.

Cronicle has a server group system, and only certain groups are eligible for becoming master. It categorizes servers into groups by matching their hostname against a regular expression pattern. It is possible that when your server hostname changed, it no longer matches the correct master group pattern.

To dump out the current server group definitions, stop Cronicle on all your servers, SSH to your master server, become root, and run this command:

/opt/cronicle/bin/storage-cli.js get global/server_groups/0

This should output a JSON document that looks somewhat like this:

{
	"type": "list_page",
	"items": [
		{
			"id": "allgrp",
			"title": "All Servers",
			"regexp": ".+",
			"master": 0
		},
		{
			"id": "mastergrp",
			"title": "Master Group",
			"regexp": "(dev01|dev02|devrelease|mtx\\d+)\\.",
			"master": 1
		}
	]
}

As you can see, each group has a regexp which is a regular expression match on the hostname, and a master attribute that determines if the servers in the group can become master.

You may have to adjust the regexp string in your master group, so that your server's new hostname matches the pattern.

To correct the server groups, the Cronicle storage-cli.js script can spawn your local text editor (usually vi, whatever is in your EDITOR environment variable) and allow you to edit the storage record right in your Terminal. Please be very careful doing this, as you can permanently damage Cronicle. Here's the command:

/opt/cronicle/bin/storage-cli.js edit global/server_groups/0

Once you have corrected the regexp patterns, save and exit your editor, and the JSON document will be rewritten back to storage (either local disk, AWS S3 or Couchbase, whatever you have configured).

Please Note: It is very important that you only correct the regexp of the server group in the list. Do not remove or add any groups, as the number of items in the array is stored in a separate document.

Now try starting cronicle, waiting 60 seconds, and seeing if the Web UI is happy again:

/opt/cronicle/bin/control.sh start

Please note that it takes a full minute (60 seconds) for Cronicle to decide to become master, so please be patient here.