-
Notifications
You must be signed in to change notification settings - Fork 16
Testing Framework Overview
This article pertains to the ongoing effort to produce a framework for testing appliances (and producing screenshots along the way).
Before testing, the following things have to be observed:
Cloning the appliance code
Should be pretty self-explanatory. It is better to perform shallow clones (depth = 1) for performance reasons. There are ways to get the full list of turnkeylinux-apps repos from GitHub but optimally we'd need to either make a separate list or even better, clean up the obsolete ones to avoid bogus test failures for appliances no longer supported.
Building the appliance
At the moment, this can be done through make CHROOT_ONLY=y, however when containerisation (and, possibly, fab-buildtasks merge) comes to TKLDev that would be much better from a separation POV. Docker is also an option.
Launching all services that would start during normal operation
At the moment, we can do it through executing /etc/rc[23].d/S*. But that is hacky compared to a containerised «real» init run.
Also, have to avoid running confconsole in chroot — it will break the TTY.
To fully initialize the appliance and allow testing password-restricted areas. This is pretty simple.
Detecting services
This part is kludgy.
What I've chosen to do when I wrote this was to parse the appliance's confconsole file to detect user-advertised services automatically (so I don't end up having to describe which services each appliance has manually). After parsing the names are used to look up appropriate CasperJS scenario files for each case. This is not particularly reliable since there is no predefined format for confconsole service description files. But it (kind of) works and it would make a lot of sense for this to remain an automatic step in the future (one test case for all appliances with the corresponding application installed).
CasperJS is called to perform the actual testing (= browsing through the exposed website checking if the needed elements are present). CasperJS has its own test framework that we are going to use here. In principle, Selenium could be used to do the same but obviously the code would not be compatible.
Screenshots are also taken in the process. Viewport size can be adjusted to simulate different devices' screen resolutions (right now the default size for TKL screenshots is 640x480, hosting space/bandwidth are relevant here). With some websites, there are issues with performing web authentication through CasperJS but so far I have not encountered any that would not work at all.
Also, there are two (perhaps more) factors which will lead to the test run time becoming more and more important over time:
- The expansion of the TurnKey Appliance Library (which already includes more than 100 products).
- The potential benefits of integrating the testing solution with our GitHub workflow á la CI (testing if an appliance builds and works as before after merging a contributed PR).
Thus it will also make a lot of sense to eventually implement parallel testing, which is probably better handled up the stack (e. g. by Ansible running tasks in parallel rather than coding it into the CasperJS code from scratch), with granularity being website rather than appliance. Since we are not modifying anything, there should be no concurrency issues.
Everything has to be terminated gracefully and the screenshots (if taken) are to be uploaded… somewhere. Probably pushed to the repo, too. When this (or not exactly this) will be used to produce a batch of screenshots in the future, a lot of pull requests will have to be generated automatically through the hub CLI tool (it adds e. g. git pull-request).
Being able to perform a test run as quickly as possible is nice (not just for releases, but it would help immensely during appliance development). However for that we would also need to look into making fab faster and incorporating at least some parts of buildtasks there. There is no precise roadmap for that right now but it is a thing.