-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
blog: drafts smart testing article. #424
base: develop
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
--- | ||
layout: post | ||
title: Smart Testing For Fast Feedback | ||
author: hemanik | ||
tags: [ testing, guide, smart-testing ] | ||
--- | ||
|
||
h1. Smart Testing For Fast Feedback | ||
|
||
p<>. ??As an engineer, you should constantly work to make your feedback loops shorter in time and/or wider in scope.?? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. for quotations block use |
||
|
||
p>. ??"-@KentBeck":https://twitter.com/KentBeck/status/531964254946328576?? | ||
|
||
|
||
One of the biggest issues that plague the software development life cycle is the slow feedback loop, the time between changing code (or tests) and getting results from the tests. | ||
|
||
h2. Impact of Slow Feedback Loop | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This whole section is very long. I would split it into smaller parts - subsections for example: "delays", "context switching", "resources" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also, the name of this section could be just: "Motivation" |
||
|
||
Consider you are a team doing "Test Driven Development(TDD)":https://en.wikipedia.org/wiki/Test-driven_development, then the general workflow is you write some tests, add code to make those tests pass and run those tests to see if the tests pass and refactor if the tests fail and repeat the process. | ||
|
||
This vicious cycle of programming and testing doesn’t just slow you down in terms of time spent (especially with long-running tests), the involved "context switching":https://www.petrikainulainen.net/software-development/processes/the-cost-of-context-switching/ also drains your mental energy, which might ultimately destroy your productivity. | ||
|
||
Here, fast feedback means that the time between changing code (or tests) and getting results from running the tests is reduced to a minimum. In other words, you end up with a fast edit-compile-test loop. The great thing about this, as "Joel Spolsky put it":http://www.joelonsoftware.com/articles/fog0000000023.html: “the faster the Edit-Compile-Test loop, the more productive you will be”. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would change the word |
||
|
||
For almost every new feature added to the application, there is also a test written. This way, whenever someone changes the user interface, the system will be able to catch the differences and signal the team about any issues. They're still not integrated into the main pipeline. The plan is to merge them in to gain more confidence in what they build. The tests should run whenever anyone pushes changes to the git repository. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure what is meant by this paragraph. It doesn't seem to fit here. |
||
|
||
Work starts on the new CD pipeline and everything is set up accordingly. The pipeline structure looks like below: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm missing the purpose of the next lines - it should be a description of the developer's workflow? I'm not sure if it is really necessary to describe the whole process here, just highlight the most important things... |
||
|
||
code quality → unit tests → build → staging deployment → e2e test | ||
|
||
For every developer pushing code to the repo, the system kicks off a process and runs the whole pipeline. | ||
|
||
Running it at every code push is costly, not only in terms of resource utilization (machines, networks, servers) but also in terms of developer time. Whenever the build breaks, all work should stop and the build should be fixed before introducing more features (and possibly more problems). The developer has to wait for the build to pass in order to move on. | ||
|
||
Imagine there are 2 or more developers working on that branch. They would have to push to version control only when they are sure the functionality is 100% ready. Otherwise, we would break the application for every person working on the project. This leads to more problems. | ||
|
||
Putting it all together it is very important to know that we should refrain from optimizing for the best case. It is always better to optimize for failure, accept it, embrace it. | ||
|
||
For teams doing some form of Continuous Integration/Delivery/Deployment, if the pipeline is slow, then the feedback loop is slow. If the code takes 30 minutes to pass through the pipeline and build only to fail in UAT or even worse, production, precious time and money are wasted. | ||
|
||
Feedback is very important but if it comes slow, or after you pushed to production, it has no preventative value. It only serves as a lesson learned. Having feedback from the early beginnings of a project, and receiving it fast, is the cornerstone to Continuous Delivery / Deployment. It allows you to optimize ahead of time, with little to no impact on your users. | ||
|
||
Hence, going fast is not only about shipping fast but also about failing fast. The faster you fail, the cheaper the solution. | ||
|
||
h2. How does Smart Testing shorten the Feedback Loop? | ||
|
||
There are a couple things you can do to shorten the feedback loop. You should absolutely invest in your tests and make them faster. Besides trying to implement faster tests, however, you can also optimize the way you run them. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would keep only the first sentence:
then describe the problems of the approaches - not running the tests is not a good idea; making test faster is hard (as you wrote there); parallelization is great, but you can do it to some limit; buying more resources - the limit is the budget. So we have prioritization and minimization of the test suite, but how to do it? Which tests should we choose? and then you can jump to the options/strategies etc... |
||
|
||
At first glance, the way you go about running tests might not seem to have a big impact on the edit-compile-test loop. If a test takes a minute to finish, does it really matter if we can shave off a second or two by tweaking the running step? Yes, it does. Seconds add up over time. Each additional step requires a little more brain power and incurs a significant context-switching cost. | ||
|
||
"Smart Testing":http://arquillian.org/smart-testing/ aids you in optimizing the way you run tests. | ||
|
||
During development, don’t run the entire test suite each time you change a bit of code. Aside from the fact that running all tests is often too slow, it’s always better (and faster!) to first get feedback on local code changes before integrating with other code. Reducing the scope and testing a small subset of code in isolation is not only faster, it also helps you find bugs, and it’s a must-have for TDD. (It goes without saying that at some point you or your continuous integration system should run all the tests.) | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health. | ||
|
||
If slow feedback loop is what is hindering your productivity, add Smart Testing to your toolbox and give this awesome tool a try. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't that be
Smart Testing For Fast Feedback Loop
or maybe justSmart Testing
?