A guide to smoke testing

I couldn’t believe it happened again!

We released a new feature and, soon after, the core functionality stopped working! To make matters worse, I wasn’t aware of the issue until I received a phone call from a disgruntled customer.

At this point in my PM career, I found myself anxious any time we went to release a product because doubts would creep into my head that something would go wrong. When issues arose and we had to revert to second releases, friction grew among our customers. Trust began to slip because our new features would crash old ones.

Our testing approach put us in that position.

We had to act swiftly to fix our approach. One thing that helped us rectify the situation was smoke testing. In this article, you’ll learn what smoke testing is and how you can adapt it immediately to avoid painful situations.

What is smoke testing?

Smoke testing is a simple process to ensure the core part of your solution works smoothly when you add new ones. It means that new features don’t affect your application.

Although it may seem obvious that new features shouldn’t negatively impact old ones, software development is complex. Unexpected effects will happen as the code base grows.

Smoke testing vs. other testing methods

Quality assurance is fundamental in software development. Without that, you’ll continuously receive unpleasant surprises.

I like how a friend of mine, Julio de Lima, explains testing in software development:

Imagine a forest. It has a healthy ecosystem, but what would happen when you plant a different tree there? You shouldn’t be surprised when other trees start dying. You need to see the big picture to ensure the new doesn’t break the old.

Smoke testing is one method, but there are others as well. Some of these include:

• Sanity testing — This method ensures the functionality works as expected in multiple scenarios. It focuses solely on the change applied and expected results
• Regression testing — Here, you aim to ensure the application remains healthy with the new changes. While smoke testing targets the application core, regression testing looks beyond that
• User testing — In this case, you evaluate how users react to the new changes. Does that help them achieve the results as expected?

Key components of smoke testing

Returning to the example I started with, we continuously faced painful situations because our application broke after releases. Smoke testing helped tremendously with this. To set it up there are three key components:

1. Core — Define the core components of your application. That’s what smoke testing will cover. Think about the features that, when broken, fail to deliver your value proposition
2. Criteria — Make the expected behavior clear for each core component part. You’ll need to define use cases and desired results
3. Execution — You can automate smoke testing to reduce manual activities. For example, smoke testing could be a step before releasing it to production; if that fails, the build isn’t concluded

This exercise is best performed with the whole product team. You’ll benefit from multiple perspectives, product management, software engineering, and user experience.

How to perform smoke testing

I’d encourage you to aim for fully automated smoke testing that’ll increase your product quality. However, you shouldn’t start automating before you clearly understand what should and shouldn’t be a part of the smoke testing.

In my case, I got together with the team and we defined what we couldn’t afford to break. When we released something to our pre-production environment, we ran through our list and tested the core features; whenever we found an issue, software engineers would correct it, and we’d test again.

We learned a lot on the way and noticed that some of our tests were redundant and missed others. In short, it took us a few cycles until we got it rounded.

After manual smoke testing is rounded enough to prevent you from breaking the core of your application, it’s time to automate. If you work with continuous integration, make the smoke testing part of your build process. That’s my favorite approach, as it ensures what gets deployed doesn’t break your core.

If you lack continuous integration, get together with your engineering team and figure out how to automate smoke testing.

Smoke testing in different contexts

One of the biggest challenges of digital products is that situations are different. What works in one place will not necessarily work in another place. Yet, that’s not an excuse to skip smoke testing.

Keeping the core of your product working is paramount. Don’t compromise there.

Throughout my journey, I learned the following:

• Stick with principles — Value proposition is the key to your business; ensure it works
• Collaborate — Get together with your team and ensure that you figure out how to keep your application running sustainably
• Staging — Whenever possible, create a staging environment reflecting your production space. Aim for a reasonable data sample if you cannot have a complete replica. It’s better to run smoke testing with accurate data

Another aspect can be when a vendor takes care of software development for you. Collaborate with them to ensure smoke testing takes place and be involved as much as possible.

Challenges of smoke testing

The more rigid your testing processes are, the less output you create. This isn’t something all stakeholders will understand easily.

Let me share some key points.

I don’t encourage making sure that every solution is perfect from the beginning. You’ll learn that some solutions don’t create the desired results, so try to drop them as fast as possible.

My recommendation is first to build to learn, then to scale. This requires deliberate and prudent use of tech debt. Yet, whatever you add to the product, your core value proposition shouldn’t suffer.

Key takeaways

Smoke testing ensures that the core of your application works as expected after adding changes or new features. Start by manually learning what to cover, then automate it once you get a solid script.

Skipping smoke testing will inevitably lead to undesired and costly results. Strive to have a minimum level of quality assurance, but don’t compromise on guaranteeing your value proposition is delivered.

Featured image source: IconScout