RSS Feed

What is User Story and Acceptance Criteria (Examples)

A Perfect Guide to User Story Acceptance Criteria with real-life scenarios:

In the Software Development industry, the word ‘Requirement’ defines what our goal is, what the customers exactly need and what will make our company to increase its business.

Be it a product company which makes software products or a service company which offers services in various software fields, the prime base for all of them is the requirement and the success is defined by how well the requirements are met.

The term ‘requirement’ has different names in different project methodologies.

In Waterfall, it is referred to as ‘Requirement/Specification Document’, in Agile or SCRUM it is referred to as ‘Epic’, ‘User Story’.

Under Waterfall model, the Requirement documents are huge docs of 200 or more pages as the whole product is implemented in one phase. But this is not the case with Agile/SCRUM because in these methodologies the requirements are given for small functionalities or features as the product is prepared in a step by step manner.

What is a User Story?

A user story is a requirement for any functionality or feature which is written down in one or two lines and max up to 5 lines. A user story is usually the simplest possible requirement and is about one and only one functionality (or one feature).

The most commonly used standard format for a User Story creation is stated below:

As a <user role/customer, I want to < goal to be accomplished> so that I can <reason of the goal>.

Example:

As a WhatsApp user, I want a camera icon in the chat write box to capture and send pictures so that I can click and share my pictures simultaneously with all my friends.

User Story and Acceptance Criterion

What is an Acceptance Criteria?

An acceptance criterion is a set of accepted conditions or business rules which the functionality or feature should satisfy and meet, in order to be accepted by the Product Owner/Stakeholders.

This is a very important part of user story completion and it should be studied by the Product Owner and Business Analyst very meticulously because missing a single criterion can cost a lot. This is a simple numbered or bulleted list.

Its format is as follows:

Given some precondition when I do some action then I expect the result”.

Example (w.r.t to above user story):

  • Let’s consider that I’m chatting with a friend and I should be able to capture a picture.
  • When I click on a picture, I should be able to add a caption to the image before sending it.
  • If there is some problem with starting my phone camera, an error message like ‘Camera could not be started’. etc., should be shown accordingly.

Hence, the User story defines the requirement for any functionality or feature while the Acceptance Criteria defines the ‘Definition of done’ for the user story or the requirement.

As a QA it is very important to understand the user story and its acceptance criteria profoundly with not even a single doubt remaining at the ‘start of testing’. Moving forward let’s understand why it is extremely important to dig ‘deep’ in user stories and acceptance criteria.

Digging deep into User stories

To start with, let us first understand the importance of an ‘in-depth’ study of a basic and fundamental thing i.e. User Stories.

The following cases are my own real experiences.

Case #1:

Before 3 years, I was working on a Mobile Application Project and the product was an application that was designed for the delivery people.

You would have seen a delivery person coming to your place for delivery. And they have a mobile phone on which they ask you to give your signature after delivery. This signature reflects on the portal of the courier service providers like DTDC, FedEx etc.

Let’s imagine that the mobile app is just launched and their portals are already existing and up.

Problem: For a Sprint your Product owner has a user story for this mobile app that “As a Portal Admin, I should be able to view the signature taken by the delivery person at the time of delivery”. Here the portal (web app) is changed and updated accordingly to reflect the signature.

As a QA you have to verify if the signature captured in the mobile app is reflecting as expected in the portal.

If you look at this user story, it looks simple but there is a hidden requirement here that “For historical deliveries, there was no signature reflection functionality, so what should happen if the portal guys view historical deliveries?” Should historical data be wiped out? Should we allow crashes or errors for such data?

Of course not at all, this should be handled graciously.

Solution: When the respective DB tables are updated to add a new column for the Signature location, the old data should have a NULL or 0 value which should be checked and a message stating ‘No signature exists’ should be shown.

This can be called as a miss from the Product Owner or Business Analyst, but this has to be done. Implementing one feature successfully but breaking something along with it is not desirable by the customers. This needs to be done along with the same user story and in the same sprint.

Case #2

6 years ago, I was working on a Retirement Planning Finance Application (with no BA) which was a global application where Finance folks like CA, Finance Advisors could use it for different currencies to project the investment plans, savings, etc., over a large period to their customers.

Problem: The Product Owner gives you a User Story that “As an Advisor, I want to view the report of my customer based on the financial details provided”.Here there were 2 hidden requirements and I would call it as an incomplete story because:

a] The reports should consider the daily currency conversion rate and not the historical one as in the last viewed report and

b] If the currency is changed after providing the customer’s financial details, the reports should show in the changed currency.

Solution: I raised this concern directly with our Product Owner and made him aware that both of these had to be done as soon as possible. He agreed with me and created 2 different stories for the upcoming sprints with priority.

Take Away: These were caught because we all were very well aware of the products, their design, structure etc. Such knowledge can only be achieved by understanding the product completely, by understanding the inter-operability of modules and by studying the user story thoroughly even if it’s a 2 liner.

Make notes to make things easier and discuss with the BA’s and the developers about their thinking.

In-Depth look at Acceptance Criteria

Understanding the acceptance criteria and all the other conditions& rules exhaustively is even more important than understating a user story. Because if a requirement is incomplete or vague, it can be taken up in the next sprint but if an acceptance criterion is missed, then the user story itself can’t be released.

I guess we all would have used net banking at some point and most of us use it every day and I download my historical statements a lot. If you observe it carefully, there are certain specific options available for downloading them.

There is an option to select the type of file for downloading your statement. There is an option to choose if you want to download only the Credits/Debit /both.

Now imagine that the Product Owner gives you this User story “As a customer, I want to download my account statement so that I can view all my transactions done for a specific period”.

With the following Acceptance Criteria:

  • Considering that I am on the Download Historical Statement Page, I should select the period for which I want to download the statement.
  • Considering that I am on the Download Historical Statement Page, I should select the account for which I want to download the statement.
  • Considering that I am on the Download Historical Statement Page, I should not be allowed to download the statement for future ‘To’ date.
  • Considering that I am on the Download Historical Statement Page, I should not be allowed to select ‘From’ date 10 years beyond in the past.
  • Considering that I download my statement, I should be able to view the downloaded file.
  • Considering that I am on the Download Historical Statement Page, I should be able to download my statement in doc, excel and pdf formats.

If you go through this acceptance, there are 3 things missing here:

  • Name and format of the file name that will be downloaded.
  • What information (Column names) is to be displayed in the file.
  • The options list to select what kind of a transaction the customer wants i.e. only debits or only credits or both.

Such cases may happen once in a while, however still study well about each acceptance criteria and try to visualize it with reference to the user story. The more you study deeply about the conditions and business rules the more will be your knowledge about the feature.

Bugs found in the initial stage cost nothing compared to what it may cost in the ‘testing’ stage.

Acceptance Criteria

Importance of finding Discrepancies in User Story/Acceptance Criteria

It is always important to do a deep dive in the user stories and acceptance criteria at an early stage even before the development or testing commences.

Because it involves:

#1) Wastage of Time:

If the discrepancies or mistakes in the user story/acceptance criteria are found when development is going on or testing is going on, then a lot of rework may need to be done in the remaining sprint time.

It doesn’t happen that even if the Product Owner missed few things, they will move the user story to the coming sprint. 95% chances are that they ask the team to do the necessary implementation and release it in the same sprint.

Hence it becomes a nightmare for the team as they have to spend extra time, come on weekends or work late night. This can be avoided by studying and discussing the user story/acceptance criteria at the earliest possible stage.

#2) Wastage of Efforts:

The developers and QA have to revisit the implemented code and test cases again. Updating, adding and removing as the per requirement is not an easy task. It becomes too painful as there is already a pressure to deliver on time.

In such a situation, there are chances of mistakes in the development or testing stage. If you come across such situation go for ‘DevQA Pairing’. As an icing on the cake, you may not get a compensation for the extra work.

User Story and Acceptance Criteria discrepancies

[image source: media.licdn]

Conclusion

Deep understanding of User Story and acceptance criteria can only be achieved by spending immense time on studying it.

There is no specific tool or course available in the market to do this for you as this is all about logical thinking, experience, and knowledge about the product.

Participating in Pre-plan meeting actively, talking to the BA, studying on your own can only help you to achieve this. The more efforts you put, the more you learn and grow.

Be it the QA’s or developers, everybody has to be on the same page about the user stories and their acceptance criteria, only then the expectations of the customer can be achieved successfully.

Advertisements

Practical Software Testing QA Process Flow

A Complete Overview of End-to-End QA Software Testing Process Flow:

Note – We are re-publishing this useful post with updated content.

The job of a software testing professional is not an easy one. It is filled with challenges, which is equally demanding as well.  Testers are supposed to be alert and enthusiastic in each and every phases of the application life-cycle .

Though there are challenges, there are several tremendous opportunities as well to learn and explore the various aspects of testing methodologies, processes and of course the software in detail.

The role of a test engineer begins very early. And right from the conceptualization of the project, testers are involved in discussions with the product owner, project manager and various stakeholders.

#1) Requirement

A project cannot take off without having a clear requirement. This is the most crucial phase where ideas need to get written in a well understandable and formatted document. If you are a part of the project where you are participating in the requirement gathering phase, then consider yourself lucky.

Wondering Why? It is because you are witnessing a project in making from the scratch. Though there is pride in being since inception, it comes with some responsibilities and challenges too.

Challenges

One cannot imagine all the requirements to gather in a single sitting. Be patient enough.

A lot of discussions will happen, some of which may be simply irrelevant to your project but even then they may contain some vital information for your project. Sometimes the speed of discussions may exceed your grasping capabilities or you would simply not pay attention to the product owner.

Every piece of information that is missed has a huge impact on the overall understanding and testing of the project. To overcome this, here are some best practices which should be followed during this phase.

Best Practices

  • Keep your mind open and pay attention to every word of the product owner.
  • Don’t just listen, clear your doubt however small it seems to be.
  • Always use notebooks for fast note keeping. You should use a laptop, only if you can really type at a fair speed.
  • Repeat the sentences and get them clarified from the PO which you think is what you understood.
  • Draw block diagrams, link text etc. to make requirements more clear at a later period of time.
  • If the teams are in different locations, try hard making it a WebEx or similar recording. It will always help when you have a doubt after the discussions are over.

Though there is no separate wall as such for each and every phase, requirements do change even very late in developments. So, the idea is to grab the most of the requirement and document this properly.

Once it is been documented with all the necessary points, distribute and discuss this all the stakeholders so that any suggestions or changes are caught early and before moving on, everyone is on the same page.

#2) Test Strategy

Testers are supposed to come out with a test strategy which is not just sufficient to test the software better, but should also instill confidence in every stakeholder regarding the quality of the product.

The most crucial aspect of this phase is to create a strategy which when worked upon should deliver a software product that is error free, sustainable and accepted by its end users.

Test strategies are something you will not change every other day.  In some cases, you need to discuss your test strategies with the customers also. So this part should be dealt with high importance.

Best practices

  • Here are some of the best practices, which when followed can provide you a great relief and result in a smooth testing.
  • Go through the requirement document once again. Highlight the import points with respect to the environment of the target software.
  • Make a list of environments where the software will actually be deployed.
  • Environments can be understood as a type of Operating Systems or Mobile Devices.
  • If Windows is the operating system, list down all the versions of the window where you will be testing your software. If the versions viz. Windows 7, Windows 10 or Windows Server(s) are still not defined in the requirement document, then it is the high time when these should be discussed.
  • On a similar note, get the target browsers with the versions discussed and documented if the AUT is a web-based system.
  • Make a list of all the third party software’s that the application will need (If required/supported). These may include Adobe Acrobat, Microsoft Office, any add-ons etc.

Here the idea behind is to keep every necessary and required platform, devices, and software’s that our application will need to function before us so that a comprehensive strategy can be formulated.

#3) Test Planning

After the testers are armed with all the information regarding AUT, the planning phase is where the Strategy is implemented.

Like test strategy, test planning is also a crucial phase.

Challenges

Since the success (or failure) of the AUT depends largely on how the tests were carried out, this phase becomes an important aspect of the entire test life cycle. Why? Because a part of testing is defined in this phase.

In order to overcome some challenges, these best practices can really be helpful.

Best practices

  • Always keep in mind, not to leave any stone unturned when it comes to testing your application.
  • It’s time to formulate test strategy.
  • Create a matrix of the environment so that the software is tested on all platforms.
  • Like, Windows 10+Internet Explorer 11+ Windows Office 2010+ …
  • Like Android 4.2.2+ Chrome browser.
  • If your application works with multiple databases (If documented), keep the databases (MySQL, Oracle, SQLServer) in the test matrix in such a way that they are too integrated with some test.
  • Configure test machines accordingly and name them as SetUp1, SetUp2, etc.
  • SetUp1 will have Windows 7+ IE 10+ Office 2007+.
  • SetUp2 may have Windows 10+ IE Edge+ Office 2013+.
  • SetUp3 may have an Android phone with your .apk file installed.
  • Congratulations! Your test setup is ready and you have also included every possible combination of platforms that your application will work upon.

#4) Testing

Finding Bugs

Finally, your application build is out and you are ready to find bugs! Now it’s’ time to work on test planning and find as many bugs as possible. There will be some phases in between if you work in an agile environment, then simply follow those scrum methods.

Below diagram depicts the categorization of various testing types:

Types of Testing

Challenges

Testing is a cumbersome process which itself is error prone! One finds many challenges while testing an application. Given below are some best practices to rescue.

Best Practices

Cheer up! You are trying to find defects in the code. You need to pay attention to the overall working of the software

Maximizing Quality by Going Above and Beyond Full Stack Testing

Quality expectations are increasing day by day and the search for perfection will never come to an end, thereby raising the bar on Product Quality and End-user experience.

An application that just “does the job” is not enough anymore for an average user and even expectations from the professional tools designed for a narrow range of users are growing gradually. Market demand is changing rapidly and QA processes should adapt to it accordingly.

Why is Full Stack not Enough?

Usually, full stack testing provides enough confidence on product quality from both the QA and end-user perspective. However, some aspects of it are not covered. Reasons may be different like not mentioned in the requirements, not obvious or the QA team is just not aware of how to do these tests or what is the acceptance criteria etc.

Functional testing on all layers (backend/frontend) currently is a bare minimum for modern quality assurance, but this is not at all enough for a great product. Teams should be motivated and they must do all they can to deliver the best possible product quality and end-user experience.

In an Agile approach, relatively small teams work on small pieces of functionality in a short sprint. Under such cases, tight communication between the team members is possible without destroying the process. This, in turn, could help to share knowledge, have discussions on code improvements or just will help to ask the right questions to the product owners.

When to Start Testing?

The crispy answer to this question will be to start as early as possible. In some cases, thinking about the QA at the ideal stage can point out weak architecture places and may require some information from QA standpoint. If an application is not testable, then it will have a huge impact on troubleshooting and the testing costs.

Some of which are:

  • Time to develop the test strategy: Teams have to find out what is the possible way to do an e2e test with a limited access and validate the data without direct access to the DB etc.
  • Testing time: As standard approaches do not work and the team has to use a workaround.
  • Troubleshooting costs: Time to reproduce the defect will grow.

Going Above and Beyond Full Stack Testing

The best bugs are the ones that were caught early even before the implementation. This is considered to be the prime achievement of proactive testing and the right QA team involvement in the creation of technical requirement/development process.

Teams could improve efficiency and extend test scenarios in the following way.

Stages where testing can be performed:

Stages where testing can be performed

Requirements review with the developer & Product owner/field expert: This is the perfect opportunity for the QA team to learn more about the business rules & the value behind a project, and also have a direct discussion with PO about the requirements. Whereas one or few sessions will be useful, but it will not provide enough information to understand the project completely.

Best ways are to do some exploratory testing, dig into the code and read about the field. Upon completion of the session, one can ask educated questions to clarify some application-specific questions.

Architecture review: This, in turn, will be helpful for both the developers as well as the QA. QA should be aware of architecture, its limitations, bottlenecks and the various testing approaches. Developers, on the other hand, need to know what test scenarios a QA plans to execute and either extend them or confirm weak points in architecture.

At this stage, the team fully understands requirements: There should be no bugs in the functionality and end user should be happy.

What Else Needs to be Tested?

When the question arises on what else is there to test? The answer would be some non-functional tests. There is a list of non-functional tests to be carried out. Few of which are briefed below.

Lots of Non-Functional tests:

#1) Performance Test

There are multiple aspects of the system performance that may drive the users crazy. Hence, performance testing technique is implemented.

#2) Speed Test 

Simplest tests to see how quickly the system responds to the call/command from a user.

#3) Load Tests

This is similar to speed test, but with a different count of clients to see if the system can handle a production-like load.

#4) Stability Tests

This is a type of load test that checks how well system a handles load over a long period. Best way to do it is to emulate user activity cycles for few days to see if the system is able to free the resources it was using for peak user activity.

#5) Scalability Test

Architecture-specific tests for clustered applications to see if the system can have a stable flat response time with an increasing load.

Each of the metrics requires its own on test strategy. Strategies for these tests are defined based on the load profile and its expected user count.

In some cases, Rise-up – Cool-down tests (load varying over time) are used to see if the system can get to the normal state after increased load (higher than nominal).

There are multiple tools that can help to complete the task like very simple Gatling, simple to use but agile SoapUI to universal Jmeter that has lost of components and scripting abilities.

System Performance Test:

System performance test

#6) Usability

Ideally, workflow, design and UX feature description is provided to the team. In some cases, the team creates it by itself.

QA’s will be the first users who can provide feedback, but it may not be easy to find the right balance. Design and usability are opened to a wide interpretation and may not have the exactly correct answer and it can waste a lot of time.

The ideal solution here is the document with the company color schemes, preferred fonts, and design patterns. Some things are better when left to the qualified professionals.

#7) Industry Standards

In certain cases, there are some expectations from the users.

A typical example will be the login page. The user has certain expectations and creativity like swapping login and password inputs that will not go well.

Some design elements/styles are also becoming a part of the user expectation and should be kept in mind. QA specialist should pay attention in not just to the application he tests, but also on the overall direction, that the IT world is taking.

#8) Security Testing

The most complex aspect of testing that rarely gets enough attention and requires very high competence in IT and also time is the Security testing technique. There is a limited count of standard tests that can be done on all the other tests that are platform/application specific.

Data encryption, access limitations, SQL injections are pretty simple tests, but usually, exploits are using different types of bugs. The best way in security testing would be to expertise in the architecture and reading.

There are many useful articles available on different types of security attacks and how to avoid them, reading which could make the coffee break a more interesting one.

#9) Being Initiative

This is one of the most important advice. Learning new skills, offering improvements and being interested in the future of the project are few initiatives which one can implement. Learning about the technology stack used in a project will help to choose an efficient test strategy that will include only the relevant scenarios and increase the efficiency of testing.

This is what allows to go “above and beyond” apart from simply completing the tasks from 9 am to 5 pm.

Conclusion

Quality assurance during the earlier stages of a project can be a very cost-effective one. However, going above and beyond can truly raise the bar for product perfection.

Single usability issue, singe security bug or inability to handle peak load may lead the existing clients to lose hope. And this QA approach requires high motivation of each and every team member and may not work on all the teams, but is worth to give a try.

What is Beta Testing? A Complete Guide

Beta Testing is one of the Acceptance Testing types, which adds value to the product as the end-user (intended real user) validates the product for functionality, usability, reliability, and compatibility.

Inputs provided by the end-users helps in enhancing the quality of the product further and leads to its success. This also helps in decision making to invest further in the future products or the same product for improvisation.

Since Beta Testing happens at the end user’s side, it cannot be the controlled activity.

What is Beta Testing? A Complete Guide

Beta Testing is one of the Acceptance Testing types, which adds value to the product as the end-user (intended real user) validates the product for functionality, usability, reliability, and compatibility.

Inputs provided by the end-users helps in enhancing the quality of the product further and leads to its success. This also helps in decision making to invest further in the future products or the same product for improvisation.

Since Beta Testing happens at the end user’s side, it cannot be the controlled activity.

What is Beta Testing – Definition

Beta Testing is one of the Customer Validation methodologies to evaluate the level of customer satisfaction with the product by letting it to be validated by the end users, who actually use it, for over a period of time.

Product experience gained by the end users are asked for feedback on design, functionality, and usability and this helps in assessing the quality of the product.

Real People, Real Environment, Real Product are the three R’s of Beta Testing and the question that arises here in Beta Testing is “Do Customers like the Product?”.

Purpose of Beta Testing

The points mentioned below can even be considered as the objectives for Beta Test and are very much required to produce far better results for a product.

#1) Beta Test provides a complete overview of the true experience gained by the end users while experiencing the product.

#2) It is performed by a wide range of users and the reasons for which the product is being used varies highly. Marketing managers focus on target market’s opinion on each and every feature, while a usability engineer / common real users focus on product usage and easiness, technical users focus on installation and uninstallation experience, etc..

But the actual perception of the end users clearly exhibits why they need this product and how they are going to use it.

#3) Real world compatibility for a product can be ensured to a greater extent through this testing, as a great combination of real platforms is used here for testing on a wide range of devices, OS, Browsers, etc.

#4) As a wide range of platforms which the end users are actually using, might not be available to the internal testing team during the QA, this testing also helps to uncover the hidden bugs and gaps in the final product.

#5) Few specific platforms will cause the product to fail with showstopper bug which was not covered during QA. And this helps in improvising/fixing the product to be a compatible one with all possible platforms.

#6) Known Issues, which are accepted by the Product Management team, may take a great turn when the end user faces the same issue and may not be comfortable while using the product. In such cases, this testing helps to analyze the impact of known issues on the entire product as the user experience gets hampered and is not acceptable for any successful business.

When is Beta Testing Done?

Beta Testing is always performed right after the completion of Alpha Testing, but before the product is released to the market (Production Launch / Go Live). Here the product is expected to be at least 90% – 95% completed (stable enough on any of the platforms, all features either almost or fully complete).

Ideally, all the technical Products should undergo the Beta Testing phase as they are mainly dependent on platforms and process.

Any Product undergoing Beta Test should be reviewed against certain Readiness Checklist before launching it.

Few of them are:

  • All the components of the Product are ready to start this testing.
  • Documentation that has to reach the end users should be kept ready – Setup, Installation, Usage, Uninstallation should be detailed out and reviewed for correctness.
  • Product Management team should review if each and every key functionality is in good working condition.
  • Procedure to collect Bugs, feedback etc should be identified and reviewed to publish.

Usually, one or two test cycles with 4 to 6 weeks per cycle are the duration of Beta Test. It gets extended only if there is a new feature added or when the core component is modified.

Stakeholders and Participants

Product Management, Quality Management, and User Experience teams are the Stakeholders in Beta Testing and they closely monitor each and every move of the phase.

End users/Real users who actually want to use the product are the Participants.

Strategy

Beta Test strategy:

  • Business objectives for the product.
  • Schedule – Entire phase, cycles, duration of each cycle etc.
  • Beta Test Plan.
  • Testing approach to be followed by the participants.
  • Tools used to log bugs, measure productivity, collect feedback – either through surveys or ratings.
  • Rewards and Incentives to the participants.
  • When and How to end this testing phase.

Beta Test Plan

Beta Test Plan can be written in many ways based on the extent to which it is performed.

Here I am listing out the common items for any Beta Test Plan to include:

  • Objective: Mention the objective of the project so as to why it is undergoing Beta Testing even after performing rigorous internal tests.
  • Scope: Clearly mention what are the areas to be tested and what is not to be tested. Also mention any specific data to be used for a particular feature (say use test credit card for payment validations – Card no, CVV, Expiry Date, OTP, etc).
  • Test Approach: Clearly mention whether the testing is exploratory, what to focus on – functionality, UI, response, etc. Mention the procedure to log bugs and also what all to provide a proof (Screenshots/Videos).
  • Schedule: Clearly specify the Start and End Dates with time, number of cycles, and duration per cycle.
  • Tools: Bug logging tool and its usage.
  • Budget: Incentives for bugs based on it severity
  • Feedback: Collecting Feedback and Evaluating methods.
  • Identify and review the Entry and Exit criteria.

Entry Criteria

  • Alpha Testing should be signed off.
  • Product’s Beta version should be ready and launched.
  • User Manuals, Known Issues list should be documented and must be kept ready to be published.
  • Tools to capture bugs, feedback should be ready and usage documentation should be published.

Exit Criteria

  • No Showstopper bugs in any of the platform.
  • All Major bugs discovered in Beta Test phase should be fixed.
  • Beta Summary Report.
  • Beta Testing Sign Off.

A strong Beta Test Plan and its effective execution it will result in the success of the testing phase.

How is Beta Testing Performed

This type of testing can be performed in several ways, but there are five different stages in general.

#1) Planning

Define the goals in advance. This helps in planning the number of users required to participate in the testing and the duration required to complete and reach the goals.

#2) Participants Recruitment

Ideally, any number of users can participate in testing, but due to budget constraints, the project has to set up a  minimum and maximum limit on the number of users participating. Usually, 50 – 250 users are targeted for mid-complex products.

What is Beta Testing? A Complete Guide

Beta Testing is one of the Acceptance Testing types, which adds value to the product as the end-user (intended real user) validates the product for functionality, usability, reliability, and compatibility.

Inputs provided by the end-users helps in enhancing the quality of the product further and leads to its success. This also helps in decision making to invest further in the future products or the same product for improvisation.

Since Beta Testing happens at the end user’s side, it cannot be the controlled activity. 

This article gives you a complete overview of Beta Testing, thereby explaining its meaning, purpose, need for it, challenges involved etc in a clear easy to understand format.

What is Beta Testing – Definition

Beta Testing is one of the Customer Validation methodologies to evaluate the level of customer satisfaction with the product by letting it to be validated by the end users, who actually use it, for over a period of time.

Product experience gained by the end users are asked for feedback on design, functionality, and usability and this helps in assessing the quality of the product.

Real People, Real Environment, Real Product are the three R’s of Beta Testing and the question that arises here in Beta Testing is “Do Customers like the Product?”.

Recommended Reading:

Purpose of Beta Testing

The points mentioned below can even be considered as the objectives for Beta Test and are very much required to produce far better results for a product.

#1) Beta Test provides a complete overview of the true experience gained by the end users while experiencing the product.

#2) It is performed by a wide range of users and the reasons for which the product is being used varies highly. Marketing managers focus on target market’s opinion on each and every feature, while a usability engineer / common real users focus on product usage and easiness, technical users focus on installation and uninstallation experience, etc..

But the actual perception of the end users clearly exhibits why they need this product and how they are going to use it.

#3) Real world compatibility for a product can be ensured to a greater extent through this testing, as a great combination of real platforms is used here for testing on a wide range of devices, OS, Browsers, etc.

#4) As a wide range of platforms which the end users are actually using, might not be available to the internal testing team during the QA, this testing also helps to uncover the hidden bugs and gaps in the final product.

#5) Few specific platforms will cause the product to fail with showstopper bug which was not covered during QA. And this helps in improvising/fixing the product to be a compatible one with all possible platforms.

#6) Known Issues, which are accepted by the Product Management team, may take a great turn when the end user faces the same issue and may not be comfortable while using the product. In such cases, this testing helps to analyze the impact of known issues on the entire product as the user experience gets hampered and is not acceptable for any successful business.

When is Beta Testing Done?

Beta Testing is always performed right after the completion of Alpha Testing, but before the product is released to the market (Production Launch / Go Live). Here the product is expected to be at least 90% – 95% completed (stable enough on any of the platforms, all features either almost or fully complete).

Ideally, all the technical Products should undergo the Beta Testing phase as they are mainly dependent on platforms and process.

Any Product undergoing Beta Test should be reviewed against certain Readiness Checklist before launching it.

Few of them are:

  • All the components of the Product are ready to start this testing.
  • Documentation that has to reach the end users should be kept ready – Setup, Installation, Usage, Uninstallation should be detailed out and reviewed for correctness.
  • Product Management team should review if each and every key functionality is in good working condition.
  • Procedure to collect Bugs, feedback etc should be identified and reviewed to publish.

Usually, one or two test cycles with 4 to 6 weeks per cycle are the duration of Beta Test. It gets extended only if there is a new feature added or when the core component is modified.

Stakeholders and Participants

Product Management, Quality Management, and User Experience teams are the Stakeholders in Beta Testing and they closely monitor each and every move of the phase.

End users/Real users who actually want to use the product are the Participants.

Strategy

Beta Test strategy:

  • Business objectives for the product.
  • Schedule – Entire phase, cycles, duration of each cycle etc.
  • Beta Test Plan.
  • Testing approach to be followed by the participants.
  • Tools used to log bugs, measure productivity, collect feedback – either through surveys or ratings.
  • Rewards and Incentives to the participants.
  • When and How to end this testing phase.

Beta Test Plan

Beta Test Plan can be written in many ways based on the extent to which it is performed.

Here I am listing out the common items for any Beta Test Plan to include:

  • Objective: Mention the objective of the project so as to why it is undergoing Beta Testing even after performing rigorous internal tests.
  • Scope: Clearly mention what are the areas to be tested and what is not to be tested. Also mention any specific data to be used for a particular feature (say use test credit card for payment validations – Card no, CVV, Expiry Date, OTP, etc).
  • Test Approach: Clearly mention whether the testing is exploratory, what to focus on – functionality, UI, response, etc. Mention the procedure to log bugs and also what all to provide a proof (Screenshots/Videos).
  • Schedule: Clearly specify the Start and End Dates with time, number of cycles, and duration per cycle.
  • Tools: Bug logging tool and its usage.
  • Budget: Incentives for bugs based on it severity
  • Feedback: Collecting Feedback and Evaluating methods.
  • Identify and review the Entry and Exit criteria.

Entry Criteria

  • Alpha Testing should be signed off.
  • Product’s Beta version should be ready and launched.
  • User Manuals, Known Issues list should be documented and must be kept ready to be published.
  • Tools to capture bugs, feedback should be ready and usage documentation should be published.

Exit Criteria

  • No Showstopper bugs in any of the platform.
  • All Major bugs discovered in Beta Test phase should be fixed.
  • Beta Summary Report.
  • Beta Testing Sign Off.

A strong Beta Test Plan and its effective execution it will result in the success of the testing phase.

How is Beta Testing Performed

This type of testing can be performed in several ways, but there are five different stages in general.

#1) Planning

Define the goals in advance. This helps in planning the number of users required to participate in the testing and the duration required to complete and reach the goals.

#2) Participants Recruitment

Ideally, any number of users can participate in testing, but due to budget constraints, the project has to set up a  minimum and maximum limit on the number of users participating. Usually, 50 – 250 users are targeted for mid-complex products.

 

#3) Product Launch

  • Installation packages should be distributed to the participants – Ideally, share the link from where they can download and Install.
  • Share User Manuals, Guides, Known Issues, Scope of testing to the participants etc.
  • Share the Bug logging methods to the participants.

#4) Collect and Evaluate Feedback

  • Bugs raised by the participants are handled by the bug management process.
  • Feedback & Suggestions are collected by the participants based on their experience with the Product.
  • Feedbacks are evaluated to analyze and make out the customer to satisfy the product.
  • Suggestions are considered to improve the product in its next versions.

#5) Closure

  • Once a certain point is reached and when all the features are working, no bugs are arising, and exit criteria are met then decide to conclude Beta Testing Phase.
  • Distribute Rewards / Incentives to the participants as per the plan decided and thank them formally to maintain good relationship (this helps in further beta test on the product, much more feedbacks, suggestions, etc)

Managing This Testing Phase

Managing the entire beta phase is not less than a challenge, as it cannot be controlled once started. So, it’s always a good practice to set up forum discussions and include all the participants to take part in it. Limit the discussions to the Beta aspects of the product and then follow the process.

Conduct Surveys for experience on the product and encourage the participants to write testimonials on the product

Identify the validators to monitor Beta Test Progress at frequent intervals and then allow them to communicate with the participants if required.

Challenges

Identifying and recruiting a right participant is the major challenge. Participants may or may not actually have the necessary skills to the required level. They may not be technical experts to test each and every aspects of the product, which will result in testing the product at very high levels.

Hidden bugs may be difficult to be uncovered in some cases. Another challenge is to collect the feedback. Not all the feedback can be considered as valuable ones nor not all can be evaluated. Only the relevant ones are to be picked to evaluate the customer satisfaction level.

Feedback should be delivered to the relevant teams which are again a tedious job for the Product Management Team. Also, Beta Testing cannot have well-defined plans always. It may have to wind up in a hurry in case of time constraints. This makes the goals unsuccessful and the product is not thoroughly experienced by the participants.

When does Beta Testing Fail:

  • No proper plan to execute.
  • Poor test management.
  • Tight deadlines due to delays in previous phases.
  • Released unstable product.
  • An improper number of participants – too few or too many.
  • Too short or too long test periods.
  • Ineffective tools.
  • No effective feedback management.
  • Poor Incentives.

Related Useful Terms:

Beta Software – It is the preview version of the software released to the public before final release.
Beta Version – It is the Software version releases to the public that includes almost all of the features in which development is not completed yet and may still have some errors.
Beta Testers – Beta Testers are those who work on the testing beta version of the software release.

How Companies can Make Beta Test Successful

Given below are few pointers which explain how to perform this testing successfully.

  1. First decide, how many days you want to keep the beta version available for testers.
  2. Identify the ideal user groups to perform this test – Either limited group of users or in public.
  3. Provide clear test instructions (user manual).
  4. Make the beta software available to these groups – Gather feedback and defects.
  5. Based on feedback analysis decide which issues need to be fixed before the final release.
  6. Once the suggestions and defects are fixed, again release the changed version for verification to the same groups.
  7. Once all tests are completed, do not accept any further feature change request for this release.
  8. Remove the beta label and release the final software version.

How to Get Started as a Beta Tester

Once your application as a beta tester is accepted by a company, then follow the below steps:

  • Download and read the software requirement specifications, known defects, and modules to test.
  • Download and install the beta software.
  • Start testing.
  • Prepare the bug report for the issues found in the application.
  • Also, note down your suggestions/feedback about the application to improve the user experience.
  • Submit the bug report and feedback to the company.

Adding Beta Testing Experience in Your Resume

Many entry-level candidates complaint about not getting real-time testing experience on software projects. Testing beta releases are the best opportunity for freshers to show their skills and also to get hands-on experience on real projects.

You can even put this experience on your resume with details (like the project, project description, test environment etc.) about the beta application which you tested. This will definitely catch the employer’s attention especially when you are a fresher seeking job in the software testing field.

How to Find an Opportunity as a Beta Tester

Option #1: Get software testing experience

Let’s take an example of Microsoft. You can apply to become a beta tester for Microsoft. If you check these opportunities at Microsoft there are currently more than 40 beta software available for testing. Microsoft Corporation is accepting defects and suggestions for these products.

This is a huge opportunity for you. Browse this list, select a product and start testing it locally. Use all your testing skills to find and log defects. Who knows – this might even land you the job of your dreams in any of such companies offering beta versions to test.

You can also find some more beta application testing opportunities on the link given here.

Option #2: Make some extra money

Some companies even pay you money to test their beta applications. Video game testing industry is one of the best starting points for paid beta testing opportunities. Most video game companies pay a decent amount to the beta testers for testing the beta versions of their video game releases.

But be careful before making any investment as there are many scam sites asking for money to join as a game tester. Before making any commitment make sure you investigate the site carefully. You can also find real Beta Tester jobs.

I mentioned the second option just as one of the opportunities for you but my main purpose is to educate you on beta test opportunities that you can use to improve your testing skill on real-life projects and the experience to mention in your resume to reach your dream job.

Conclusion

Until the users like a product, it can never be considered as successful.

Beta Testing is one such methodology which allows the users to experience the product before it reaches the market. Thorough testing on varied platforms and valuable feedback from the real users ultimately results in successful Beta Testing of the Product and ensures that the Customer is satisfied with its usage.

5 Excuses Every Software Tester Must Stop Giving

5 Excuses Every Software Tester Must Stop Giving

I have unlocked a pattern from my experience of hiring around hundred testers over a period of time and interviewing some thousand others.

From all the discussions I had with fellow testers during interviews, I felt happy numerous times seeing the quality talent which we have in our community of testers.

But let me also share the other side of the story, the patterns I am talking about. It makes me sad. 

I never feel happy watching potential performers being locked into a virtual cage of responsibilities. I feel discontent seeing the rock stars performing on a controlled stage.

If you are still wondering what’s the matter and what is the baseline, it is this- Considerably big part of our testing community is not having enough growth on multiple fronts for years after they start their career as a Tester. Forget 360 degree, it is not even near half of that.

Sorry for being harsh, but it is real.

Who is to blame for this? Maybe awareness in the industry to some extent. Company policies and senior management too may be. But more than anything else, it is the Tester himself/herself.

Read it again, it is YOU. It is us. Because we become victim to the excuse.

Below are the few patterns/excuses I uncovered.

5 Excuses Every Software Tester Must Stop Giving

Note: I nowhere mean that this is the case with every tester and every organization. But I have seen enough examples to say most of us are victims here.

#1) We don’t control our Test Environment, we have (read ‘very’) limited access

I often hear the statements/excuses- ‘We have just Read-only access to the environment’. Even worst- ‘We only have access to logs and nothing else’.  Everything else is done by either a Development team or some other team.

The work which will give us so many beautiful and fruitful insights about testing and many other technical kinds of stuff is not done by us. And maybe we are happy this way, but we should not.

Tell me one reason why you shouldn’t control your test environment and take maximum benefits of same.

Here are few possible benefits if you are wondering what I am talking about-

  1. You have full control over your test environment to make sure personally that it is exact or near replica of the production environment. This will help you avoid few surprises at least when your deliverables hit production.
  2. You know all involved components, all software used along with their versions for your product to function. With time, trust me you will have many insights about their working, limitations and possible failure points.
  3. You have enough access to do at least first level debugging in case there are Infra level issues. E.g.: Setup running slow, Checking CPU, Memory utilization and logs at every level of flow are no rocket science.
  4. You own control over setup, so you know what you are changing and what build you are deploying. You are much more confident than before while shipping the release ahead.
  5. You learn it and you learn it all. Though it is Linux based or Windows based.

Makes sense? Let me continue if you have at least agreed that there are benefits.

Now the question is what change you can make in your team/organization to make this work. And how exactly you will do it.

Well, I Don’t Know. I don’t know your team, your leads/managers, and your organization so I won’t be able to help you on this. I can surely share few things which might work. You try working very closely with your Dev or whichever team owning this and see what and how they do stuff. Everything.

The way they login to the environment (server), the way they logout and everything in between. Once you gain some knowledge, you will have the confidence to say few words, give some suggestions when a similar situation occurs again or if a similar activity is to be done again.

Obviously sooner or later your this confidence will be seen by your developers/leads/managers. That is the time you can ask for the control. And tell me one strong reason why they won’t give you that? They should be more than happy to do so if you have shown the needed capabilities.

Believe me, they have much more work to do, so it shouldn’t be hard for them to release the control to you. At least I hope so.

#2) We don’t deploy a build, some other team does it for us

You come to office on a Monday morning. You notice there are few issues in build causing blocker. You need new build from your Build repository. You raise request or contact your Dev team or deployment team. Ohh, they are busy with something. But they manage and do it after some time.

Now tell me, why all this? It is not as complex as it looks.

When to take a new build is surely something a Developer can suggest as he is the one who will be committing a feature or a bug fix. But when you simply need to trigger it and deploy it, why should you wait or depend on someone.

Having ability and authority to deploy whenever you want makes your work lot easier without any wait. Do you see that? It will also increase your Turnaround time in daily testing. Though it is debugging some defect with added logger, or taking the new build to verify resolved bugs. Or be it taking fresh build and starting with testing of a new feature.

Let me tell you something from my personal experience. It is not only about time. Deployment teaches you so many things that you can’t imagine. The reason is, it fails often. The application stops working with new code sometimes. Many times deployment itself fails. And every time something fails, it pushes you to debug it. It pushes you to search for something on Google or ask someone a question or the best ask yourself a question.

It makes you think.

Of course, it is not really a software testing, but it surely is testing your abilities.

I will say a large portion of my learning apart from pure software testing are from Deployment and Maintenance of environments.

  • I don’t remember how many trial and error attempts I performed for something, count must be few hundred.
  • I don’t remember how many times I went to check some release manual of some software.
  • And I don’t remember how many times I hit that Google search button or navigated to that stack overflow link.

All I know is- it gave me a lot, taught me a lot.

The remedy for this is very similar to the first excuse we talked about.

Learn it, show it and ask for it. It might work.

#3) We don’t debug an issue, we find steps and log it

I won’t go very hard on this as in an interview you don’t get a chance every time to discuss particular defect from candidate’s own experience and investigation/debugging done by him/her.

However, I can surely say that not all of us challenge our senses before passing on the defect to the developer.

Many times the log says it all. Many times the pattern of an issue says it all. Many times it fails because some prerequisite service was down.

So in short, many times it is actually possible for us to get to the exact root cause or at least reach near it.

So please ask yourself a few questions, not for helping developer but for making your test cycles fast and for your own growth.

You are the remedy here. No one else.

#4) I don’t know why it happened. Developer resolved it and I simply verified it

Ohh, come on. Sorry but hearing this annoys me a lot.

Every single time, it annoys me when I ask Tester a question- Why particular defect was faced or what was the RCA (Root Cause Analysis) and he/she answers ‘I don’t know’.

Have we taken Black box word this literally? How can we not ask Developer about what exactly he fixed? Why can’t we ask it if the authority is an issue?

I am damn sure that 99% times we don’t know the Root Cause Analysis (RCA) or fix because we don’t ask for it.

Believe me, knowing the exact fix, module in which fix was done, whether it was in the front end or back end, whether it was injected in any feature development or some other defect’s fix, will always help you in your testing. Always.

Apart from this, this information helps you know a lot of technical stuff which may be otherwise you will never come across.

Remedy? Ask it. Demand it. Whichever works.

#5) I didn’t get the opportunity to work on anything else than Manual Testing

Maybe this excuse is valid to some extent considering the possible workload on you or restricted responsibilities given, but not completely valid.

The problem here is, we say that we didn’t get the opportunity to perform non-functional types of testing, or we are not supposed to do it or we don’t have time.

Agreed, learning always demands time. But isn’t there anything you can change?

While testing that new feature or while testing that one API,

  • Isn’t it possible for you to keep watch on the response time?
  • Isn’t it possible that you ask your Business Analyst about the load this particular feature/module/API is expected to handle in production so that you can test it?
  • Isn’t it possible to perform at least basic security checks like session expiry, URL tampering or XSS handling on that website form you are supposed to test?
  • Isn’t it possible to at least say that this particular ‘Submit’ or ‘Help’  button doesn’t seem to be properly located or not easily noticeable and play your small role in making the product more usable?
  • Isn’t it possible for you to start with something and then keep learning it more and more?

The answer is YES, small or big depending on various factors surrounding you but it is surely not NO.

If your responsibility doesn’t demand it (because there is some other team for it), no one will stop you investing extra time for your own learning. Self-learning in extra or free time is always possible I think. So find those possible extras, and start doing it. Now.

Conclusion

I hope I triggered some thought process and some learning ambitions within you.

Please forgive if you found any of my words harsh, but believe me, I wanted to be 100% clear so that it reach out to you all with needed seriousness.

Hope it works positively for you all.

I will be more than happy to know any uncovered aspect around possibilities I talked about.

%d bloggers like this: