(Certified || !Certified) != (Qualified || !Qualified)

This is probably not the first time you are reading about the importance of certification for testing skills. However, it feels that some leaders/managers in the field still don’t completely understand the value (if any) that certified testers add to the team; and also testers themselves aren’t always aware if they should invest time or money for these certificates.

So rather than focusing on what industry says and suggests about certification, here I will discuss what I have learnt -- as a hiring manager and ISTQB foundation, intermediate and advanced certificate holder -- about the significance and/or impact of certification on the testing community.

I didn’t have any certificates specific to testing skills when I started my career as a tester in India. However, I when began looking for my first job in the UK, whenever I was contacted by a recruiter, I would often be asked if I have an “ISTQB/ISEB certificate in Software Testing’’. This was even after my MPhil research degree in the field of Quality Engineering from a top UK university, and couple of years experience as a part of test team in the international CMM certified (maturity level 5) company in India!

I tried my hardest to convince recruiters about my experience and skills as an efficient tester but it seemed that most of the time the certificate was an essential skill requirement set by hiring managers. Therefore, I didn’t have any other option but to spend about one week to read the official book published by BCS and pass the foundation certificate. And to be honest, it was possibly the first ever certificate I passed where I felt that I didn't learn anything new.

Since then I have had many online/offline discussions with recruiters and senior managers about why they feel that certifications are important. Most of the time the argument from recruiters is that the hiring manager requires it. And the hiring manager requires it because certification helps to filter out the candidates who are serious about their career as a tester.

I won’t criticise certification completely and can possibly see where it can help. If the company uses the same vocabulary about test processes that the syllabus uses, then it can help the certificate holder to be efficient at work (however, I still haven’t come across any such companies). If the candidate hasn't worked as a tester before or hasn't done much reading about the field, then possibly the certification can help introduce candidates to testing. This is mainly because as far as I am aware most degree subjects at a college/university don’t teach anything about testing (other than possibly a couple of paragraphs mentioned in a few Software Engineering books).

So certifications can sometimes be important for reasons that are useful to the role. If the context is suitable, there’s nothing wrong in asking for candidates who are certified, or in sending the team to such courses. And if the course provider and participants are good, then sometimes while attending such courses we can also learn some things about testing that are not a part of the certification syllabus. The courses can be both fun and informative.

I won’t say that if someone is not certified, they are a bad tester; it just means that the candidate might not be as good as they could be. However, even if the candidate scored well in the multiple-choice certification exam, that doesn't make him or her an excellent tester, as the certification can’t teach or develop all the skills required to be a good tester.

And there are, of course, many good testers in the industry who never had any such certification. Many things help a tester to be better: an eagerness to dive deep to understand the domain; a willingness to improve the technical skills required to investigate or identify the product issues; and the experience, attitude, curiosity, confidence and ethics to efficiently deliver a good quality product.

So, if we are recruiting, we would be wrong to rely mainly on a candidate’s ISTQB certification. We should consider all factors, such as in-use testing processes, experience, the role requirement, and so on. The institutes providing certifications will always try to sell the certification courses, but at the same time they also describe the syllabus. So rather than blaming only the providers, I also believe that hiring managers and candidates also misjudge the importance of certifications.

The testing community should neither deny the utility of certifications, nor always demand them. In the end, on-the-job training, testing-specific texts, courses, and -- sometimes -- these certifications all can help us to be better testers.

Quality of the Test Team

An important factor regarding the effectiveness of software testing is the human factor: ‘the test team’.

There are some tools available to help with process and to implement and to manage and to check, but the testing is mainly driven by its participants, that is, the members of the test team. The test team is responsible for improving the whole development process – not only quality of checking and quality of error logging. Therefore the quality of the test-team itself matters a lot. 

And I believe that - other than available support from team leaders or quality of development team members - the specific factors affecting quality of test team members are: experience of the testing role, attitude, training given to the team, team size, technical skills, domain experience, and also the range of team.

That testing processes have an influence on development and delivery is indisputable. However, rather than the processes, the testers themselves are the biggest influence on test effectiveness; mainly their attitude and motivation - if team members are unwilling to perform testing effectively, the efforts will be doomed to fail.

The quality of the team influences the time needed for defect prevention, defect detection, defect collection and also the quality of the output. Effective size also matters. The larger teams tend to need more time to agree on a common position due to more discussions among members.

Also, the types of people in the team may influence the work results. If one very dominant person is more or less controlling the team, their opinion will dominate the results. This can be advantageous; if they are good at contributing, but can also have the adverse effect. Their input in everything might discourage some members and also not help them to grow or get empowered.

On the other hand, if teams are comprised only of people that are known to agree on everything, the results may not be respected by the people performing the next process steps, rendering the steps taken so far useless. If the right people form the team, the issues raised can be solved smoothly, with good results in terms of quality of output. A team consisting of the wrong combination of people may consume considerably more effort. 

Review of Cost Benefit Analysis Techniques

Cost benefit Analysis is an art consisting of a series of techniques useful for decision making (Radice, 2002). The cost benefit analysis of SPI is important because it is process of determining the money to be gained by following the process. It can be also used to determine how much money is lost from creating and using a new and improved software process.

In his book, Rico (2004) explains techniques to do cost benefit analysis for different SPI techniques; such as Benefit/Cost ratio (B/CR), Return On Investment (ROI) and Net Present Value (NPV); with different SPI models and strategies such as Software Inspections, Personal Software Process (PSP), Team Software Process (TSP), Capability Maturity Model (CMM), ISO 9001 and Capability Maturity Model Integration (CMMI). O’Neil (2005) has also explained to calculate return on investment using Net Present Value (NPV) formula.

Benefit is the amount of money which results from a SPI method. Benefit is the economic value resulting from a new and improved software process. Cost is the amount of money that is spent on a SPI method. Cost is the amount of money you must spend to get something back. Benefit/cost ratio (B/CR) is simply the ratio of benefits to costs. B/CR is a measure of how much money is gained from using a SPI method. For example, a B/CR ration of 2:1 means that for every pound we spent, two pounds is returned.

ROI (Return on Investment) is the amount of money that is gained after spending an amount of money. That is, ROI refers to the amount of money gained. For example, an ROI of 10% means that for every pound you invest, 10 cent is returned.

Net Present Value (NPV) is what money is worth in the future. NPV is the economic value of today’s money in the future less inflation. For example, if we assume inflation rate of 5% per year, £10 today will be worth £9.52 a year from now.

At first, these techniques look quite easy, essential and simplistic in the field of software process improvement as they do not involve more than one or two significant terms or inputs; and also support core of cost benefit analysis that benefits do exist, and costs must be counted.

However, these techniques might not be good for a variety of reasons. Firstly, they all are inadequate for expressing the depth of detail necessary to describe software processes. These techniques offer a formula, but using them early for cost benefit analysis of SPI techniques is not possible. They also make no attempts to describe the relationship between the variables other than the statistical relationship. They cannot also work with incomplete data when some metrics data is missing. One must wait until later in the life cycle to be able to use them, thus predictions not available when needed.

Also that, SPI decisions are primarily taken by humans, and therefore the outcome is often uncertain. Uncertainty arises in many situations. For example, experts may be uncertain about their own knowledge. Uncertainty can be either caused by the unpredictability of future events, or it can also be caused by limitations on the accuracy of the data. Available cost benefit analysis techniques do not include any explicit analysis of risk or uncertainty. They either assume that all future costs and benefits are known or assume that there is no reasonable way to include uncertainty in the calculations. Such assumptions are obviously unrealistic, and it is clear that decisions would benefit from more explicit consideration of the uncertainties affecting future costs and benefits.

Organisations want to improve their existing processes, but they do not know exactly what to change. They cannot predict with certainty what will be the future output and how many defects the work-product will have after following certain processes. It is because they are not sure which inputs they have not yet executed, would produce a failure if executed. If they could know this, they could use the information for defect prevention or process improvement.

There is necessity of the approach to recognise the uncertainty and factor it into the cost benefit analysis, which clearly offers following advantages:

• The explicit recognition of uncertainties helps decision makers understand the quality of process used to support a particular decision and gives them an idea of potential problems in the analysis.
• Analysing the impacts of uncertainty on a cost benefit analysis often highlights factors for which better information is needed.
• This also reveals factors that have the greatest influence on the possible results of the project. Once these factors are recognised, it may be possible to modify them to get maximum return on investment.
• If uncertainties are analysed before the project is begun, it is frequently possible to suggest conditions that indicate when particular process should be terminated.