It was a rainy evening in Bucharest on Friday, the 13th of June.
Me and other 3 friends have participated to one of our greatest professional challenges: Software Testing World Cup.
Do you think you can cope with short deadlines?
What was your shortest deadline you had to meet ever (business related)? 180 days?
What if I tell you that, tens of teams had to deliver a bug report and a test report in less than 180 minutes? Would you take a chance to prove that you have the greatest team?
Well, that was what have we tried to do.
The infos provided by the organizers have been the following:
Testing mission: You will be testing a product that creates demo accounts and reports for small businesses. This product is currently in the marketplace and receives updates weekly from its Agile development team. Your mission is to investigate and report on the current stability of the system. The most recent addition to the product is a function that emails out invites to the demo account and to a customer center to view the demo accounts. The customer center and reputation product are not in scope and can be considered oracles for data correctness.
In scope for this competition are only URLs with salestool-demo.appspot.com.
Our primary concern beyond the email function is to have the system tested on as many devices and screen sizes possible. Additionally a login account should not be able to see locations created by other login accounts.
Technical details: the app uses google app engine, a NoSQL database called datastore or NDB and it runs on Django Framework.
In scope are all forms of testing except the following:
1) Load Testing:
DO NOT LOAD TEST THE SYSTEM. Teams caught load testing the system will be sent a bill for excess machine hours that their testing incurs.
The application lives on a Google app engine that automatically scales instances to meet demand. Load testing the application does not test the application, but the Google infrastructure.
2) Security Testing:
This is done at your own risk. We know of a session hijack bug in the system, but most of the security is provided by Google, and your security testing may set off their detection. Your risk, and our lowest priority.
Despite of the latest piece of information "security is our lowest priority", our team managed to find very useful Security bugs: Persistent XSS in printing report, Java script Injections, Application Error disclosures, open redirect. Those kind of bugs may have a great impact for both customers and stakeholders. How would a manager feel when he got an email with your report, and when he opens it he will be redirected to a non-desired website?
From functional point of view, things are not much better: identical accounts could be created using the same HTTP request, because no validation has been performed.
The contest goes on at this moment with Africa and South America editions. We are waiting for the final results. Hopefully we are going to get at least an award for the Most Valuable Security Bug.
Technical Issues we had to cope with:
I have not been able to log into the testing app for the first competition hour, until my password has been reset.
The login credentials were public for everyone (user: email, password: name of the team). So you could identify the credentials for anybody else, anytime.
The HP Agile Manager bug tracking system allowed anyone to see the submitted bugs for any other team (just by changing an id within the URL). This "feature" was clearly specified at the beginning of the competition as not possible and not an intended behavior.
What I would do different next time?
I would start writing the test report much earlier.
I would keep my calm for a longer time... :)
This competition is not just about how fast you are, or skilled in finding defects, but it is also about communication.
It is very important to be part of a team. Participating lonely in such a competition reduces your chances to submit a good test report to 0.