It was a rainy evening in Bucharest on Friday, the 13th
of June.
Me and other 3 friends have participated to one of our
greatest professional challenges: Software Testing World Cup.
Do you think you can cope with short deadlines?
What was your shortest deadline you had to meet ever
(business related)? 180 days?
What if I tell you that, tens of teams had to deliver a bug
report and a test report in less than 180 minutes? Would you take a chance to
prove that you have the greatest team?
Well, that was what have we tried to do.
The infos provided by the organizers have been the
following:
Testing mission: You will be testing a product that
creates demo accounts and reports for small businesses. This product is
currently in the marketplace and receives updates weekly from its Agile
development team. Your mission is to investigate and report on the current
stability of the system. The most recent addition to the product is a function
that emails out invites to the demo account and to a customer center to view
the demo accounts. The customer center and reputation product are not in scope
and can be considered oracles for data correctness.
In scope for this competition are only URLs with salestool-demo.appspot.com.
Our primary concern beyond the email function is to have the
system tested on as many devices and screen sizes possible. Additionally a
login account should not be able to see locations created by other login
accounts.
Technical details: the app uses google app engine, a NoSQL
database called datastore or NDB and it runs on Django Framework.
In scope are all forms of testing except the following:
1) Load Testing:
DO NOT LOAD TEST THE SYSTEM. Teams caught load testing the
system will be sent a bill for excess machine hours that their testing incurs.
The application lives on a Google app engine that
automatically scales instances to meet demand. Load testing the application
does not test the application, but the Google infrastructure.
2) Security Testing:
This is done at your own risk. We know of a session hijack
bug in the system, but most of the security is provided by Google, and your
security testing may set off their detection. Your risk, and our lowest
priority.
Despite of the latest piece of information "security is
our lowest priority", our team managed to find very useful Security
bugs: Persistent XSS in printing report, Java script Injections, Application
Error disclosures, open redirect. Those kind of bugs may have a great impact
for both customers and stakeholders. How would a manager feel when he got an
email with your report, and when he opens it he will be redirected to a
non-desired website?
From functional point of view, things are not much
better: identical accounts could be created using the same HTTP request,
because no validation has been performed.
The contest goes on at this moment with Africa and South
America editions. We are waiting for the final results. Hopefully we are going
to get at least an award for the Most Valuable Security Bug.
Technical Issues we had to cope with:
I have not been able to log into the testing app for the
first competition hour, until my password has been reset.
The login credentials were public for everyone (user: email,
password: name of the team). So you could identify the credentials for anybody
else, anytime.
The HP Agile Manager bug tracking system allowed anyone to
see the submitted bugs for any other team (just by changing an id within the
URL). This "feature" was clearly specified at the beginning of the
competition as not possible and not an intended behavior.
What I would do different next time?
I would start writing the test report much earlier.
I would keep my calm for a longer time... :)
Conclusions:
This competition is not just about how fast you are, or
skilled in finding defects, but it is also about communication.
It is very important to be part of a team. Participating
lonely in such a competition reduces your chances to submit a good test report
to 0.
For me it will be unfor… wait for it…gettable.
