Broken Logic: Avoiding the Test Site Fallacy

Wednesday, July 25, 2012

Fergal Glynn


Article by Ian Broderick

Dynamic Application Security Testing (DAST) has become an integral part of the SDLC in most organizations today. DAST tool vendors demonstrate their tools by allowing prospects to scan test sites so they can see how the scanner works and the reports generated.

We recently featured a webinar from Veracode Senior Security Researcher, Isaac Dawson, on why we should not gage the effectiveness of a particular scanner by only looking at the results from scanning these public test sites. If you would like to view the webinar click here. In addition, we are sharing highlights from our Q&A session with Isaac:

Q: Could you elaborate a little bit on why you were working on this project?

Isaac Dawson: The reason that I was looking at all these public test sites was primarily that our customers, when they first do a comparison of our scanner versus other scanners, they run these public test sites. We need to make sure that we are finding all the issues that everybody else finds and more if possible and we need to do this for every release of our scanner. Anytime we make any changes we need to make sure that we find the same issues.

Q: Does Veracode offer a test site for people who want to test drive the Veracode application scanner?

Isaac Dawson: We believe it is not a good idea to do that, so we do not have a test site running.

Q: What is your recommendation for the ideal way to test a scanner? You mentioned open source options, can you tell us a little bit more about how you would recommend testing prior to purchase?

Isaac Dawson: Sure. It depends entirely on your technical skill level. If you are not comfortable running or creating your own test cases to make sure that the scanner can handle specific types of coverage issues like going through stretch frameworks or coding your own vulnerabilities to test the scanner, there are some open source applications available. Examples of open source applications that you can download are VBWA, Webstore and Hacme Bank. All of these applications are available for download. They are open source, so you can get them installed, review the code to see exactly why it is a vulnerability and why the scanner can or cannot find it.

Q: Are there any other test sites that you didn’t include in this review?

Isaac Dawson: There are a number of test sites that are not included. There are a few from Acunetix. If you are looking at a new scanner and if the organization is locking you into scanning that specific site, be aware that you need to go above and beyond just scanning that single site with multiple scanners, you need to do more in-depth analysis.

Q: If people are working on doing their own scanner, can they get in contact with you to talk about it?

Isaac Dawson: Sure. I am available at

Cross-posted from Veracode

Possibly Related Articles:
Information Security
Testing Application Security SDLC Scanners Tools Secure Coding Network Security DAST
Post Rating I Like this!
Maureen Robinson This article does a great job of summarizing the aspects regarding application testing.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.