A fellow consultant asks me to define Pen Testing and Vuln Testing

Recently my friend and fellow IT consultant Marc Mintz (Mintz Infotech, https://mintzit.com/) asked me to clarify some of what I do for his clients. Here’s his question:


Glenn: I don’t know if my target market really understands pen and vulnerability testing, but since they should, I’d like to have some information for them.

I. What is Pen and Vulnerability testing

II. What are the benefits of Pen and Vulnerability testing.

III.What businesses are required to have this security testing?

IV. What is involved – what does it look like and how is your organization impacted during the process.

V.Costs, both in down time and $$$

VI. Everything else I don’t know enough to include.


So here’s my response:


Often shortened to “pen testing,” this is a limited subset of security analysis. In the certification world, you’ll find distinctions between Pen Testers and Security Analysts, with pen testers being more glorified but analysts doing the real work.

Pen testers look for openings they can penetrate. Simple as that. Except it’s not simple. The real question is, what are you testing


The critical consideration is the scope of the pen testing. For a web application, the app itself, its hosting and its web server software would be the scope. Notice that this is very limited: it does not include, for instance, any email services that may be involved – and may be critical.

For a corporate network, the scope might include all external IP addresses, all external email, chat, messaging, voicemail and VOIP services, all hosting arrangements, all data network providers – or only a subset of these, or even perhaps far more than these, depending on the proposed scope of the pen test.


Are you just looking for potential vulnerable points, or are you actually trying to perform a penetration? These are two very different things. Real pen testing might actually bring your business down (I might break things trying to get in), while simply scanning for vulns shouldn’t (unless badly done, which is a real possibility). But finding a list of vulns does *not* actually determine if your business can be penetrated; in fact, thinking you’re safe if you fix that list is a big vulnerability of its own.
If you really want to know that you’re cast-iron set-in-concrete secure, turn me loose to do full pen testing, and I’ll let ‘er rip. I’ll find a hole somewhere, in the network layers or at the human layer (depending on scope). Hardly anyone actually does this except the government. Most people want vuln testing, which gives them a solid to-do list of things to fix. This is the way to go for proof of compliance or due diligence or similar legal concepts. Security? You likely get a little security out of vuln testing, though not as much as some people think. But if you’re really getting ferocious about security, you want something much deeper generally called security analysis. A security analyst might note, for instance, that your firewall device has a hardware fault or your email server is an open relay, and that you should fix them.


There are somewhat similar requirements across several industries, but of course specifics have to be slavishly followed. For HIPAA-compliant organizations, an annual Risk Analysis includes things like pen testing, auditing and user training. For schools, for the most part, they only need to deal with simple records storage security under FERPA. Military and mil-contractor organizations, on the other hand, have to follow FIPS guidelines, which require frequent and fearsome pen testing. Business and financial outfits have various Dodd-Frank and Gramm–Leach–Bliley security requirements that include risk analysis, which in turn includes pen testing, user training, auditing and so much more.

My point is that pen testing is one tool in the box for proof of compliance, but it’s not the only one. Not by a long shot.


Any hacker worth his/her salt is going to work in ways they hope they won’t be detected, assuming data theft is the goal. Pen testing, on the other hand, is frequently (dismayingly) done during business hours, very much to the detriment of the business’s operations. That’s why I see statements in contracts like “testing must be halted immediately if the customer’s operations are affected.” I’m sorry, but this is ignorant.

On the other hand, denial of service is a legitimate goal, though you don’t really want to test it. You’ll just be testing the resilience of your data and hosting providers’ networks, and that is a very big no-no. Pen testing that results in DOS, then, is extremely, specifically bad. If you’re signing a contract for pen testing, make sure it includes provisions that testing be done during non-business hours, if you have off hours.


Costs are always an issue of balance: What does it cost you to fail to comply? You’d better be very clear on your legal requirements to answer that question. What does it cost you to audit or pen test? Probably, but not certainly, less. The issue is that you’re not playing poker, where there are odds and perhaps sustainable losses. You’re playing Russian roulette, where loss means the potential for total destruction of your business or even more devastating losses for your customers, clients or patients. If you think I’m trying to scare you to lessen any sticker shock, I am.

For a full-scale, mil-spec pen test against a large organization, expect price tags somewhere in the $15,000-25,000 (each) range for mandatory thrice-annual tests. The critical thing here is that setup is the biggest expense (i.e. takes the longest time), so a single-incident pen test for a smaller business could easily approach or surpass this price tag, depending on the scope of testing. This makes understanding your scope, which is to say your compliance requirements, the critical point.

Even more, because pen testers are in strong demand, at least in certain sectors, most of them don’t want to deal with smaller businesses. The risks aren’t worth the legal issues, which are substantial. This means those smaller orgs are often better served by training internal staff to perform pen testing than they are by hiring outside contractors. In some cases this doesn’t fulfill legal requirements for testing to be performed by a separate institution, but if you’re at a scale that requires full-scale external-provider pen testing for compliance, you already know this.


The landscape is changing very rapidly here. If you’re hosting all your servers and services internally, serious pen testing could temporarily shatter your working infrastructure. Do not ask me how I know this. In some situations this in unavoidable because extreme security or data location requirements force you to do your own hosting this way.

On the other hand, if you’re utilizing contemporary infrastructure there’s no reason you should have significant or any downtime. Host your documentation on Google and your pen tester is testing Google, not you (which will get them in some serious trouble). Host your servers on Amazon and they’re testing Amazon’s cloud resiliency, and asking for some very unwelcome attention.

Yes, keep your secret sauce on your own hardware, but otherwise don’t run your own steam engines, generators or servers. Don’t worry, though. One round of pen testing (really, vuln testing) will show you where the easy openings are. Just remember that if your pen testers bring you down during operating hours, they’re doing their job poorly (with the notable exception of 24-hour operations).

Marc is, and you, gentle reader, are also welcome to contact me if you have questions, want to know more, or need pen testing or training services.

Test safely.

* * *