About six years ago, social engineering penetration tests became the norm for the A-Team.  In many of these tests, our team would attempt as many as 10-20 unique exploits against various applications and operating system functions.  This often included exploits against Flash, Java, Adobe Reader, MS Office and IE.  While one or two exploit attempts may have succeeded, the majority would fail.  When it came time to present our report to the client, it would inevitably focus on the successful compromises and their associated vulnerabilities.  In almost every case, our clients would ask about the attacks that failed and what control(s) prevented them.

Consider the following...  I email you an infected PDF file with embedded Java script that, if you are running a vulnerable version of Adobe Reader, will provide me a command and control shell.  If I get a shell back, I know that the recipient received the attachment, they that opened it, that their system had vulnerable software, and that all of their controls (including security awareness) failed to prevent the infection.  However, if I never got a shell, it would be difficult or impossible to determine why the attack failed.  It could be that an email gateway (cloud or on premise) blocked the attachment, antivirus software on the Exchange server could have caught it, antivirus on the desktop, end-user security awareness, etc.  It's also likely that the end-users’ system was patched for the vulnerability I was trying to exploit.  Or, the recipient may have opened the infected document and successfully exploited their system.  In this case, egress filters like web gateways, firewalls, and proxies could also have prevented the command and control communications.  In any of these cases, I have little or no evidence as to why my attack failed.

It was this problem that lead to a unique and valuable solution:  Why not use penetration testing software and exploits to validate controls rather than to just exploit vulnerabilities?  We started by just re-sending the same exploits that we attempted during our social engineering penetration tests, but instead of attacking the workstations of unsuspecting end-users, we sent the exploits to our client's IT security staff.  Then, while receiving emails with infected attachments and while clicking links to browser-based exploits, our clients would monitor their controls to determine which control successfully prevented each attack.  This quickly evolved into much larger groups of exploits and a systematic approach to validating the effectiveness of the technical controls that protect end-user systems.

Today, we refer to this process as a Rapid Hybrid Pentest.  Using commercial penetration testing software, like Metasploit and Core Impact, we generate 25-30 unique exploits.  The exploits target all of the most common software and include both web-based and email-based attacks.  In general, we try to match the exploits up with the vulnerabilities currently being exploited by malware in the wild.  We deliver all of the links to our clients and have them click on them one-by-one as they monitor their controls to determine which get caught and which slip through.  Within a couple hours, we are able to determine which controls work, which are misconfigured, and which don't work at all.  While we've developed this into both a professional service as well as a self-service web application, the process is simple and can be done by anyone with a copy of Metasploit.