Microsoft Secure Score. If you’re an IT administrator or security professional in an organization that uses Office 365, then you’ve no doubt used the tool or at least heard the term. It started as Office 365 Secure Score, but it was renamed in April 2018 to reflect a wider range of elements being scored.
What does it do? The tool looks at configurable settings and actions primarily within your Office 365 and Azure AD environment, and awards points for selections that meet best practices. In their words, “From a centralized dashboard you can monitor and improve the security for your Microsoft 365 identities, data, apps, devices, and infrastructure.”
But what doesn’t Microsoft Secure Score do? Microsoft is very good at telling you the great things its products can do, so I won’t repeat them here. The concept is sound, and I applaud them for giving users a tool that prioritizes secure configurations. They have come a long way from having auditing turned off by default in their products, e.g., Server 2000. I will point out why Microsoft Secure Score isn’t enough when it comes to understanding and testing the security of your Microsoft 365 environment.
I am a Certified Public Accountant (CPA), and as such, I’ve spent a good portion of my life performing audits and assessments. A key independence rule CPAs abide by is: an auditor must not audit his or her own work. Microsoft isn’t exactly independent when scoring its own product’s settings and capabilities. The financial motivation exists for Microsoft to setup a scoring system that makes users feel good about using Microsoft products. Interoperability and performance will always be a higher priority than security.
This fact is furthered by the scoring system setup, which unlocks higher point opportunities with higher priced subscriptions. For example, Microsoft Cloud App Security and Azure Advanced Threat Protection are unlocked with E5 licenses, or as a $5.50 per user per month add on to an existing E3 license. This can be as much as a 70% price increase. If you want more chances to raise your overall score and have a higher score ceiling, spend more money…a very beneficial side-effect for Microsoft.
Also, remember that Secure Score is reflective of a Microsoft opinion and their subjective value for security controls they believe are important. This differs from widely accepted standards from organizations like NIST (National Institute of Standards and Technology) or CIS (Center for Internet Security) which are vendor neutral and have been refined, improved, and evolved over time.
First let me say that Secure Score can be dented and bent to fit different environments. Scoring for certain areas can be manually entered if you have a third-party solution for a control. It will be incumbent on the person checking those controls to match what Secure Score is asking for. This is an all-or-nothing proposition as indicated within Secure Score, “Marking as resolved through third-party indicates that you have completed this action in a non-Microsoft app, and will give you the full point value of this action.”
This is a key area where the Secure Score blanket fails to keep all areas of the entity covered and warm. There are bound to be components and configuration requirements that don’t quite fit what Secure Score evaluates or how it is scored. Think of the myriad of application combinations to handle Customer Relationship Management (CRM), Mobile Device Management (MDM), Security Information and Event Management (SIEM), Data Loss Prevention (DLP), and Multifactor Authentication (MFA) just to name a few. An independent assessment of the environment that references best practice hardening guides for specific products comprising the solution is the only way to complete a proper evaluation.
Reason number 3: Security is a journey, and a scorecard makes it a destination.
Don’t get me wrong, I like scores and grades. CPA’s generally like to measure and quantify things. Secure Score quantifies security, gives you trends over time on your score, and even allows you to measure your score against others based on a global average, industry average, and similar seat count average.
What I don’t like is how the scores can be manipulated, or how they can be construed. If the O365 administrator wants to improve their percentage of points achieved, the simplest way is to select “ignore” for the scoring areas that they have earned 0 points. Per Secure Score documentation, “Once you ignore an improvement action, it will no longer count toward the total Secure score points you have available.” Lower the denominator, keep the numerator, and poof! We are more secure. Or are we?
Executives looking at a scorecard may also be satisfied once it has reached a certain percentage of the total available. A project which will move the Secure Score from 650 out of 807 points to 710 out of 807 points appears to make the company about 8% more secure to a non-security decision maker handling the company budget. That project may not make the cut. In reality, any scoring shortage could represent a critical configuration issue that puts information assets at risk. That point may get lost if the focus is score.
We hear stories all the time about breach activities that were being reported by automated logging systems, except no one was looking at the logs. IT management puts a tool in place and checks a box that implies the organization is secure in that area. Secure Score is ripe for this. Several improvement actions that will increase your score involve reviewing reports. When a link for a report is clicked, Secure Score assumes the report was reviewed and awards points. To keep the points, the link must be clicked within specific time intervals from within the Secure Score user interface, but this process does not record what was reviewed, or any notes or actions resulting from the review. There is no substitute for the actual review process and confirming that the review is happening.
Also consider an environment made up of multiple applications from different vendors where automated security evaluations, like Secure Score, are put in place. Each application that makes up the system interacts with other applications, potentially creating security control blind spots. For example, an email system that hands-off outbound email to a 3rd party DLP solution. Are there security holes in the process that transfers data in and out of the DLP application? Identifying those weaknesses requires a wholistic view, measured against current accepted best practices, that just isn’t offered by Secure Score or any other automated solution.
In conclusion, I think Secure Score has a place in monitoring and evaluating an organization’s information security posture. Microsoft is taking recommendations from its user base and is working to improve Secure Score’s results and widen its coverage. It is a barometer of an information security environment that could produce important information when properly utilized.
The bottom line though is that it is just one tool. It cannot replace a diligent information security program; or at a higher level, an information security management system. Independent assessment and review of controls, policies, procedures, and the people managing the environment work in tandem to assure the confidentiality, integrity, and availability of an organizations information assets. Consider the diversity of an organizations landscape:
These areas are all interdependent, yet all have their own unique traits and ways to be assessed and secured. No one measurement tool is enough.
By Jeffrey T. Lemmermann, CPA, CISA, CITP, CEH - Information Assurance Consultant
