Operating System Security is radically under-appreciated in security and this has been the case since the big bang of business security practices in the mid-90s. OS security, along with application security is now the front line in the battle against hackers, but this point has not been widely realised…

Terminology

We have a lot of terms in security that have no commonly understood definition, and it seems everybody’s version of the definition is the correct one. First up, what is meant by “Operating System?” The usual meaning in most businesses’ operational sense is a bare install – in other words, the box as in its shipped state, with no in-house or any other applications installed – there will [hopefully!] be nothing installed on the computer other than what is made available from the vendor’s install media. This is different from most university Computer Science course syllabus definitions of the phrase, but we will go with the former definition.
So what of “operating system security” then? Computer operating systems are designed with configuration options, such as file system permissions, that allow administrators to apply some level of protection to information (files, databases, etc) hosted by the computer. Operating system security relates to the degree to which available controls have been applied, in the face of remote and local exploit risks.

Vulnerability Assessment: Current Perceptions and Misconceptions

Mostly, when people think of vulnerability assessment the first thing that comes to mind is network penetration testing or use of automated scanning tools. However, cutting a long story short, neither of these two approaches give us a useful or efficient way of assessing vulnerability for our critical infrastructure.
Penetration testing these days is mostly just performed as a requirement of auditors, and to be too analytical means to be slower, and most businesses will not tolerate this. The quality of delivery is poor, and furthermore the tests are so restricted as to make them close to useless. Usually the only useful item of information to come out of these engagements is the port scan results.
The whole story on automated scanners is a long one (for a longer discussion on this matter refer to Chapter 5 of Security De-engineering) but just to summarize: unauthenticated scanning of critical infrastructure with no further analysis is a recipe for disaster. The marketing engine behind such products claims they can replace manual efforts. Such a claim suits the agenda of many in the security industry but overall it will be the security industry’s customers who will suffer – and are suffering. The expectations were set way too high with these tools. Again, the most valuable output from these tools will be the port scan results.
Further have been made in the way of products misnomered in the Vulnerability Management genre (“management”? – vulnerability is not managed it is only enumerated) and some of these offer authenticated scanning. While there have been some recent improvements in this area, the most important items of OS security remain unchecked by these tools. Furthermore databases such as Oracle are given scant coverage.

Why Analyze OS Security Controls?

With regard the subsection heading “Why Analyze OS Security Controls”, there are two categories of answer to this question. The first goes something like “because we have perfectly fine security standards, signed off by our CEO, that tells us we need to analyze security controls”. The second type of answer is related to technical risk and the efficiency of our vulnerability management programs – and this is the subject of the remainder of this article.
I mentioned the limitations of penetration testing previously and it should not be seen as a panacea. In a scenario where internal IT staff, including security personnel, do not have intimate knowledge of the IT landscape, the two-week penetration test costing 40K USD will barely touch the surface. The gap in knowledge will be filled slightly by a penetration test, but the only scenario where a penetration test can be valuable is one where both target staff and penetration testers are highly experienced in their field. In this case the 40K USD, 40 man-day test is used to try and spot misconfigurations that the internal staff may have missed. This is a good use of funds in most cases. Any other scenario is unlikely to provide much value for businesses. In summary, penetration testing is not the answer for businesses.
We have heard a great deal about APT in recent months. APT has been attributed with many of the recent high profile attacks. Malware is released at rates faster than the anti-virus software vendors can release pattern updates. Ever more ingenious malware is being created by the bucket-load. Then there is the problem of unaware users. Regardless of whatever awareness program or punishment businesses deal out, there will always be some user somewhere who clicks on some click they shouldn’t follow, and corporate policies that disable Javascript or blacklist URLs are doomed to failure.
Overall…the perimeter has shifted. The perimeter is no longer the perimeter if you see what I mean, in that it is no longer the border or choke point external firewalls. Business workstation subnets are owned by Botnet R Us.
Many of the attacks are carried out with undisclosed vulnerabilities and because they are undisclosed to the public, there is no patch available to mitigate the software vulnerability. This is the point where many analysts raise a white flag. But this is also the point where OS controls can save the day in many cases. At least we can say that thoughtful use of OS security controls can prevent a business from becoming a “low hanging fruit”.
Taking a Unix system as an example: an attacker may have remotely compromised a listening service but they only gain the privileges of the listening service process owner. In most cases this is not a root compromise. The attacker will need to elevate their local privileges in most cases. How do they do this? They look for bad file system permissions and anything running under root privileges. They look for anything related to the root account, such as cron jobs owned by lower privileged users that were configured to run under root’s cron.
With effective controls on server operating systems we have the possibility to severely restrict privilege escalation opportunities, even in cases where zero-day/undisclosed vulnerabilities are used by attackers. Some services can even be “chroot jail” configured, but this requires knowledge of “under the hood” operating system controls.
So think of the analysis of operating system controls as a kind of a twist on a penetration test – sort of a penetration test inside out. The approach is “I have compromised the target, now how I would I compromise the target?” Using a root / administrative account allows a vastly more efficient situation where a great deal more can be learned about a system in a short period of time as compared with a remote penetration test.
I mentioned previously about the new perimeter in corporate networks – the perimeter is now critical infrastructure. Many businesses do not have internal segmentation. There is only a DMZ subnet and then a flat internal private network with no further network access control. But assuming that they do have internal network access control, then we can imagine that the new perimeter is the firewall between workstation subnets and critical servers such as database servers and so on. The internal firewall(s) makes up the perimeter along with…you got it…the operating system of the database, LDAP, AD, and other critical application hosts [penny drops].
Businesses can proportion the resources deployed in operating system control security assessment depending on the criticality of the device. The criticality will depend on a number of factors, not least network architecture. In the case of a flat private network with no internal segmentation, effectively every device is a critical device and the budget required for security is going to be much higher than in the case where segments exist at differing levels of business criticality.
Hopefully then the importance of operating system and database security controls has been made clearer. Thoughtful deployment of automated and manual analysis in this area is a nice efficient use of corporate resources with huge returns in risk mitigation. The longer terms costs of information risk management will be less where better targeting of resources is deployed – and certainly, some of the suggestions for vulnerability assessment I have outlined in this article can help a great deal in reducing the costs of vulnerability management for businesses.