Clouds and Vulnerability Management

In the world of Clouds and Vulnerability Management, based on observations, it seems like a critical issue has slipped under the radar: if you’re running with PaaS and SaaS VMs, you cannot deliver anything close to a respectable level of vulnerability management with these platforms. This is because to do effective vulnerability management, the first part of that process – the vulnerability assessment – needs to be performed with administrative access (over SSH/SMB), and with PaaS and SaaS, you do not, as a customer, have such access (this is part of your agreement with the cloud provider). The rest of this article explains this issue in more detail.

The main reason for the clouding (sorry) of this issue, is what is still, after 20+ years, a fairly widespread lack of awareness of the ineffectiveness of unauthenticated vulnerability scanning. More and more security managers are becoming aware that credentialed scans are the only way to go. However, with a lack of objective survey data available, I can only draw on my own experiences. See – i’m one of those disgraceful contracting/consultant types, been doing security for almost 20 years, and been intimate with a good number of large organisations, and with each year that passes I can say that more organisations are waking up to the limitations of unauthenticated scanning. But there are also still lots more who don’t clearly see the limitations of unauthenticated scanning.

The original Nessus from the late 90s, now with Tenable, is a great product in terms of doing what it was intended to do. But false negatives were never a concern in with the design of Nessus. OpenVAS is still open source and available and it is also a great tool from the point of view of doing what it was intended to do. But if these tools are your sole source of vulnerability data, you are effectively running blind.

By the way Tenable do offer a product that covers credentialed scans for enterprises, but i have not had any hands-on experience with this tool. I do have hands on experience with the other market leaders’ products. By in large they all fall some way short but that’s a subject for another day.

Unauthenticated scanners all do the same thing:

  • port scan to find open ports
  • grab service banners – this is the equivalent of nmap -sV, and in fact as most of these tools use nmap libraries, is it _exactly_ that
  • lets say our tool finds Apache HTTP 14.x, it looks in its database of public disclosed vulnerability with that version of Apache, and spews out everything it finds. The tools generally do little in the way of actually probing with HTTP Methods for example, and they certainly were not designed to try, for example, a buffer overflow exploit attempt. They report lots of ‘noise’ in the way of false positives, but false negatives are the real concern.

So really the tools are doing a port scan, and then telling you you’re running old warez. Conficker is still very widespread and is the ultimate player in the ‘Pee’ arena (the ‘Pee’ in APT). An unauthenticated scanner doesn’t have enough visibility ‘under the hood’ to tell you if you are going to be the next Conficker victim, or the next ransomware victim. Some of the Linux vulnerabilities reported in the past few years – e.g. Heartbleed, Ghost, DirtyCOW – very few can be detected with an unauthenticated scanner, and none of these 3 examples can be detected with an unauthenticated scanner.

Credentialed scanning really is the only way to go. Credentialed based scanners are configured with root/administrative access to targets and are therefore in a position to ‘see’ everything.

The Connection With PaaS and SaaS

So how does this all relate to Cloud? Well, there two of the three cloud types where a lack of access to the operating system command shell becomes a problem – and from this description its fairly clear these are PaaS and SaaS.

 There are two common delusions abound in this area:

  • [Cloud maker] handles platform configuration and therefore vulnerability for me, so that’s ok, no need to worry:
    • Cloud makers like AWS and Azure will deal with patches, but concerns in security are much wider and operating systems are big and complex. No patches exist for 0days, and in space, nobody can hear you scream.
    • Many vulnerabilities arise from OS configuration aspects that cannot be removed with a patch – e.g. Conficker was mentioned above: some Conficker versions (yes its managed very professionally) use ‘at’ job scheduling to remain present even after MS08-067 is patched. If for example you use Azure, Microsoft manage your PaaS and SaaS but they don’t know if you want to use ‘at’ or not. Its safer for them to assume that you do want to use it, so they leave it enabled (when you sign up for PaaS or SaaS you are removed from the decision making here). Same applies to many other local services and file system permissions that are very popular with the dark side.
  • ‘Unauthenticated scanning gets me some of the way, its good enough’ – how much of the way does it get you? Less than half way? its more like 5% really. Remember its little more than a port scan, and you shouldn’t need a scanner to tell you you’re running old software. Certainly for critical cloud VMs, this is a problem.

With PaaS and SaaS, you are handing over the management of large and complex operating systems to cloud providers, who are perfectly justified, and also in many cases perfectly wise, in leaving open large security holes in your platforms, and as part of your agreement with them, there’s not a thing you can do about it (other than switch to IaaS or on-premise).

Scangate Re-visited: Vulnerability Scanners Uncovered

I have covered VA tools before but I feel that one year later, the same misconceptions prevail. The notion that VA tools really can be used to give a decent picture of vulnerability is still heavily embedded, and that notion in itself presents a serious vulnerability for businesses.

A more concise approach at a run down on the functionality of VA warez may be worth a try. At least lets give it one last shot. On second thoughts, no, don’t shoot anything.

Actually forget “positive” or “negative” views on VAs before reading this. I am just going to present the facts based on what I know myself and of course I’m open to logical, objective discussion. I may have missed something.

Why the focus on VA? Well, the tools are still so commonplace and heavily used and I don’t believe that’s in our best interests.

What I discovered many years ago (it was actually 2002 at first) was that discussions around these tools can evoke some quite emotional responses. “Emotional” you quiz? Yes. I mean when you think about it, whole empires have been built using these tools. The tools are so widespread in security and used as the basis of corporate VM programs. VM market revenues runs at around 1 billion USD annually. Songs and poems have been written about VAs – OK I can’t back that up, but careers have been built and whole enterprise level security software suites built using a nasty open source VA engine.

I presented on the subject of automation in VA all those years ago, and put forward a notion that running VA tools doesn’t carry much more value as compared to something like this: nmap -v -sS -sV <targets> . Any Security Analyst worth their weight in spam would see open ports and service banners, and quickly deduce vulnerability from this limited perspective. “Limited”, maybe, but is a typical VA tool in a better position to interrogate a target autotragically?

One pre-qualifier I need to throw out is that the type of scanners I will discuss here are Nessus-like scanners, the modus operandi of which is to use unauthenticated means to scan a target. Nessus itself isn’t the main focus but it’s the tool that’s most well known and widely used. The others do not present any major advantages over Nessus. In fact Nessus is really as good as it gets. There’s a highly limited potential with these tools and Nessus reaches that limit.

Over the course of my infosec career I have had the privilege to be in a position where I have been coerced into using VAs extensively, and spent many long hours investigating false positives. In many cases I set up a dummy Linux target and used a packet sniffer to deduce what the tool was doing. As a summary, the findings were approximately:

  • Out of the 1000s of tests, or “patterns”, configured in the tools, only a few have the potential to result in accurate/useful findings. Some examples of these are SNMP community string tests, and tests for plain text services (e.g. telnet, FTP).
  • The vast majority of the other tests merely grab a service “banner”. For example, the tool port scans, finds an open port 80 TCP, then runs a test to grab a service banner (e.g. Apache 2.2.22, mickey mouse plug-in, bla bla). I was sort of expecting the tool to do some more probing having found a specific service and version, but in most cases it does not.
  • The tool, having found what it thinks is a certain application layer service and version, then correlates its finding with its database of public disclosed vulnerabilities for the detected service.

Even for some of the plan text services, some of the tests which have the potential to reveal useful findings have been botched by the developers. For example, tests for anonymous FTP only work with a very specific flavour of FTP. Other FTP daemons return different messages for successful anonymous logins and the tool does not accommodate this.

Also what happens if a service is moved from its default port? I had some spectacular failures with running Nessus against a FTP service on port 1980 TCP (usually it is listening on port 21). Different timing options were tested. Nessus uses a nmap engine for port scanning, but nmap by itself is usually able to find non-default port services using default settings.

So in summary, what the VA tools do is mostly just report that you are running ridiculous unencrypted blast-from-the-past services or old, down-level services – maybe. Really I would hope security teams wouldn’t need to spend 25K USD on an enterprise solution to tell them this.

False positives is one thing, but false negatives is quite another. Popular magazines always report something like 50% success rate in finding vulnerabilities in staged tests. Why is it always 50%? Remember also that the product under testing is usually one from a vendor who pays for a full spread ad in that magazine.

Putting numbers to false negatives makes little sense with huge, complex software packages of millions of lines of source code. However, it occurred to me not so long ago whilst doing some white box testing on a client’s critical infrastructure: how many of the vulnerabilities under testing could possibly be discovered by use of a VA tool? In the case of Oracle Database the answer was less than 5%. And when we’re talking Oracle, we’re usually talking critical, as in crown jewels critical.

If nothing else, the main aspect I would hope the reader would take out of this discussion is about expectation. The expectation that is set by marketing people with VA tools is that the tools really can be used to accurately detect a wide range of vulnerability, and you can bet your business on the tools by using them to test critical infrastructure. Ladies and gentlemen: please don’t be deceived by this!

Can you safely replace manual testing with use of these tools? Yes, but only if the target has zero value to the business.

 

What’s Next For BYOD – 2013 And Beyond

There are security and business case arguments about BYOD. They cast different aspects and there’s peta-bytes of valid points out there.

The security argument? Microsoft Windows is still the corporate OS of choice and still therefore the main target for malware writers. As a pre-qualifier – there is no bias towards one Operating System or another here.

Even considering that in most cases, when business asks for something, security considerations are secondary, there is also the point that Windows is by its nature, very hard to make malware-resistant. Plenty of malware problems are not introduced as a result of a lack of user awareness (for example, unknowingly installing malware in the form of faked anti-virus or browser plug-ins), plus plenty of services are required to run with SYSTEM privileges. These factors make Windows platforms hard to defend in a cost-effective, manageable way.

Certainly we have never been able to manage user OS rights/privileges and that isn’t going to change any time soon. There is no 3rd party product that can help. Does security actually make an effective argument in cases where users are asking for control over printers and Wifi management? Should such functions be locked anyway? Not necessarily. And once we start talking fine-grained admin rights control we’re already down a dark alley – at least security needs to justify to operations as to why they are making their jobs more difficult, the environment more complex and therefore less reliable. And with privilege controls, security also must justify to users (including C-levels) as to why their corporate device is less usable and convenient.

For the aforementioned reasons, the security argument is null and void. I don’t see BYOD as a security argument at all, mainly because the place where security is at these days, isn’t a place where we can effectively manage user device security – the doesn’t change with or without BYOD, and this is likely to be the case for some years to come yet. We lost that battle, and the security strategy has to be planned around the assumption that user subnets are compromised. I would agree that in a theoretical case where user devices are wandering freely, not at all subject to corporate controls, then the scope is there for a greater frequency of malware issues, but regardless, the stance has to be based on an assumption that one or more devices in corporate subnets has been compromised and the malware is designed to connect ingress and egress.

How about other OS flavors, such as Apple OS X for example? With other OS flavors, it is possible to manage privileges and lock them down to a much larger degree than we can with Windows, but as has been mentioned plenty of times now, once another OS goes mainstream and grows in corporate popularity, then it also shows up on the radars of malware writers. Reports of  malware designed to exploit vulnerabilities in OS X software started surfacing earlier in 2012, with “The Flashback Trojan” given the widest coverage.

I would venture that at least the possibility exists to use technical controls to lock down Unix-based devices to a much larger degree, as compared with MS Windows variants, but of course the usability experience has to match the needs of business. Basically, regardless of whether our view is utopic or realistic, there will be holes, and quite sizable holes too.

For the business case? Having standard build user workstations and laptops does make life easier for admins, and it allows for manageability and efficiency, but there is a wider picture of course. The business case for BYOD is a harder case to make than we might have at first thought. There are no obvious answers here. One of the more interesting con articles was from CIO Magazine earlier in 2012: BYOD: If You Think You’re Saving Money, Think Again and then Cisco objectively report that there are plenty in the pro corner too: Cisco Study: IT Saying Yes To BYOD.

So what does all this bode for the future? The manageability aspect and therefore the business aspect is centered around the IT costs and efficiency analysis. This is more of an operational discussion than an information risk management discussion.

The business case is inconclusive, with plenty in the “say no to BYOD” camp. The security picture is without foundation – we have a security nightmare with user devices, regardless of who owns the things.

Overall the answer naturally lies in management philosophy, if we can call it that. There is what we should do, and what we will do….and of course these are often out by 180 degrees from each other. The lure of BYOD will be strong at the higher levels who usually only have the balance sheet as evidence, along with the sales pitches of vendors. Accountant-driven organisations will love the idea and there will be variable levels of bravery, confidence, and technical backing in the IT rationalization positions. Similar discussions will have taken place with regard to cloud’ing and outsourcing.

The overall conclusion: BYOD will continue to grow in 2013 and probably beyond. Whether that should be the case or not? That’s a question for operations to answer, but there will be plenty of operations departments that will not support the idea after having analyzed the costs versus benefits picture.

Migrating South: The Devolution Of Security From Security

Devolution might seem a strong word to use. In this article I will be discussing the pros and cons of the migration of some of the more technical elements of information security to IT operations teams.

By the dictionary definition of the word, “devolution” implies a downgrade of security – but sufficed to say my point does not even remotely imply that operations teams are subordinate to security. In fact in many cases, security has been marginalized such that a security manager (if such a function even exists) reports to a CIO, or some other managerial entity within IT operations. Whether this is right or wrong…this is subjective and also not the subject here.

Of course there are other department names that have metamorphosed out of the primordial soup …”Security Operations” or SecOps, DevOps, SecDev, SecOpsDev, SecOpsOps, DevSecOps, SecSecOps and so on. The discussion here is really about security knowledge, and the intellectual capital that needs to exist in a large-sized organisation. Where this intellectual capital resides doesn’t bother me at all – the name on the sign is irrelevant. Terms such as Security and Operations are the more traditional labels on the boxes and no, this is not something “from the 90s”. These two names are probably the more common names in business usage these days, and so these are the references I will use.

Examples of functions that have already, and continue to be pharmed out to Ops are functions such as Vulnerability Management, SIEM, firewalls, IDS/IPS, and Identity Management. In detail…which aspects of these functions are teflonned (non-stick) off? How about all of them? All aspects of the implementation project, including management, are handled by ops teams. And then in production, ops will handle all aspects of monitoring, problem resolution, incident handling ..ad infinitum.

A further pre-qualification is about ideal and actual security skills that are commonly present. Make no mistake…in some cases a shift of tech functions to ops teams will actually result in improvements, but this is only because the self-constructed mandate of the security department is completely non-tech, and some tech at a moderate cost will usually be better than zero tech, checklists, and so on.

We need to talk about typical ops skills. Of course there will be occasional operations team members who are well versed in security matters, and also have a handle on the business aspects, but this is extra-curricular and rare. Ops team members are system administrators usually. If we take Unix security as an example, they will be familiar with at least filesystem permissions and umask settings, so there is a level of security knowledge. Cisco engineers will have a concept of SNMP community strings and ACLs. Oracle DBAs will know how about profiles and roles.

But is the typical security portfolio of system administrators wide enough to form the foundations of an effective information security program? Not really. In fact its some way short. Security Analysts need to have a grasp not only on, for example, file system permissions, they need to know how attackers actually elevate privileges and compromise, for example, a critical database host. They need to know attack vectors and how to defend against them. This kind of knowledge isn’t a typical component of a system administrator’s training schedule. Its one thing to know the effect of a world-write permission bit on a directory, but what is the actual security impact? With some directories this can be relatively harmless, with others, it can present considerable business risk.

The stance from ops will be to patch and protect. While this is [sometimes] better than nothing, there are other ways to compromise servers, other than exploiting known vulnerabilities. There are zero days (i.e. undeclared vulnerabilities for which no patch has been released), and also means of installing back doors and trojans that do not involve exploiting local bugs.

So without the kind of knowledge I have been discussing, how would ops handle a case where a project team blocks the install of a patch because it breaks some aspect of their business-critical application? In most cases they will just agree to not install the patch. In consideration of the business risk several variables come into play. Network architecture, the qualitative technical risk to the host, value of information assets…and also is there a work-around? Is a work-around or compromise even worth the time and effort? Do the developers need to re-work their app at a cost of $15000?

A lack of security input in network operations leads to cases where over-redundancy is deployed. Literally every switch and router will have a hot swap. So take the budget for a core network infrastructure and just double it – in most cases this is excessive expenditure.

With firewall rules, ops teams have a concept of blocking incoming connections, but its not unusual that egress will be over-looked, with all the “bad netizen”, malware / private date harvests, reverse telnet implications. Do we really want our corporate domain name being blacklisted?

Another common symptom of a devolved security model is the excessive usage of automated scanners in vulnerability assessment, without having any idea that there are shortcomings with this family of product. The result of this is to “just run a scanner against it” for critical production servers and miss the kind of LHF (Low Hanging Fruit) false negatives that bad guys and malware writers just love to see.

The results of devolution will be many and varied in nature. What I have described here is only a small sampling. Whatever department is responsible for security analysis is irrelevant, but the knowledge has to be there. I cover this topic more thoroughly in Chapter 5 of Security De-engineering, with more details on the utopic skills in Chapter 11.

The Perils Of Automation In Vulnerability Assessment

Those who have read my book will be familiar with this topic, but really speaking even if literally everyone had read the book already, I would still be covering this matter because the magnitude of the problem demands coverage, and more coverage. Even when we’re at the point of “we the 99% do understand that we really shouldn’t be doing this stuff any more”, the severity of the issue demands that even if there should still be a lingering one per cent, yet further coverage is warranted.

The specific area of information security in which automation fails completely (yet we still persist in engaging with such technology) is in the area of vulnerability scanning, in particular unauthenticated vulnerability scanning, in relation to black box scanning of web applications and networks. “Run a scanner by it” still appears in so many articles and sound bytes in security – its still very much part of the furniture. Very expensive, software suites are built on the use of automated unauthenticated scanning – in some cases taking an open source scanning engine, wrapping a nice GUI around it with pie charts, and slapping a 25K USD price tag on it.

As of 2012 there are still numerous supporters of vulnerability scanning. The majority still seem to really believe the premise that it is possible (or worse…”best practices”), by use of unauthenticated vulnerability scanning, to automatically deduce a picture of vulnerability on a target – a picture that does not come with a bucket load of condiments in the way of significant false negatives.

False positives are a drain on resources – and yes, there’s a bucket load of those too, but false negatives, in critical situations, is not what the doctor ordered.

Even some of the more senior folk around (note: I did not use the word “Evangelist”) support the use of these tools. Whereas none of them would ever advocate substituting manual penetration testing for an auto-scan, there does seem to be a great deal of “positivity” around the scanning scene. I think this is all just the zen talking to be honest, but really when we engage with zen, we often disengage with reality and objectivity. Its ok to say bad stuff occasionally, who knows, it might even be in line with the direction given to one’s life by one’s higher consciousness.

Way back in the day, when we started off on our path of self-destruction, I ran a pressie on auto-scanning and false expectations, and I duly suffered the ignominy of the accusation of carrying Luddite tendencies. But…thing is see: we had already outsourced our penetration testing to some other firm somewhere – so what was it that I was afraid of losing? Yes, I was a manual tester person, but it was more than 12 months since we outsourced all that jazz – and I wasn’t about to start fighting to get it back. Furthermore, there were no actual logical objections put forward. The feedback was little more than just primordial groans and remote virtual eye rolling – especially when I displayed a chart that showed unauthenticated scanning carrying similar value to port scanning. Yes – it is almost that bad.

It could be because of my exposure to automated scanners that I was able to see the picture as clearly as I did. Actually in the first few runs of a scanning tool (it was the now retired Cybercop Scanner – it actually displayed a 3D rotating map of a network – well, one subnet anyway) I wasn’t aware myself of the lack of usefulness of these tools. I also used other tools to check results, but most of the time they all returned similar results.

Over the course of two years I conducted more than one hundred scans of client perimeters and internal subnets, all with similar results. During this time I was sifting thru the endless detritus of false positives with the realization that in some cases I was spending literally hours dissecting findings. In many cases it was first necessary to figure out what the tool was actually doing in deducing its findings, and for this I used a test Linux box and Ethereal (now Wireshark).

I’m not sure that “testing” as in the usage of a verb is appropriate because it was clear that the tool wasn’t actually doing any testing. In most cases, especially with listening services such as Apache and other webservers, the tool just grabs a banner, finds a version string, and then does a correlation look-up in its database of public declared vulnerability. What is produced is a list of public declared vulnerability for the detected version. No actual “probing” is conducted, or testing as such.

The few tests that produce reasonably reliable returns are those such as SNMP community strings tests (or as reliable as UDP allows) or another Blast From The Past – finger service “intelligence” vulnerability (no comment). The tools now have four figure numbers of testing patterns, less than 10% of which constitute acceptably accurate tests. These tools should be able to conduct some FTP configuration tests because it can all be done with politically correct “I talk to you, you talk to me, I ask some questions, you give me answers” type of testing. But no. Something like a test for anonymous FTP enabled – works for a few FTP servers, but not for some of the other more popular FTP packages. They all return different responses to the same probe you see…

I mentioned Cybercop Scanner before but its important not to get hung up on product names. The key is the nature of the scanning itself and its practical limitations. Many of our beloved security softwares are not coded by devs who have any inkling whatsoever of anything to do with security, but really, we can have a tool deduced and produced with all the miracles that human ingenuity affords, but at some point we always hit a very low and very hard ceiling, in terms of what we can achieve with unauthenticated vulnerability assessment.

With automated vulnerability assessment we’re not doing anything that can destabilize a service (there are some DoS tests and “potentially disruptive tests” but these are fairly useless). We do not do something like running an exploit and making shell connection attempts, or anything of the sort. So what we can really achieve will always be extremely limited. Anyway, why would we want to do any of this when we have a perfectly fine root account to use? Or is that not something we really do in security (get on boxes and poke around as uid=0)? Is that ops ninja territory specifically (See my earlier article on OS Security, and as was said recently by a famous commentator in our field: “Platforms bitches!”)?

The possibility exists to check everything we ever needed to check with authenticated scanning but here, as of 2012, we are still some way short – and that is largely because of a lack of client demand (crikey)! Some spend a cajillion on a software package that does authenticated testing of most popular OSs, plus unauthenticated false positive generation, and _only_ use the sophisticated resource intensive false positives generation engine – “that fixes APTs”.

The masses seem to be more aware of the shortcomings with automated web application vulnerability scanners, but anyway, yes, the picture here is similarly harsh on the eye. Spend a few thousand dollars on these tools? I can’t see why anyone would do that. Perhaps because the tool was given 5 star ratings by unbiased infosec publications? Meanwhile many firms continue to bet their crown jewels on the use of automated vulnerability assessment.

The automobile industry gradually phased in automation over a few decades but even today there are still plenty of actual homo sapiens working in car factories. We should only ever be automating processes when we can get results that are accurate within the bounds of acceptable risks. Is it acceptable that we use unauthenticated automated scanning as the sole means of vulnerability assessment with the top 20% of our most critical devices? It is true that we can never detect every problem and what is safe today, maybe not safe tomorrow. But also we don’t want to miss the most glaring critical vulnerabilities either – but this is exactly the current practice of the majority of businesses.