On Hiring For DevSecOps

Based on personal experience, and second hand reports, there’s still some confusion out there that results in lots of wasted time for job seekers, hiring organisations, and recruitment agents.

There is a want or a need to blame recruiters for any hiring difficulties, but we need to stop that. There are some who try to do the right thing but are limited by a lack of any sector experience. Others have been inspired by Wolf Of Wall Street while trying to sound like Simon Cowell.

It’s on the hiring organisation? Well, it is, but let’s take responsibility for the problem as a sector for a change. Infosec likes to shift responsibility and not take ownership of the problem. We blame CEOs, users, vendors, recruiters, dogs, cats, “Russia“, “China” – anyone but ourselves. Could it be we failed as a sector to raise awareness, both internally and externally?

So What Are Common Understandings Of Security Roles?

After 25 years+ we still don’t have universally accepted role descriptions, but at least we can say that some patterns are emerging. Security roles involve looking at risk holistically, and sometimes advising on how to deal with risk:

  • Security Engineers assess risk and design and sometimes also implement controls. BTW some sectors, legal in particular, still struggle with this. Someone who installs security products is in an IT ops role. Someone who upgrades and maintains a firewall is an IT ops role. The fact that a firewall is a security control doesn’t make this a security engineering function.
  • Security Architects take risk and compliance goals into account when they formulate requirements for engineers.
  • Security Analysts are usually level 2 SOC analysts, who make risk assessments in response to an alert or vulnerability, and act accordingly.

This subject evokes as much emotion as CISSP. There are lots of opinions out there. We owe to ourselves to be objective. There are plenty of sources of information on these role definitions.

No Aspect Of Risk Assessment != Security. This is Devops.

If there is no aspect of risk involved with a role, you shouldn’t looking for a security professional. You are looking for DEVOPS peeps. Not security peeps.

If you want a resource to install and configure tools in cloud – that is DEVOPS. It is not Devsecops. It is not Security Engineering or Architecture. It is not Landscape Architecture or Accounting. It is not Professional Dog Walker. it is DEVOPS. And you should hire a DEVOPS person. If you want a resource to install and configure appsec tools for CI/CD – that is DEVOPS. If you want a resource to advise on or address findings from appsec tools, that is a Security Analyst in the first case, DEVSECOPS in the 2nd case. In the 2nd case you can hire a security bod with coding experience – they do exist.

Ok Then So What Does A DevSecOps Beast Look Like?

DevSecOps peeps have an attack mindset from their time served in appsec/pen testing, and are able to take on board the holistic view of risk across multiple technologies. They are also coders, and can easily adapt to and learn multiple different devops tools. This is not a role for newly graduated peeps.

Doing Security With Non-Security Professionals Is At Best Highly Expensive

Another important point: what usually happens because of the skills gap in infosec:

  • Cloud: devops fills the gap.
  • On-premise: Network Engineers fill the gap.

Why doesn’t this work? I’ve met lots of folk who wear the aforementioned badges. Lots of them understand what security controls are for. Lots of them understand what XSS is. But what none of them understand is risk. That only comes from having an attack mindset. The result will be overspend usually – every security control ever conceived by humans will be deployed, while also having an infrastructure that’s full of holes (e.g. default install IDS and WAF is generally fairly useless and comes with a high price tag).

Vulnerability assessment is heavily impacted by not engaging security peeps. Devops peeps can deploy code testing tools and interpret the output. But a lack of a holistic view or an attack mindset, will result in either no response to the vulnerability, or an excessive response. Basically, the Threat And Vulnerability Management capability is broken under these circumstances – a sadly very common scenario.

SIEM/Logging is heavily impacted – what will happen is either nothing (default logging – “we have Stackdriver, we’re ok”), or a SIEM tool will be provisioned which becomes a black hole for events and also budgets. All possible events are configured from every log source. Not so great. No custom use cases will be developed. The capability will cost zillions while also not alerting when something bad is going down.

Identity Management – is not deploying a ForgeRock (please know what you’re getting into with this – its a fork of Sun Microsystems/Oracle’s identity management show) or an Azure AD and that’s it, job done. If you just deploy this with no thought of the problem you’re trying to solve in identity management, you will be fired.

One of the classic risk problems that emerges when no security input is taken: “there is no personally identifiable information in development Virtual Private Clouds, so there is no need for security controls”. Well – intelligence vulnerability such as database schema – attackers love this. And don’t you want your code to be safe and available?

You see a pattern here. It’s all or nothing. Either of which ends up being very expensive or worse. But actually come to think of it, expensive is the goal in some cases. Hold that thought maybe.

A Final Word

So – if the word risk doesn’t appear anywhere in the job description, it is nothing to do with security. You are looking for devops peeps in this case. And – security is an important consideration for cloud migrations.

How To Break Into Information Security

I’ve been asked a few times recently, usually by operations folk, to give some advice about how to break into the security sector, so under much pain I decided to commit my thoughts on the subject to this web log post. I’ve commented on this subject before and more extensively in chapter 6 of Security De-engineering, but this version is more in line with the times (up to 2012 I was advising a wide pass-by trajectory of planet infosec) and it will be shorter – you have my word(s).

blog-image

First I’d just be wary about trying to get into security just because of financial reasons (David Froud has an excellent blog and one of his posts covered this point well). At the time of writing it is possible to get into the field just by having an IT background and a CISSP. But don’t do that unless you have what’s REALLY required (do not judge what is REALLY required for the field based on job descriptions – at the time of writing, there are still plenty of mistakes being made by organisations). Summarising this in a very brief way:

  • You feel like you have grown out of pure IT-based roles and sort of excelled in whatever IT field you were involved in. You’re the IT professional who doesn’t just clear their problem tickets and switch off. You are, for example, looking for ways to automate things, and self-teach around the subject.
  • Don’t think about getting into security straight from higher education. Whereas it is possible, don’t do it. Just…don’t. Operational Security (or opsec/devopssec) is an option but have some awareness of what this is (scroll down to the end for an explanation).
  • Flexibility: can jump freely from a Cisco switch to an Oracle Database on any Operating System. Taking an example: some IT folk are religious about Unix and experience a mental block when it comes to Windows – this doesn’t work for security. Others have some kind of aversion to Cloud, whereas a better mindset for the field is one that embraces the challenge. Security pros in the “engineer” box should be enthusiastic about the new opportunities for learning offered by extended use of YAML, choosing the ideal federated identity management solution, Puppet, Azure Powershell, and so on. [In theory] projects where on-premise applications are being migrated to Cloud are not [in theory] such a bad place to be in security [in theory].
  • You like coding. Maybe you did some Python or some other scripting. What i’ve noticed is that coding skills are more frequently being seen as requirements. In fact I heard that one organisation went as far as putting candidates through a programming test for a security role. Python, Ruby, Shell ([Li,U]nix) and Powershell are common requirements these days. But even if role descriptions don’t mention coding as a requirement – having these skills demonstrates the kind of flexibility and enthusiasm that go well with infosec. “Regex” comes up a lot but if you’ve done lots of Python/Ruby and/or Unix sed/awk you will be more than familiar with regular expressions.

There is a non-tech element to security (sometimes referred to as “GRC”) but this is something you can get into later. Being aware of international standards and checking to see what’s in a typical corporate security policy is a good idea, but don’t be under the impression that you need to be able to recite verses from these. Generally speaking “writing stuff” and communication is more of a requirement in security than other fields, but you don’t need to be polished at day zero. There are some who see the progression path as Security Analyst –> Security Consultant (Analyst who can communicate effectively).

Another common motivator is hacker conferences or Mr Robot. Infosec isn’t like that. Even the dark side – you see Elliott with a hoody writing code with electronic techno-beats in the background, but hackers don’t write code to compromise networks to any huge degree, if at all. All the code is written for them by others mostly. And as with the femtocell and Raspberry Pi incidents, they usually have to assume a physical presence on the inside, or they are an internal employee themselves, or they dupe someone on the inside of the organisation under attack. Even if you’re in a testing role on the light side, the tests are vastly restricted and there’s a very canned approach to the whole thing with performance KPIs based on reports or something else that doesn’t link to actual intellectual value. Its far from glamorous. There’s an awful lot of misunderstanding out there. What is spoken about at hacker confz is interesting but its not usually stuff that is required to prove the existence of vulnerability in a commercial penetration test – most networks are not particularly well defended, and very little attention is given to results, more so because in most cases the only concern is getting ticks in boxes for an audit – and the auditors are often 12 years old and have never seen a command shell. Quality is rarely a concern.

Its a good practice to build up a list of the more influential bloggers and build up a decent Twitter feed and check what’s happening daily, but also, here are the books that I found most useful in terms of starting out in the field:

  • TCP/IP Illustrated – there are 3 volumes. 1 and 2 are the most useful. Then…
  • Building Internet Firewalls – really a very good way to understand some of the bigger picture ideas behind network architecture design and data flows. I hear rolling of eyes from some sectors, but the same principles apply to Cloud and other “modern” ideas that are from the 90s. With Cloud you have less control over network aspects but network access control and trust relationships are still very much a concern.
  • Network Security Assessment – the earlier versions are also still pertinent unless you will never see a Secure Shell or SMB port (hint: you will).
  • Security Engineering – there’s a very good chapter on Cryptography and Key Management.
  • The Art of Software Security Assessment – whether or not you will be doing appsec for a living you should look at OWASP‘s site and check out Webgoat. They are reportedly looking to bolster their API security coverage, which is nice (a lot of APIs are full of the same holes that were plugged in public apps by the same orgs some years ago). But if you are planning on network penetration testing or application security as a day job, then read this book, its priceless and still very applicable today.
  • The Phoenix Project – a good background illustrative for gaining a better understanding of the landscape in devops.

Also – take a look at perhaps a Windows security standard from the range of CIS benchmarks.

Finally – as i alluded earlier – opsec is not security. Why do i say this? Because i did come across many who believe they made it as a security pro once they joined a SOC/NOC team and then switched off. Security is a holistic function that covers the entire organisation – not just its IT estate, but its people, management, availability and resilience concerns, and processes. As an example – you could be part of a SOC team analysing the alerts generated by a SIEM (BTW some of the best SIEM material online is that written by Dr Anton Chuvakin). This is a very product centric role. So what knowledge is required to architect a SIEM and design its correlation rules? This is security. The same applies to IDS. Responding to alerts and working with the product is opsec. Security is designing the rulebase on an internal node that feeds off a strategically placed network tap. You need to know how hackers work among other areas (see above). Security is a holistic function. A further example: opsec takes the alerts generated by vulnerability management enterprise suites and maybe does some base false positives testing. But how does the organisation respond effectively to a discovered vulnerability? This is security.

One Infosec Accreditation Program To Bind Them All

May 2013 saw a furious debate ensue after a post by Brian Honan (Is it time to professionalize information security?) that suggested that things need to be improved, which was followed by some comments to the effect that accreditation should be removed completely.

Well, a suggestion doesn’t really do it for me. A strong demand doesn’t really do it either, in fact we’re still some way short. No – to advocate the strength of current accreditation schemes is ludicrous. But to then say that we don’t need accreditations at all is completely barking mad.

Brian correctly pointed out “At the moment, there is not much that can be done to prevent anyone from claiming to be an information security expert.” Never a truer phrase was spoken.

Other industry sectors have professional accreditation and it works. The stakes are higher in areas such as Civil Engineering and Medicine? Well – if practitioners in those fields screw up, it cost lives. True, but how is this different from Infosec? Are the stakes really lower when we’re talking about our economic security? We have adversaries and that makes infosec different or more complex?

Infosec is complex – you can bet ISC2’s annual revenue on that. But doesn’t that make security even more deserving of some sort of accreditation scheme that works and generates trust?

I used the word “trust”, and I used it because that what’s we’re ultimately trying to achieve. Our customers are C-levels, other internal departments, end users, home users, and so on. At the moment they don’t trust infosec professionals and who can blame them? If we liken infosec to medicine, much of the time, forget about the treatment, we’re misdiagnosing. Infosec is still in the dark ages of drilling holes in heads in order to cure migraine.

That lack of trust is why, in so many organizations, security has been as marginalized as possible without actually being vaporized completely. Its also why security has been reduced down to the level of ticks in boxes, and “just pass the audit”.

Even though an organization has the best security pros in the world working for them, they can still have their crown jewels sucked out through their up-link in an unauthorized kinda way. Some could take this stance and advocate against accreditation because ultimately, the best real-world defenses can fail. However, nobody is pretending that the perfect, “say goodbye to warez – train your staff with us” security accreditation scheme can exist. But at the same time we do want to be able to configure detection and cover some critical areas of protection. To say that we don’t need training and/or accreditation in security is to say the world doesn’t need accreditation ever again. No more degrees and PhDs, no more CISSPs, and so on.

We certainly do need some level of proof of at least base level competence. There are some practices and positions taken by security professionals that are really quite deceptive and result in resources being committed in areas where there is 100% wastage. These poor results will emerge eventually. Maybe not tomorrow, but eventually the mess that was swept under the carpet will be discovered. We do need to eliminate these practices.

So what are we trying to achieve with accreditation? The link with IT needs to be re-emphasized. The full details of a proposal are covered in chapter 11 of Security De-engineering, but basically what we need first is to ensure the connection at the Analyst level with IT, mainly because of the information element of information technology and information security (did you notice the common word in IT and IT security? Its almost as though there might be a connection between them). 80% of information is now held in electronic form. So businesses need expertise to assist them with protection of that information.

Security is about both business and IT of course. Everybody knows this even if they can’t admit it. There is an ISMS element that is document and process based, which is critical in terms of future proofing the business and making security practices more resource-efficient. A baseline security standard is a critical document and cannot be left to gather dust on a shelf – it does need to be a “living” document. But the “M” in ISMS stands for Management, and as such its an area for…manage-ers. What is quite common is to find a security department of 6 or more Analysts who specialize in ISMS and audits. That does not work.

There has to be a connection with IT and probably the best way to ensure that is to advocate that a person cannot metamorphosize into a Security Analyst until they have 5 years served in IT operations/administration, network engineer, or as a DBA, or developer. Vendor certs such as those from IBM, Microsoft, Cisco – although heavily criticized they can serve to indicate some IT experience but the time-served element with a signed off testimonial from a referee is critical.

There can be an entrance exam for life as an Analyst. This exam should cover a number of different bases. Dave Shackleford’s assertion that creative thinkers are needed is hard to argue with. Indeed, what i think is needed is a demonstration of such creativity and some evidence of coding experience goes a long way towards this.

Flexibility is also critical. Typically IT ops folk cover one major core technology such as Unix or Windows or Cisco. Infosec needs people who can demonstrate flexibility and answer security questions in relation to two or more core technologies. As an Analyst, they can have a specialization with two major platforms plus an area such as application security, but a broad cross-technology base is critical. Between the members of a team, each one can have a specialization, but the members of the team have knowledge that compliments each other, and collectively the full spectrum of business security concerns can be covered.

There can be specializations but also proportional rewards for Analysts who can demonstrate competence in increasing numbers of areas of specialization. There is such a thing as a broad-base experienced Security Analyst and such a person is the best candidate for niche areas such as forensics, as opposed to a candidate who got a forensics cert, learned how to use Encase, plastered forensics on their CV, and got the job with no other Analyst experience (yes – it does happen).

So what emerges is a pattern for an approximate model of a “graduation”-based career path. And then from 5 years time-served as an Analyst, there can be another exam for graduation into the position of Security Manager or Architect. This exam could be something similar to the BCS’s CISMP or ISACA’s CISA (no – I do not have any affiliations with those organizations and I wasn’t paid to write this).

Nobody ever pretended that an accreditation program can solve all our problems, but we do need base assurances in order for our customers to trust us.

The Search For Infosec Minds

Since the early 2000s, and in some of my other posts, I have commented in different forms on the state of play, with a large degree of cynicism, which was greeted with cold reservation, smirks, grunts, and various other types of un-voiced displeasure, up to around 2009 or so. But since at least 2010, how things have changed.

If we fast forward from 2000 to 2005 or so, most business’s security function was reduced down to base parrot-fashion checklists, analysis and thinking were four letter words, and some businesses went as far as outsourcing security functions.

Many businesses who turned their backs on hackers just after the turn of the millennium have since found a need to review their strategies on security hiring. However 10 years is a long time. The personnel who were originally tasked with forming a security function in the late 90s, have since risen like phoenixes from the primordial chasm, and assisted by thermals, they have swooped up to graze on higher plains. Fast forward again to 2012, and the distance between security and IT is in the order of light years in most cases. The idea that security is purely a compliance game hasn’t changed, but unlike the previous decade, it is in many cases seen as no longer sufficient to crawl sloth-like over the compliance finishing line every year.

Businesses were getting hacked all through the 2000s but they weren’t aware of it. Things have changed now. For starters the attacks do seem to be more frequent and now there is SIEM, and audit requirements to aggregate logs. In the past, even default log settings were annulled with the result that there wasn’t even local logging, let alone network aggregation! Mind you, even after having been duped into buying every well-marketed detection product, businesses are still being hacked without knowing it. Quite often the incident comes to light after a botnet command and control system has been owned by the good guys.

Generally there is more nefarious activity now, as a result of many factors, and information security programs are under more “real” focus now (compliance-only is not real focus, in fact it’s not real anything, apart from a real pain the backside).

The problem is that with such a vast distance between IT and security for so long, there is utter confusion about how to get tech’d up. Some businesses are doing it by moving folk out of operations into security. This doesn’t work, and in my next post I will explain why it doesn’t work.

As an example of the sort of confusion that reigns, there was one case I came across earlier in 2012 where a company in the movies business was hacked and they were having their trailers, and in some cases actual movies, put up on various torrent sites for download. Their response was to re-trench their outsourced security function and attempt to hire in-house analysts (one or two!). But what did they go looking for? Because they had suffered from malware problems, they went looking for, and I quote, “Malware Reverse Engineers”. Malware Reverse Engineers? What did they mean by this? After some investigation, it turns out they are really were looking for malware reverse engineers, there was no misnomer – malware reverse engineers as in those who help to develop new patterns for anti-virus engines!! They had acquired a spanking new SIEM, but there was no focus on incident response capability, or prevention/protection at all.

As it turns out “reverse engineer[ing]” is now a buzzword. Whereas in the mid-2000s, buzzwords were “governance” and “identity management” (on the back of…”identity theft” – neat marketing scam), and so on. Now there are more tech-sounding buzzwords which have different connotations depending on who you ask. And these tech sounding buzzwords find their way into skills requirements sent out by HR, and therefore also on CVs as a response. And the tech-sounding buzzwords are born from…yes, you got it…Black Hat conferences, and the multitude of other conferences, B-sides, C-sides, F-sides and so on, that are now as numerous as the stars in the sky.

The segue into Black Hat was quite deliberate. A fairly predictable development is the on-going appearance of Infosec managers at Black Hats, who previously wouldn’t touch these events with a barge pole. They are popping up at these events looking to recruit speakers primarily, because presumably the speakers are among the sharper of the crayons in the box, even if nobody has any clue what they’re talking about.

Before I go on, I need to qualify that I am not going to cover ethics here, mainly because it’s not worth covering. I find the whole ethics brush to be somewhat judgmental and divisive. I prefer to let the law do the judgment.

Any attempt to recruit tech enthusiasts, or “hackers”, can’t be dismissed completely because it’s better than anything that could have been witnessed in 2005. But do businesses necessarily need to go looking for hackers? I think the answer is no. Hackers have a tendency to take security analysis under their patronage, but it has never been their show, and their show alone. Far from it.

In 2012 we can make a clear distinction between protection skills and breaking-in skills. This is because as of 2012, 99.99…[recurring to infinity]% of business networks are poorly defended. Therefore, what are “breaking-in skills”? So a “hacker” breaks into networks, compromises stuff, and posts it on pastebin.com. The hackers finds pride and confidence in such achievements. Next, she’s up on the stage at the next conference bleating about “reverse engineering”, “fuzzing”, or “anti forensics tool kits”…nobody is sure which language is used, but she’s been offered 10 jobs after only 5 minutes into her speech.

However, what is actually required to break into networks? Of the 20000+ paths which were wide open into the network, the hacker chose one of the many paths of least resistance. In most cases, there is no great genius involved here. The term “script kiddy” used to refer to those who port scan, then hunt for public declared exploits for services they find. There is IT literacy required for sure (often the exploits won’t run out-of-the-box, they need to be compiled for different OSs or de-bugged), but no creativity or cunning or …whatever other mythical qualities are associated with hacking in 2012.

The thought process behind hiring a hacker is typically one of “she knows how to break into my network, therefore she can defend against others trying to break in”, but its quite possible that nothing could be further from the truth. In 2012, being a hacker, or possessing “breaking-in skills”, doesn’t actually mean a great deal. Protection is a whole different game. Businesses should be more interested about protection as of 2012, and for at least the next decade.

But what does it take to protect? Protection is a more disciplined, comprehensive IT subject. Collectively, the in-house security teams needs to know the all the nooks and crannies, all the routers, databases, applications, clouds, and operating systems and how they all interact and how they’re all connected. They also collectively need to know the business importance of information assets and applications.

The key pillars of focus for new-hire Security Analysts should be Operating Systems and Applications. When we talk about operating systems and security, the image that comes to mind is of auditors going thru a checklist in some tedious box-ticking exercise. But OS security is more than that, and it’s the front line in the protection battle. The checklists are important (I mean checklists as in standards and policies) but there are two sides to each item on the checklist: one is in the details of how to practically exploit the vulnerability and the potential tech impact, the other are the operational/business impacts involved with the associated safeguard. In other words, OS security is far more than a check-list, box-ticking activity.

In 12 years I never met a “hacker” who could name more than 3 or 4 local privilege elevation vectors for any popular Operating System. They will know the details of the vulnerability they used to root a server last month, but perhaps not the other 100 or so that are covered off by the corporate security standard for that Operating System. So the protection skills don’t come by default just because someone has taken to the stand at a conference.

Skills such as “reverse engineering” and “fuzzing” – these are hard to attain and can be used to compromise systems that are well defended. But the reality is that very few systems are so well defended that such niche skills are ever needed. In 70+ tests for which i have either taken part or been witness, even if the tests were quite unrestricted, “fuzzing” wouldn’t be required to compromise targets – not even close.

A theoretical security team for a 10000+ node business, could be made of a half dozen or so Analysts, plus a Security Manager. Analysts can come from a background of 5 years in admin/ops or devs. To “break into” security, they already have their experience in a core technology (Unix, Windows, Oracle, Cisco etc), then they can demonstrate competence in one or more other core technologies (to demonstrate flexibility), programming/scripting, and security testing with those platforms.

Once qualified as a Security Analyst, the Analyst has a specialization in at least two core technologies. At least 2 analysts can cover application security, then there are other areas such as incident handling and forensics. As for Security Managers, once in possession of 5 years “time served” as an Analyst, they qualify for a manager’s exam, which when passed qualifies them for a role as a Security Manager. The Security Manager is the interface, or agent, between the technical artist Analysts and the business.

Overall then, it is far from the case that Hackers are not well-suited for vocational in-house security roles (moreover I always like to see “spare time” programming experience on a resume because it demonstrates enthusiasm and creativity). But it is also not the case that Analyst positions are under the sole patronage of Black Hat speakers. Hackers still need to demonstrate their capabilities in protection, and doing “grown-up” or “boring” things before being hired. There is no great compelling need for businesses to hire a hacker, although as of today, it could be that a hooligan who throws security stones through security windows is as close as they can get to effective network protection.

How To Break Into Security – Planet Earth Edition

The venerable Brian Krebs has recently been running some stories from various demigods of the infosec world, aimed at those wishing to enter the information security field – aspiring graduate ninjas, and others seeking the mythical pot of gold at the end of the rainbow.

First up there was there was Tomas Ptacek’s edition, then we had some pearls of wisdom from Bruce Schneier, Jeremiah Grossman, Richard Bejtlich, and then Charlie Miller.

Thomas Ptacek claimed about the security field: “It’s one of the few technology jobs where the most fun roles are well compensated”, and “if you watched “Sneakers” and ideated a life spent breaking or defending software, great news: infosec can be more fun in real life, and it’s fairly lucrative.” Well…I am not refuting any of this, but it is certainly quite unusual for jobs in information security to be fun.

Thomas talks of the benefits of having extensive programming experience – and this is something I advocate myself quite strenuously (more on that later). Thomas’s viewpoint was centered around appsec, which is fine. I think for myself, in terms of defending networks, we need to look at two main areas: appsec and operating systems / databases. There is some good advice in Thomas’s article about breaking into application security, although I wouldn’t say that this area is everything. There’s a little too much religious fervor about appsec in the article for my liking, just as one often sees a lack of balance in other areas, such as CISSP-worship, and malware “reverse engineering” – basically – “my area is the alpha and omega – all that was, is, and ever will be”.

There are other areas that matter in security other than application security. I wouldn’t say its all about appsec, but I would say though that the two main areas are appsec and operating system and database configuration. “But operating systems and databases are also applications” I hear you say. Yes, but when we’re talking appsec in infosec, we’re usually talking about web applications. There are few times when we suffer web attacks where that one single exploit leads to something really bad happening, apart from perhaps a SQLi that in itself reveals sensitive database hosted information.

With regard web applications I think Thomas is spot-on with his comments about learning about web application security assessment, and how to get clued up in this area. Also the comments about Nessus and getting into penetration testing – sad but true.

With regard Bruce Schneier’s “breaking into” edition – nothing he says is factually incorrect (most of what we talk about is subjective, neither black or white, but grey) but the comments are not at all close to the coal-face realities of most business’s in-house or service providers’ practices. Wannabe security pros reading this will be sure to get grandiose visions of their future lives as a security pro – but in 90%-plus of cases the vocational activities of security professionals do not match the picture painted by Bruce Schneier. I’ll explain more on this later.

Richard Bejtlich’s response was centered around getting into penetration testing with Metasploit and Jeremiah Grossman’s was the most all-encompassing and in my opinion, the response that had the most value for security pro wannabes – although Charlie Millar’s wasn’t far behind. In particular Mr Grossman plays down the effectiveness of accreditation programs, in favor of practical experience – wise words indeed. Charlie Millar had similar opinions.

In Charlie Millar’s response there was a lot of talk of really specialized nieche areas like reverse engineering and so on, but he does temper this with “I really do a lot of reverse engineering and binary analysis, which is unusual.” and “for those starting out, it probably makes more sense to learn some languages more useful for web applications, like PHP or Java or something.  The majority of jobs I come across in application security are web applications, so unless you’re a dinosaur like me, you probably want to become a web app expert.  Web application security is a lot easier to get started in as well.”

Charlie Millar’s response sort of segues me into the wider scope here, and that is of the realities “out there”. The articles are based on responses by folk who’ve rightly become esteemed professionals in their field and there is some really valuable insight there. The thing is – there is a lot of talk of the security field being a place for artists and magicians, and of being technically demanding, but there are very few places where technical acrobatics skills in security are seen as having any value to businesses, or even security line managers for that matter – and therefore such intellectual capital just does not appear on the balance books of these businesses.

We’ve been through a vicious 70 to 80% of a sine wave of pain in security since the late 90s. The security world painted by the fellowship of five assembled by Brian Krebs (speaking of whom, it would be nice to hear his version of the “breaking into” story) seems much more like the world of the late 90s than today. Security was heavily de-engineered through the 2000s. Things started to change around 2010, but we’re still very much in non-tech territory, and many of the security line managers who will interview prospective Security Analysts will not have an IT background, and their security practice will be hands-off, non-tech, check-list based. Anything “tech” to do with information risk management will be handled by an ops team, but there won’t be any “reverse engineering” or “fuzzing” over there…far from it. More like – firewall configuration, running bad vulnerability management suites, monitoring IDS/SIEM logs.

Picking up on some of Bruce Schneier’s comments: “You can be an expert in viruses, or policies, or cryptography.” Policies? OK, this part is true – if you want to be an expert in policies, whatever that is, you can certainly find this in 90%+ of businesses – but is this something that is an economically viable and sustainable position, or even for that matter anything that any homo sapien would ever really want to do? Probably accountancy would be a better bet.

“Viruses?” Hmm. There are increasing numbers of openings for “malware reverse engineers”, where really what they’re looking for is incident response – they want to know what happened after they discovered that some of their laptops were connecting out to various addresses in places they hadn’t heard of, prior to the click of doom. If you get interviewed for one of these positions, be prepared to answer questions about SIEM technologies and incident response. These openings are not usually associated with reverse engineering to the level of detail of those pattern-makers in the anti-virus software market- and if they are, they needn’t be, and the line manager will get to realize this after a while.

And “cryptography”? We have Bruce’s comments and then we have a title heading from a book by Shostack and Stewart: “Amateurs Study Cryptography; Professionals Study Economics”. So who do you believe here? To be fair, the book was published at the height of the de-engineering phase of security and it sort of fitted with the agenda of the times. I would go with Bruce Schneier again here, but with some qualification (the final paragraph also talks about math): most security departments won’t ever go anywhere near anything mathematical or even crypto-related, and when they do, it will be with a checklist approach that goes something like “is a strong key used?”, “yes”, “ok, good, tick in a box”, or “is DES used?”, “yes”, “ok, that’s bad, I think, anyway use triple DES please” – with no further assistance for the dev team.

With regard Cryptography, right or wrong, it’s really only on tiny islands where the math is seen as relevant – places where they code the apps and people are assigned to review the security of the app. And these territories are keenly disputed. Most of the concerns in the rest of the business world will never get more techy than discussions about key management – which is the more common challenge with crypto anyway (mostly we’re using public crypto algorithms in security, so the challenge is in protection of the key).

There were comments from all respondents about testing applications and breaking into networks. Again, the places where skills like reverse engineering are actually relevant are so small. Bruce Schneier painted a grand picture of thinking like a hacker, not just a mere engineer, in order to be able to create systems that are difficult to compromise by the most advanced hackers. But most humans who design systems are not even thinking about security, or they’re on such a tight deadline (with related KPIs and bonuses) that they side-step security. So as a pen test guru wannabe, you may possess extremely high levels of fuzzing, exploit coding, and reversing skills, but you will never get to use them, in fact you will intimidate most interviewers, and you’ll be over-qualified. There will be easier ways to break into systems in most cases. In fact in general, as I commented in an earlier post, security is insufficiently mature in most organizations to warrant any manual penetration testing whatsoever.

So really, what I have had to say here may sound harsh or “negative”, but I would hate for anyone to get into a field that they thought was challenging, only to discover that it’s anything but. I believe things are changing, but it’s at rather a slow pace, and the field of security has been broken for so long, that there are very few around who know how to fix it. Security is getting more challenging, that’s for sure, but for the security pro who goes looking for a job in this field because of the tech challenge aspect: just be very careful about what you’re getting into. Many jobs sound great from the job descriptions posted to recruitment agents, but this is only a show. The reality inside the team is that you may be sent to Siberia if you so much as use a tech-sounding word like “computer” or “IP address” – while this sounds unreal I can assure the reader that such a scenario is most certainly real, although it is of course more often the case that the job would just not be offered to a “techie”.

Its impossible to cover the jobs aspect of information security in just this article. I had a more comprehensive stab at it in Chapter Six of Security De-engineering. I would say to the prospective security pro though, that the advice given by the five mentioned in this article is not bad advice at all – it’s just that you may push yourself to higher levels, and not see significant benefit from it in your careers any time soon. There will always be some benefit, just not as much as you might expect. Certainly, you will have more confidence, but also probably over-qualify yourself for your current position.

As a security pro with a tech inclination, getting into security might not be as hard as you thought. Thomas Ptacek mentioned “A good way to move into penetration testing: grab some industry standard tools and use an Amazon EC2 account to set up a “shooting range” to attack. Some of the best-known tools are available for free: the Nessus scanner, for instance, while not an application security tool, is free and can land you a network penetration testing role that you can use as a springboard to breaking applications.” Believe me, this is not a difficult target, but because of the way the security industry is, you could very well land a penetration testing job with the preparation as described by Mr Ptacek.

All I had to say here is aimed at managing expectations. You may well find that you have to market yourself down a bit in order just to get a foothold in the industry. Once there, by pushing yourself to learn more and get more advanced skills, it could be that you would eventually osmosize towards the ideal job of your dreams. However, these positions are so rare in reality. Many of the folk I worked with in the earlier days of my career had these ninja skills that have been discussed in the five articles mentioned here. Once we got to the early 2000s they realized that security was no longer a place for them. Has the demand for these advanced skills returned? It has, to some extent, but still the demand is miniscule compared to the usual skills required the vast majority of businesses.