Managing the Development of Secure Systems and System Evaluation: A Comprehensive Analysis

Project Management, Risks and Organizational Psychology

Introduction to Security Management

The development of secure systems goes far beyond exclusively technical tasks. If you are an IT manager or a consultant entrusted with creating a protected system, you will need a systemic approach to choosing the goals of protection and the mechanisms for implementing them. A key role here is played not only by technologies, but also by systems engineering, risk analysis and, most importantly, team management. Understanding the capabilities and motivation of the personnel who will operate the control systems, as well as introducing the right approaches to code development — this is what separates successful projects from failed ones.

Security Project Management and Risk Assessment

The most difficult task of a security project manager is to understand the trade-off between risk and reward. Security specialists naturally focus on risks and often overlook the potential benefit.
Consider a hypothetical situation: a company has a turnover of $10 million, a profit of $1 million, and losses from theft of $150,000. A security consultant may propose measures that "will reduce the losses and increase the profit by 15%". However, it may be in the shareholders' interest to double the turnover to $20 million, even if this triples the losses from theft to $450,000. With the same margin retained, the profit will grow to $1.85 million (an increase of 85%). Therefore business owners should not fall into the trap of thinking that every vulnerability must be responded to exclusively by "treating" it. Often security is already redundant, and resources need to be redirected into another channel. But since security teams (internal or external) have an incentive to exaggerate threats, and their expertise is difficult to challenge, this balance requires tough management.

A Case: A Tale of Three Supermarkets

To illustrate how security technologies interact with business processes and with the psychology of customers, let us consider three real approaches of retail chains to the problem of shoplifting and of queues at the checkout.

1. Automation through RFID (South Africa). One supermarket decided to automate the process completely. All the goods were to have RFID tags, so that a whole trolley could be read automatically. This would have killed two birds with one stone: it would have reduced the staff of cashiers and complicated theft. However, the pilot project failed. The tags were poorly read from conductive goods (for example, tin cans), and the use of special trolleys turned out to be inconvenient and aesthetically unattractive for customers.
2. Face recognition and civil suits (Europe). Another chain initially tried to use RFID to stop professional thieves. When this did not work, they turned to face recognition systems to alert the guards. This technology also did not provide the required level of accuracy. In the end the chain moved to the tactic of "civil recovery": after a shoplifting had been recorded by the police and a fine imposed, the supermarket sued the thief in a civil court for "lost time and moral damage", and then arrested the debtor's property. However, the management became so carried away with revenge on petty thieves that it lost focus on marketing and on increasing sales. As a result the chain lost market share, and its shares fell.
3. Self-service and trust (Waitrose, England). The Waitrose chain introduced a self-service system. The customer scans his loyalty card, receives a portable scanner, scans the goods as he collects them and puts them into his bag. At the exit the data are checked, and payment goes through by credit card. This system elegantly solves the problem of theft: only holders of loyalty cards have access to self-service (which automatically excludes known shoplifters and stimulates the issuing of cards). The customer feels himself a "trusted person". If the guards notice suspicious behavior on the screen, the system can deliberately "break down" at the self-service checkout, which gives the staff a pretext politely to ask the customer to show the contents of his bag without an open conflict.

For comparison: the most "defensive" stores force customers to scan goods on special scales-terminals, which constantly interrupt the process, requiring each item to be weighed. This creates a high level of distrust and irritation. While the "defensive" chains face an outflow of customers, Waitrose continues to strengthen its position.

Organizational Questions and the Psychology of Security

Security management is not only the mathematics of risk, but also deep organizational psychology. Advisers and auditors often "settle in" with their clients, forming a consensus that gradually detaches itself from reality. This is precisely why the CEO or the senior management must periodically "shake up the pot" and attract independent external experts capable of challenging group thinking.

The Cycle of Carelessness and the Risk Thermostat

In the telephone fraud industry in the USA there exists a seven-year cycle: one of the regional companies (the Baby Bells) suffers losses, hires experts, builds up protection, after which the cycle repeats itself at the next player. This is a classic example of organizational carelessness.
John Adams, in his study of the laws on mandatory seat belts, discovered that they do not save lives, but merely redistribute the victims: the number of drivers killed falls, but the number of pedestrians and cyclists killed rises. Why? Seat belts make drivers feel protected, and they begin to drive faster in order to return their subjective perception of risk to its former level. Adams called this the "risk thermostat".
Feedback mechanisms limit the effectiveness of risk reduction systems. As soon as the level of incidents falls to a minimum, vigilance is blunted: guards fall asleep, real threats drown in false alarms, and security budgets are cut to a dangerous limit.

Interaction with Reliability

Poor internal control often arises in systems where transactions constantly fail and require manual intervention. A high tolerance for chaos undermines control, creating a high level of false alarms for all protection mechanisms simultaneously. Moreover, it tempts the staff: if employees see that errors are not tracked, they conclude that thefts too will go unpunished.
There is a direct correlation between the quality of code and security. Investment in software quality reduces the number of vulnerabilities, even if security was not a direct goal of the quality program. The most effective method of ensuring quality from the point of view of security is the code walk-through. The awareness that one's code will be read and criticized by colleagues has a sobering effect on programmers. Reliability is often the best argument for a board of directors: errors cost money, and the search for vulnerabilities can be presented as the search for bugs, without causing embarrassment to the developers.

Solving the Wrong Problem

When faced with a complex problem, organizations often begin furiously to attack an adjacent but easier task (displacement activity).
An excellent example is the smart card industry. Faced with the complexity of protecting chips from probing and from power analysis, manufacturers decided to protect the secrecy of the chip masks. As a result, technical manuals appeared that are available only under an NDA, visitors to the factories must sign non-disclosure agreements, and waste is destroyed in shredders. This physical paranoia impresses naive customers, but almost all the real attacks on smart cards use technical vulnerabilities for which knowledge of the chip topology is not required.

Incompetent Managers and Moral Hazard

In traditional companies promotion to the highest posts often depends on seniority and connections. To be a security manager means constantly to say "no", which makes this post undesirable for careerists. As a result, the average length of time in post of a cybersecurity manager in US government agencies is only 7 months. Frequent reorganizations destroy institutional memory, and the security function is reduced to "ticking boxes" to comply with formal requirements.
Another serious problem is moral hazard, which arises when systems are designed so as to shift the risk onto third parties. This deprives the company of incentives to invest in real protection. For example, if banks declare that their ATMs are infallible and that responsibility for "phantom withdrawals" lies with the customer, the bank's staff relax and stop investigating incidents. In such conditions employees become lazy, and sometimes collusive with fraudsters, knowing that the bank will write the losses off onto the victim anyway.
Moral hazard also arises when the decision-makers do not bear responsibility for the consequences. A high turnover of IT staff, reliance on contractors, or ambitious managers who do not want to argue with developers for the sake of a short-term release — all this leads to the fact that three years later, when the system is hacked, the guilty parties can no longer be found.

Methodologies, Requirements Engineering, Risk Management and Team Management

Development Methodology

Software projects usually take more time, cost more and contain more errors than expected. By the 1960s this phenomenon became known as the "software crisis". In response to it, Brian Randell in 1968 proposed the term "software engineering", expressing the hope that the problem could be solved in the same way as ships and aircraft are built — using a proven scientific base and a set of design rules.
In practice, the tools of software engineering help developers to climb ever higher up the mountain of complexity before they fall off it, but the frequency of catastrophic failures of large projects remains unchanged. Two main approaches help to manage complexity: top-down and iterative.

Top-Down Design (the Waterfall Model)
The classic model is the "waterfall", developed by Win Royce in the 1960s for the US Air Force. The idea consists in passing sequentially through the stages: from a brief statement of requirements to a specification, then to implementation and module testing, integration and system testing, and finally to deployment.
The strengths of the waterfall are that it forces the goals of the system, the architecture and the interfaces to be clarified early, it eases the task of the project manager by providing clear milestones, and it is compatible with a wide range of tools. However, its main shortcoming is the absence of systemic feedback from testing to requirements. If the requirements are not known in detail in advance, this model is inapplicable.

Iterative Design
In cases where the requirements change or the technology is developing, an iterative approach is applied.

1. The spiral model of Barry Boehm assumes the creation of a prototype and its testing at each stage, allowing managers to assess the risks and to decide whether to continue the work or to wind the project up.
2. Evolutionary development, popularized by Harlan Mills, assumes the creation of a minimal working system, its testing by real users and the gradual addition of functionality. This is exactly how the packaged software industry works: products become so complex that it is impossible to rewrite them from scratch.

The key technology of evolutionary development is regression testing. Regularly (usually once a day) all the teams check in their code, which is compiled and automatically tested on a large set of input data. This makes it possible to be sure that old functions have not broken and that old bugs have not returned. For the security engineer, regression testing is less useful, since vulnerabilities (for example, stack overflow) more often appear in new functions than return in old ones, and many security flaws arise at the junction of levels of abstraction, where automatic tests are powerless. Nevertheless, regression testing remains critically important for revealing functional failures caused by changes.

Lessons from Safety-Critical Systems

The methodologies of developing safety-critical systems (for example, flight control or automatic braking systems) offer valuable lessons for security engineers. They usually follow the "waterfall" paradigm: hazard identification and risk assessment, choice of strategy (avoidance, limitation, redundancy), tracing of hazards down to hardware and software components, analysis of operator procedures and, finally, the creation of a test plan and a Safety Case.

Hazard Elimination
Ideal design avoids hazards completely. The classic example is engine thrust-reverser circuits. In the original design a switch with two poles was used: if only one pole moved, the battery was short-circuited, which could cause a fire. The solution was to swap the battery and the motor. In the modified circuit a failure of the switch will simply short-circuit the motor, not the battery. In security this is analogous to minimizing the trusted computing base (TCB).

Fault Tree / Threat Tree Analysis
To reveal potential failures, fault tree analysis (top down) or failure mode and effects analysis — FMEA (bottom up, pioneered by NASA) is used. In security engineering this is transformed into threat tree analysis. For example, for an ATM the root of the tree is "Successful forgery of a card". The branches include: reading the data over the shoulder (shoulder surfing), cryptanalysis of DES, an attack through a false terminal, abuse of the security module, a Trojan, theft of keys, an insider or a maintenance contractor, an error in the protocol, and so on. Going round the leaves of the tree gives a guide to attacking the system, which ought to exist for the evaluators: if they find significant additional attacks, the product will not pass certification.

The Human Factor
The bible of human factors engineering is James Reason's book "Human Error". The frequency of errors depends on familiarity with and the complexity of the task, on pressure and on the presence of prompts. If the task is simple and performed often, the error rate may be 1 in 100,000. But if the task is performed for the first time in a confusing environment under pressure, the probability of success falls sharply. The security engineer must understand cognitive biases and avoid them. As in the case of pilots, who are trained in simulators combining equipment failures and bad weather, security must be built into the system from the very beginning, and not added after the fact.

Fault Masking
In systems with redundancy the problem of masking arises: if three processors vote by majority and one of them fails, the system will continue to work, but its margin of safety will have been undermined. In security an example is the case when a bank mistakenly issued the same PIN to all its customers. The error was masked by the procedures for handling PIN codes, which guaranteed that even the security service employees received only their own personal PIN envelope. The problem was discovered only a long time later.

Programming the Satan's Computer
The main difference between security and reliability lies in the failure model. Instead of random failures we are dealing with a hostile adversary who can make the components of the system fail at the most unsuitable moment and in the most destructive way. The name "programming the Satan's computer" is a counterweight to "Murphy's law". The Satan's computer is difficult to test, which is one of the main reasons why security engineering is so difficult.

Security Requirements Engineering and Managing Their Evolution

The process of creating a security policy and of obtaining the agreement of the owner of the system is called security requirements engineering. This is the most critical and difficult task, lying at the intersection of technical problems, bureaucratic wars and attempts to avoid responsibility.
Most often security requirements need adjustment for four reasons: fixing a bug, improving the system (tuning of controls), a change of environment or organizational changes.

1. Bug Fixing
If you sell critically important software, a collision with a vulnerability is inevitable. The response plan must consist of four components:

• Monitoring: It is necessary to learn about vulnerabilities as quickly as possible. It is useful to have a reward system for customers or researchers for bugs found.
• Repair: In organizations with critical requirements someone from the team must be "on duty" with a pager. Emergency fixes must go through the full cycle of testing, and the documentation (and the threat model) must be updated.
• Distribution: The patch must reach customers quickly. There is a serious tension between the desire to patch the system quickly and the need to test the fix thoroughly. This led Microsoft to introduce "Patch Tuesday" (the second Tuesday of each month), which became a sensible compromise between security, reliability and manageability.
• Reassurance: It is necessary to prepare in advance templates of press releases for incidents of different degrees of severity, so that journalists do not receive a rebuff from call center operators while you are feverishly fixing the bug.

2. Tuning of Controls and Corporate Governance
The main process of developing internal controls (for example, in banks) is their tuning on the basis of experience. It is important for the security engineer to understand the standards of internal audit. The most influential is COSO (Committee of Sponsoring Organizations) — a risk management framework that sets the tone for companies listed on US stock exchanges. Another important standard is CobiT, which is more adapted to IT needs and covers personnel management, change control and project management.
Additional drivers are laws such as Sarbanes-Oxley (which requires security measures to be documented), Gramm-Leach-Bliley (for the US financial sector) and HIPAA (for healthcare). In Europe, pressure is exerted by personal data protection legislation and by breach disclosure laws.

3. Evolving Environments and the Tragedy of the Commons
Many systems broke after a change of environment, when adequate changes in the protection mechanisms were ignored or forgotten. For example, the technology of cards and PIN codes, which worked excellently with ATMs, became vulnerable to attacks through false terminals in retail outlets.
The problem is aggravated when no one fully owns the system. Responsibility for the standards of PIN verification at ATMs is blurred: IBM, which developed most of the standards, lost its leading role, Microsoft is not interested in the ATM market, VISA ceased certifying equipment around 1990, Mastercard never did this, and the EMV consortium appeared later. In such conditions each player (an equipment manufacturer or a bank) has a motivation to push the boundaries a little, hoping that if something goes wrong it will happen to someone else. In economics this is called the tragedy of the commons: if a hundred peasants are allowed to graze sheep on a common pasture, each will add one more sheep, obtaining almost the whole benefit, while the other 99 will suffer only a small loss from the deterioration of the quality of the grass. The result is a dustbowl.

4. Organizational Changes
Organizational questions are often the root cause of failures. In the early 1990s the fashionable trend was business process reengineering, which often meant using changes in computer systems to force people to change their ways of working.
A classic example of a catastrophe is the London Ambulance Service. The managers decided to automate the system in order to cut costs and to solve problems with labor relations. When the system began to be introduced, it emerged that it could not cope with established practices (for example, crews took the "wrong" vehicles, since the senior employees had the better ones). The managers ignored this and on 26 October 1992 forcibly transferred the dispatchers to the terminals. The result was a collapse. Positive feedback loops arose: the system lost track of the vehicles, exceptions accumulated and disappeared from the screens, response time grew, the crews became upset and pressed the wrong buttons on the terminals, which increased the congestion. In the end the service was paralyzed and, by estimate, about twenty people died because help was not given in time. By the end of the day the system was switched back to a semi-manual mode.

Managing the Requirements of a Project

If a system is being created from scratch (for example, a new electronic commerce application), the process requires an iterative approach, more a spiral model than a waterfall.
An influential publication in this area is the book by Frank Swiderski and Window Snyder on Threat Modelling, describing the Microsoft methodology. The idea consists in listing the assets being protected and the assets available to the attacker, with subsequent tracing through the system in order to determine the levels of trust, the paths of attack, the barriers and the techniques (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege). The threat model is used for architecture reviews, targeted code checks and penetration tests.

Parallelization of the Process
When there is no expert to hand, the specification can be assembled by "brainstorming", attempting to list everything that can go wrong. The practice of hiring one consulting firm to draw up a security target is inferior to using several experts in parallel. People with different backgrounds (cryptography, access control, internal audit) will see the problem from different sides.
An experiment carried out at Cambridge University confirmed this: students were asked about a security policy for a company planning to bid for a license for the national lottery. The students offered valuable and original contributions at the level of policies (for example, the threat from the company's senior managers), of technical details (attacks on the refund mechanisms, the vulnerability of the radio time signal) and of routine checklist items that designers often omit (for example, "tickets must be tied to a particular draw"). The lesson: requirements engineering, like software testing, lends itself to a useful degree of parallelization.

Risk Management

At the basis of any security requirements lie business decisions about priorities: how much to spend on protection against what. This is risk management, which must fit into the broader framework of managing non-IT risks.
The most common technique is the calculation of ALE (Annualized Loss Expectancy). This is the expected loss multiplied by the number of incidents expected on average per year. A typical ALE analysis for a bank's computer systems may include hundreds of entries:

• SWIFT fraud: Loss $50,000,000, Probability 0.005 -> ALE $250,000
• ATM fraud (major): Loss $250,000, Probability 0.2 -> ALE $100,000
• Theft of cash by a cashier: Loss $3,240, Probability 200 -> ALE $648,000

Accurate figures are available for frequent losses, whereas for rare but high-risk events the probability is largely a guess. In practice the process of drawing up such a table often turns into iterative guessing: the consultant lists the threats, attaches nominal probabilities, obtains an absurd result (for example, the bank's ALE exceeds its income), and then "fits" the probabilities in order to obtain a figure justifying the maximum security budget that, in his opinion, the board of directors will agree to approve. ALEs can be useful, but they should not be made into a religion.
Insurance can help in managing large but unlikely risks. However, for companies listed on a stock exchange, an important reason for buying insurance against computer crime is due diligence. Risks that appear to be operational are in fact often legal and regulatory.

Team Management

How to manage a team of developers in order to create a product that does useful work and does not introduce vulnerabilities that become critical.

The Balance Between Responsibility and Specialization
One of the most difficult tasks is to find a balance between each developer being responsible for the security of his own code and having a security guru on whom everyone relies.
An important lesson was given by Microsoft's experience in its struggle with IBM on the operating systems market. In the late 1980s they collaborated on OS/2, but the partnership broke up in 1990. IBM followed the waterfall model, dividing developers into analysts, programmers and testers. This created a moral hazard: programmers wrote a lot of poor-quality code and "threw it over the wall" to the testers. As a result the code base was often so broken that it would not compile, and regression tests were impossible to run.
Microsoft took the position that it had no programmers or testers, but only developers: every programmer is obliged to fix his own bugs ("if you wrote it, you fix it"). Great attention was paid to making the software build every night for testing. In the end the people who wrote buggy code spent most of their time hunting and fixing their own errors, so most of the code base was written by the neater programmers. This allowed Microsoft to capture the market for 32-bit OSes.

Capability Maturity Model (CMM)
To manage quality, the Capability Maturity Model (CMM) from the Software Engineering Institute at Carnegie Mellon University is used. Research showed that competence is a function of teams, and not only of individual developers. Newly formed teams tend to underestimate the volume of work and have a high variance of execution time. Teams that work smoothly not only predict deadlines better, but also reduce the variance.
CMM offers a certification process in which established teams can obtain the status of an asset that cannot be frivolously scattered during reorganizations. The model has five levels: initial, repeatable, defined, managed and optimizing.

Culture and the Exchange of Experience
It is not enough to have skills — one must make people work together. One way is the "bug lunch", where developers are encouraged to discuss the latest subtle errors that they have managed to find and fix. It is critically important to create a culture in which people openly share what broke and was fixed. A bad practice is when those who find bugs (even in their own code) quietly fix them, since bugs correlate and there are probably more of them. A good example is air traffic control dispatchers, where it is expected that a controller who has made a mistake will not only correct it, but immediately announce it over the open channel, so that other controllers with potentially conflicting traffic can react.

Another useful exercise for team building is the adoption of a coding standard (house style). A chronic problem of poorly managed teams is a chaotic mixture of styles, where everyone does what he wants. When a programmer checks out code in order to work on it, he may spend half an hour reformatting it to suit himself. But if the goal is to write secure code, there is another reason for a single style. When you find a bug, you want to know whether it was an error of design or of implementation. If you do not understand what the programmer was thinking, this is difficult to determine. Therefore the code should have comments explaining the intent. Arguments about the "right" number and style of comments may drag on, but what is important is that there be a consistent style that people accept and that is fit for purpose. Creating such a style is a far better team-building exercise than playing paintball together.

The Concept of Assurance, Economic Incentives and the Growth of Reliability

Introduction to Assurance

If project management and methodologies answer the question "how do we build it", then Assurance answers the question "how do we know that it works and will not break". The working definition of Assurance sounds like this: "our estimate of the probability that the system will fail in a particular way".
This estimate is based on a number of factors: the process of developing the system, the character of the development team, particular technical evaluations (for example, the use of formal methods or the deliberate injection of bugs to test the testers) and, most importantly, on experience, which ultimately depends on having a model of how reliability grows (or falls) over time as the system is tested, operated and maintained.

Perverse Economic Incentives

A good starting point for the analysis of assurance is to consider the motives of the various participants. In the first and second parts of the articles on security engineering, a framework was presented for thinking rationally about security, comprising incentives, policies, mechanisms and assurance.

Incentives are critically important. If people do not actually want to protect a system, making them do so is extremely difficult. They lie outside the formal Assurance process, but are the most important part of the environment in which security policy must be determined.
Policies are often ignored: people end up protecting the wrong things, or protecting the right things in the wrong way.
Mechanisms regularly make the news. For example, export control led to DVDs being released with 40-bit keys, which were inherently vulnerable. The difficulty of protecting smart cards from probing and from power attacks led the industry to focus on protecting other, relatively unimportant things, such as the secrecy of chip masks.
Assurance traditionally focused on the implementation (finding and fixing stack overflows). In recent years, with the transition from the "Orange Book" to the Common Criteria, policy has also entered the general Assurance framework through the mechanisms of protection profiles.

However, the biggest factor missed in the traditional approach to evaluation is usability. Most failures at the system level (as opposed to purely technical ones) have a significant human component. Designers often regard assurance simply as the absence of obvious bugs, tying together technical protection mechanisms but not taking human weakness into account. (There are honorable exceptions: accounting systems are from the outset designed to cope both with errors and with intent, and security printing technologies are optimized to make it easier for ordinary people to find counterfeits.) Usability concerns not only end users, but also developers. Access control elements in standard OSes are often not used, because it is much simpler for programmers to make code work with administrator privileges.
The above factors are interdependent and in many cases come into conflict with the incentives. Customers and suppliers often want different things.
A rational PC user, for example, wants: high usability, medium assurance (high would be expensive, and a rare virus can be put up with), high mechanism strength (they do not cost much more) and simple functionality (usability is more important). But the market does not provide this.
Commercial platform suppliers choose rich functionality (each feature brings benefit to a more concentrated group of users than its cost, rapid version changes prevent the commoditization of the market, and third-party developers who capture too large a market share can be suppressed), low mechanism strength (with the exception of cryptography, where the escrow debate forced them to regard strong cryptography as the most important marketing feature), low implementation assurance (military cryptography is easily bypassed by "Trojan horses") and low usability (application programmers-developers are far more important for network effects than customers).
Earlier, on economics, we described why this will not change in the near future. Companies struggling for dominance on platform markets begin by supplying too little security, since it hinders the third-party developers (complementers) to whom they must appeal, and the security that they do supply is often the "wrong" security, since it is intended to shift costs onto users, even if those costs would better be borne by the third-party developers. Once a firm achieves dominance on a platform market, it adds security, but again the "wrong" security, since its incentive now is the maximum lock-in of its customers. We have seen this not only on the PC platform, but on mainframes, mobile telephones and even telephone exchanges. Thus suppliers provide less security than a rational customer would want and, in turn, the customer wants less than would be socially optimal, since many of the costs of connecting an insecure machine to the Internet fall on others.
The ideals of government agencies also break against economics. They would like to be able to buy commercial off-the-shelf products (COTS), replace a small number of components (for example, by plugging in cryptographic cards instead of the standard cryptography for secret versions) and simply use the result in defense networks. Therefore they want multilevel functionality and high implementation assurance. This wish list is unrealistic not only because of the cost of high assurance, but also because of the priority of time to market, the need to please the developer community and the need for frequent changes of product version in order to prevent the commoditization of markets. In addition, a million government computer users cannot expect to impose their will on 500 million users of Windows and Office.
It is precisely in this treacherous landscape, with its multitude of irresolvable political conflicts, that the game of Assurance is played out.

Project Assurance

Assurance as a process is very similar to the development of code or of documents. Just as there will be bugs in your code, there will also be errors in your testing procedures. Therefore Assurance may be a one-off project, a subject of continuous evolution, or something in between (for example, a spiral model, where a fixed number of prototyping stages is used to generate a mass of test data that were not clear at the beginning).
Nevertheless, it is useful first to consider the project questions, and then the evolutionary ones.

Security Testing

In practice, security testing usually comes down to reading the product documentation, then to analyzing the code and finally to performing a series of tests (this is known as white-box testing, as opposed to black-box testing, where the tester has the product but no design documentation or source code). The process looks like this:

1. First, architectural flaws are looked for. Does the system use guessable or overly permanent session identifiers? Is there a way to inject code, for example by pushing SQL through the web server into the backend database? Is the security policy consistent, or are there gaps in the assumptions? Do they lead to gaps between mechanisms, where malicious things can be done?
2. Then implementation flaws are looked for, such as stack overflow and integer overflow. This usually involves not only reviewing the code, but using specialized tools such as fuzzers.
3. Then a list of less common flaws is worked through. If the product uses cryptography, weak keys are sought (or keys set to permanent values) and bad random number generators; if there are components with different assumptions about trust, attempts are made to manipulate the API between them, looking for race conditions and other combinations of transactions that have malicious effects.

This is an extremely telegraphic summary of one of the most fascinating and difficult tasks in the world. Many of the things that a security tester needs to look for were described in the earlier parts of security engineering.

Formal Methods

An example of a formal method is the BAN logic, which can be used to check certain properties of cryptographic protocols. The view of a practicing engineer on formal methods may be that they are widely taught in universities but not used in the real world. This is not quite true in the security business. There exist problems, such as the design of cryptographic protocols, where intuition is often insufficient and formal verification can be useful. Military customers go further and require their use as a condition for higher levels of evaluation under the Orange Book and the Common Criteria. However, this limits the high levels of evaluation to relatively small and simple products, such as line encryption devices and operating systems for primitive computers such as smart cards.
Even in this case formal methods are not infallible. Proofs too can contain errors, and often what is proved is not what is needed. Lars Knudsen's quotation ("If it is provably secure, then it probably is not") refers to the large number of breaks of cryptographic algorithms or protocols that had previously been proved secure. These breaks usually occur because one of the assumptions of the proof is unrealistic or has become so over time.

Quis Custodiet? (Who Will Watch the Watchmen?)

Just as errors can be made by theorem provers and testers, they can also be made by the people who draw up the checklists of things to test (and by the authors of security textbooks, from whose works the compilers of checklists draw their ideas). This is the old problem of quis custodiet ipsos custodes, as the Romans put it more succinctly.
There are several things that can be done, and few of them are likely to please an organization whose goal is to obtain a declaration that the product has no defects. The obvious option is fault injection, in which a number of errors are deliberately introduced into the code at random. If there are a hundred such errors and the tester finds seventy of them plus another seventy that were not deliberately introduced, then after removing the remaining thirty deliberate errors one can expect that about thirty bugs unknown to you remain. (This assumes that the errors unknown to you are distributed in the same way as the known ones; in reality things will almost always be worse.)
Even in the absence of deliberate bug injection, a rough estimate can be obtained by looking at which bugs are found by which testers.
Thus we obtain feedback from the rate at which instances of known bugs are discovered in products after their release. Another factor is the rate at which new attacks are discovered. In the university system we train postgraduate students by allowing them to attack different things; new vulnerabilities and exploits make their way into research papers, which bring fame and ultimately promotion. The incentives in government agencies and corporate laboratories are slightly different, but the general effect is the same: a large group of able, motivated people is looking for new exploits.

Process Assurance

In recent years ever less attention is paid to product-focused Assurance measures, such as testing, and ever more to process measures, such as who developed the system. As anyone who has been involved in systems development knows, some programmers write code with an order of magnitude fewer bugs than others. There also exist organizations that produce code of far higher quality than others. This is a subject of close attention in the industry.
As noted earlier, some of the differences between high- and low-quality developers lend themselves to direct managerial intervention. The really important factor is whether people bear responsibility for fixing their own code. In the 1980s many organizations interpreted the waterfall model of systems development in such a way that one team wrote the specification, another wrote the code, a third did the testing (including fixing some of the bugs), and a fourth did the maintenance (including fixing the rest of the bugs). They communicated with each other only through the design documentation. This was justified on the grounds that it is more efficient for people to concentrate on one task at a time, so interrupting a programmer with a request to fix a bug in code he wrote six months ago and has forgotten may cost a day of productivity, whereas bringing in a support programmer for this may cost only an hour.
But the effect was that coders produced megabytes of poor-quality code and left the poor testers to clear up the mess. Over time both quality and productivity sagged. Industrial analysts attributed the experience of IBM, balancing on the brink of death in the early 1990s (which cost more than 100 billion dollars of asset value), to precisely this.
For its part, Microsoft considers that one of the most important lessons it drew in its struggle with the problems of writing ever larger programs was the introduction of a firm policy of "if you wrote it, you fix it". Bugs must be fixed as soon as possible and, even though they are as inevitable as death and taxes, programmers must never give up trying to write clean code. One of the consequences was that the people who wrote buggy software ended up spending most of their time finding and fixing bugs in their own code, so most of the code base ended up being written by the neater programmers.
Other controlled aspects of the organization can also have a significant influence on the quality of the product being released, ranging from how bright your hires are to how you train them and what working habits you instill.
For some time internal auditors have included process questions in the assessment of the quality of security code. This is harder to do than it may seem, because much of an organization's quality culture is intangible. Although some rules (such as "fix your own bugs") seem fairly universal, imposing a large number of specific rules would give rise to a bureaucratic "tick-box" culture rather than a dynamic competitive environment. Therefore recent work has aimed at a more holistic assessment of a team's capabilities; the main contender here is the Capability Maturity Model (CMM) from the Software Engineering Institute at Carnegie Mellon University.
CMM is based on the idea that competence is a function of teams, and not simply of individual developers. There is something more in a group than simply the gathering of half a dozen competent musicians, and the same applies to software. Developers start out with different coding styles, different conventions for commenting and formatting code, different ways of managing APIs and even different rhythms of workflow. As described earlier, a capable project manager will unite them into a team, ensuring agreement about conventions and ways of working, developing an understanding of the strengths of different people and matching them better with tasks. Carnegie Mellon's research showed that newly formed teams tend to underestimate the volume of work in a project and also to have a high variance of the time they require; the teams that worked together best were far better able to predict how much time they would need, in terms of average development time, but also reduced the variance.
Now, one of the problems is that firms constantly reorganize for various reasons, and this disrupts the work of established and capable teams. How can one counteract this in order to preserve capability? Well, CMM offers a certification process through which established teams can obtain the status of an asset that cannot be frivolously scattered. The details of the model are perhaps less important than this institutional role. But in any case the model has five levels: initial, repeatable, defined, managed and optimizing, with a list of new things to be added as one moves up the hierarchy. Thus, for example, project planning must be introduced to move from "initial" to "repeatable", and peer reviews to move from "repeatable" to "defined".
An even more common approach to process assurance is the ISO 9001 standard. In essence, a company must document its processes of design, development, testing, documentation, audit and management in general. At best this can provide a framework for the gradual improvement of processes; companies can track what is going wrong, go back to the source, fix it and prevent a recurrence. But very often ISO 9001 turns into an exercise in covering one's back and ticking boxes, which merely replaces chaos with a more bureaucratic chaos.
Many writers have noted that organizations have a natural life cycle, just as people do; Joseph Schumpeter argued that economic depressions perform a valuable social function, clearing out companies that are past their peak or simply unfit, just as fires rejuvenate forests. It is undoubtedly true that successful companies become complacent and bureaucratic. Many insiders choose the easy life, while others, more ambitious, leave: it used to be said that the only people who ever left IBM were the good ones. Too rapid growth also brings problems: Microsoft insiders blame many of the security and other problems of Windows products in the 1990s on the influx of tens of thousands of new employees, many of whom were motivated more by the prospect of earning millions on options and on the stock than by the mission of writing good code and running it on every computer in the known universe.
The cycle of corporate birth, death and reincarnation in the computer industry proceeds far faster than in other industries, thanks to a combination of Moore's law, network effects and a dashing entrepreneurial culture. The telecommunications industry suffered a serious trauma when two industries merged and the fifteen-year product development cycles of the telephone companies collided with the fifteen-month cycles of Cisco, Microsoft and others. The information security industry is experiencing the same pressure. Teams that for decades had steadily worked on cost-plus contracts to develop encryptors or MLS systems for the NSA suddenly faced brutal technological and market forces and were compelled to create entirely different things. Some succeeded: the MLS supplier TIS reinvented itself as a supplier of firewalls and antivirus software. Others failed and disappeared. Thus management may come to doubt the value of a team of grey-haired MLS veterans. And expert teams may depend on one or two key gurus; when they leave to found a startup, the team's capability may evaporate overnight.
A point often overlooked is that Assurance schemes, like cryptographic protocols, must support revocation. CMM would be more powerful if teams that had lost their stars, their sparkle or their relevance also lost their accreditation. Perhaps we should rather have a ranking system of the kind used in culinary guides, where you cannot declare a new establishment "the best Asian restaurant in San Francisco" unless you displace the reigning champion. Of course, if certification were a more perishable asset, it would have to give a greater market advantage, so that companies would invest as much effort in obtaining it. But on the whole the restaurant guide system works, and academic peer review works on roughly the same principle.

Assurance Growth

Another aspect of process assurance is that most customers are interested not so much in the development team as in its product. But most software is now packaged rather than custom, and is developed by continuous evolutionary improvement rather than within the framework of a one-off project. So what can usefully be said about the level of Assurance of evolving products?
The quality of such a product can reach equilibrium if the rate of introduction of new bugs into the product equals the rate of finding and removing old bugs. But there is no guarantee that this will happen. (There are second-order effects, such as aging — when repeated improvements make the code so complex that its basic reliability and maintainability fall — but for the sake of simplicity we will ignore them.)
While control of the rate of bug introduction depends on the development controls described above, measuring the rate of bug removal requires other tools — models of how software reliability improves under testing.
Quite a lot is known about growth of reliability, since it is of interest to far more people than just software engineers. Where a tester is trying to find one bug in a system, a reasonable model is the Poisson distribution: the probability p that a bug will remain undetected after t statistically random tests is defined as p = e^(-Et), where E depends on the proportion of possible input data that it affects. Thus, when the reliability of a system is dominated by a single bug — as when we are looking for the first bug in a system or the last one — the growth of reliability can be exponential.
But extensive empirical studies have shown that in large and complex systems the probability that the t-th test will end in failure is proportional not to e^(-Et), but to k/t for some constant k. Thus the reliability of the system grows far more slowly. This phenomenon was first documented in the bug history of IBM mainframe operating systems and confirmed in many other studies. Since a failure probability of k/t means a mean time between failures (MTBF) of about t/k, reliability grows linearly with testing time. This result is often formulated by the safety-critical systems community as follows: "If you want the mean time between failures to be a million hours, then you must test (at least) a million hours".
This was one of the main arguments against the development of complex, critical systems that cannot be fully tested before use, such as missile defense.
The reason for the k/t behavior was identified in [a] work, and was then proved under much more general assumptions. The model gives a number of other interesting results. Under assumptions that are often reasonable, it is the best possible: the rule that you need a million hours of testing to obtain a million hours of MTBF is inevitable, up to some constant multiplier that depends on the initial quality of the code and on the scale of testing. This amounts to proving a version of "Murphy's Law": the number of defects that survive the selection process is maximized.
The evolutionary model resembles the mathematical models of the evolution of a biological species under selective pressure. The role of "bugs" is played, roughly speaking, by genes that reduce fitness. But some of the consequences differ noticeably. "Murphy's Law", stating that the number of defects surviving the selection process is maximized, may be bad news for the engineer, but it is good news for biological species. While software testing removes the minimum possible number of bugs consistent with the tests applied, biological evolution allows the species to adapt to a changed environment with the minimum cost of early deaths and meanwhile preserves as much diversity as possible. This diversity helps the species to survive future ecological shocks.
For example, if a population of rabbits is hunted by snakes, they will be selected for vigilance rather than for speed. Diversity in speed will be preserved, so if foxes appear in the neighborhood, the average running speed of the rabbit population will rise sharply under selective predation. More formally, the fundamental theorem of natural selection states that a species with high genetic variance can adapt faster to a changing environment. But when Sir Ronald Fisher proved this in 1930, he was also proving that complex software will exhibit the maximum possible number of bugs when you transfer it into a new environment.
The evolutionary model also points to fundamental limits on the reliability that can be obtained from reusable software components, such as objects or libraries; well-tested libraries simply mean that the general failure rates will be dominated by the new code. It also explains the observation of the safety-critical systems community that test results are often a poor indicator of performance: the failure time measured by a tester depends only on the initial quality of the program, the scale of testing and the number of tests, so it gives practically no additional information about the probable performance of the program in another environment.
There are also some results that are unexpected but obvious in hindsight: for example, the contribution of each bug to the overall failure rate does not depend on whether the code containing it is executed often or rarely — intuitively it is clear that code that is executed less often is also tested less often. Finally, as I mentioned earlier, it is often more economical for different testers to work on a program in parallel rather than sequentially.
Thus, complex systems become reliable only after prolonged testing. The wide use of mass software in principle allows thorough debugging to be carried out, but in practice the constant new versions dictated by the network economy impose serious limits on what can reasonably be expected.

Evolution and Security Assurance

Evolutionary growth of reliability can be far worse for the software engineer than for the biological species, but for the security engineer it is worse still.
Rather than going into detailed mathematics, let us consider a simplified example. Suppose that a complex product such as Windows Vista has 1,000,000 bugs, each with a mean time between failures (MTBF) of 1,000,000,000 hours. Suppose that Ahmed works in bin Laden's cave, where his task is to hack the US Army's network in order to obtain a list of informers in Baghdad, while Brian is the Army's assurance guy, whose task is to stop Ahmed. Therefore he must learn about the bugs before Ahmed does.
Ahmed has to keep moving to avoid the Pakistani army, so he can carry out only 1000 hours of testing a year. Brian has the complete source code of Vista, dozens of PhDs, control over commercial evaluation laboratories, an inside track into CERT, an information-sharing agreement with the other UKUSA member countries, and he also runs a government scheme sending consultants into critical industries such as energy and telecommunications to find out how to hack them (sorry, to advise them how to protect their systems). Therefore Brian carries out 10,000,000 hours of testing a year.
After a year Ahmed finds one bug and Brian finds 10,000. But the probability that Brian has found Ahmed's bug is only 1%. Even if Brian calls up 50,000 computer science graduates to Fort Meade and makes them comb through the Windows source code, he will still be performing only 100,000,000 hours of testing each year. After ten years he will find Ahmed's bug. But by then Ahmed will have found nine more, and it is unlikely that Brian will know about all of them. Worse, Brian's bug reports will become such a powerful stream that Bill will stop fixing them.
In other words, thermodynamics is on Ahmed's side. Even a moderately resourced attacker can hack anything that is large and complex. There is nothing that can be done to stop this, for as long as there exist enough different security vulnerabilities for statistics to apply. In real life vulnerabilities correlate rather than being independent; if 90% of your vulnerabilities are stack overflows and you introduce a compiler technology to catch them, then from the point of view of modeling there was only one vulnerability. However, new ones appear constantly. Therefore, if you really are responsible for the security of the army, you cannot simply rely on a large complex commercial off-the-shelf product. You must have mandatory access controls implemented in something simple enough that it can be verified. Simplicity is the key to getting out of the statistical trap.

System Evaluation, Ways Forward

Introduction to Evaluation

The working definition of evaluation sounds like "the process of assembling evidence that a system does or does not meet a prescribed assurance target". Evaluation may be needed simply to convince your boss that you have completed the work, but often it is required in order to reassure the parties who will rely on the system. The fundamental problem lies in the tension that arises when the party implementing the protection and the party relying on it are different people.
Sometimes this tension is simple and quite manageable, as in the case where you design a burglar alarm to the standards set by insurance underwriters and it is certified by inspectors in their laboratories. Sometimes it is still noticeable but more complex, for example when designing to government security standards that try to reconcile dozens of conflicting institutional interests, or when hiring your company's auditors to check a system and report to the boss on its suitability. The situation becomes even more complex when many parties are involved, for example when a smart card vendor wants to obtain an evaluation certificate from a government agency (which is trying to promote the use of some feature, such as key escrow, that is in nobody else's interest) in order to sell the card to a bank, which in turn wants to use it to shift responsibility for fraud onto its customers. This may seem thoroughly unfair, and yet none of the people involved necessarily commits any overtly criminal act. Unfairness can be an emergent property arising because managers follow their personal and departmental incentives.
For example, managers often buy products and services that they know to be suboptimal or even defective, but which are supplied by large and well-known vendors — simply to minimize the probability of being fired when something goes wrong. (Formerly, 20 years ago, it was said that "no one was ever fired for buying IBM"). Corporate lawyers do not condemn this as fraud, but praise it as due diligence. The end result may be that someone who relies on the system — in this case the bank's customer — has no right of voice and will find it difficult to obtain compensation from the bank, the vendor, the evaluator or the government when something goes wrong.
Another serious and ubiquitous problem is that the words "assurance" and "evaluation" are often interpreted in such a way that they apply only to the narrow technical aspects of the system, ignoring systemic questions such as usability, let alone organizational questions such as proper internal control and good corporate governance. Company directors also want assurance: that prescribed procedures are observed, that there are no material errors in the reports, that the applicable laws are complied with, and dozens of other things. But many evaluation schemes (especially the Common Criteria) studiously ignore the human and organizational elements in the system. If any attention is paid to them at all, the evaluation of these elements is considered the business of the customer's IT auditors or even of the system administrator tuning the configuration files.

Nevertheless, below I will concentrate on technical evaluation.

Evaluation by the Relying Party

It is convenient to divide evaluation into two cases. The first is when the evaluation is performed by the party relying on the system; this includes insurance evaluations, the Independent Verification and Validation (IV&V) carried out by NASA for mission-critical code, and the previous generation of military evaluation criteria such as the Orange Book. The second case is when the evaluation is performed by someone other than the relying party. Nowadays this often means the Common Criteria.
Earlier we discussed the problems that concern insurers with regard to physical security systems, and how they approach the approval of equipment for use with particular sizes of risk. The approval process itself is simple enough: the insurance industry operates laboratories where the tests are conducted. They may include a fixed budget of effort (perhaps one person for two weeks, or a cost of $15,000). The evaluator starts with a fairly clear idea of what, for example, a burglar alarm should and should not do, spends the budgeted amount of effort looking for defects and writes a report. The laboratory then either approves the device, rejects it, or requires certain changes to be made.
The main failure mode of this process is that it does not always respond well enough to progress in attack technologies. In the case of high-security locks, the laboratory may require ten minutes of resistance to picking and say nothing about bumping. Nevertheless, bumping tools have improved so much that they represent a serious threat, and picks too have become better. A product that received its certificate ten years ago may now be easily opened by bumping (so the standard is obsolete) and is also vulnerable to being picked in 2–3 minutes (so the test result is obsolete too). Insurance laboratories in some countries, such as Germany, have been prepared to withdraw certifications as attacks improved; in others, as in the USA, they do so reluctantly, perhaps for fear of being taken to court.
I mentioned another model of evaluation — the one carried out from 1985 to 2000 at the US National Computer Security Center (NCSC) for computer security products offered for government use. These evaluations were carried out in accordance with the Orange Book — the Trusted Computer Systems Evaluation Criteria. The Orange Book and its accompanying documents established a series of evaluation classes:

• C1: discretionary access control for groups of users. In essence, this is considered equivalent to the absence of protection.
• C2: discretionary access control for individual users, object reuse, audit. C2 corresponds to carefully tuned commercial systems; for example, C2 evaluations were given to IBM mainframe operating systems and to Windows NT. (Both were conditional; in the case of NT, for example, it was limited to diskless workstations.)
• B1: mandatory access control — all objects have security labels, and the security policy (which means Bell-LaPadula or a variant of it) is enforced independently of user actions. Labeling is provided for all incoming information.
• B2: structured protection — as B1, but there must also exist a formal model of the security policy that is provably consistent with the security axioms. Tools must be provided for administering the system and for configuration management. The TCB (Trusted Computing Base) must be properly structured and its interface clearly defined. A covert channel analysis must be carried out. A trusted path from the user to the TCB must be provided. Rigorous testing must be carried out, including penetration testing.
• B3: security domains — as B2, but the TCB must be minimal, it must mediate all access requests, it must be tamper-resistant and it must withstand formal analysis and testing. Real-time monitoring and alerting mechanisms must exist, and structured techniques must be used in the implementation.
• A1: verified design. As B3, but formal methods must be used to prove the equivalence between the TCB specification and the security policy model.

The evaluation class of a system determined what range of information could be processed on it. The example given earlier was that a system evaluated at B3 could process information at the Unclassified, Confidential and Secret levels, or at the Confidential, Secret and Top Secret levels.
When the Orange Book was written, the Department of Defense considered that the real problem was that the computing markets for defense were too small, which led to high prices. The solution was to expand the market for high-assurance computing. Thus the goal was to develop protection measures that would be standard in all the main operating systems, rather than a costly add-on for captive government markets.
However, the business model of Orange Book evaluations followed traditional government working practices. The government user wanted to evaluate some product, the NSA allocated people for this; given the traditional caution and delays of the civil service, this could take two or three years; the product, if successful, joined the list of evaluated products, and the bill was paid by the taxpayer. The process was directed and controlled by the government — the party that was going to rely on the results of the evaluation — while the vendor was a supplicant at the gate. Because of the time the process took, evaluated products usually lagged one or two generations behind current commercial products. This meant that evaluated products were simply unacceptable on commercial markets. The defense computing market remained small, and prices high.
The Orange Book was not the only evaluation scheme operating in America. I mentioned the FIPS 140-1 scheme for evaluating the tamper resistance of cryptographic processors; it uses a number of independent laboratories as contractors. Contractors are also used for Independent Verification and Validation (IV&V), a scheme created by the Department of Energy for systems used in nuclear weapons and later adopted by NASA for manned space flight, which has many similar components (at least in the rocket-building part). In IV&V there is a simple evaluation goal — zero defects. The process is still managed and controlled by the relying party — the government. The IV&V contractor is a competitor of the company that built the system, and his pay is tied to the number of errors found.
Other governments had similar schemes. The Canadians had the Canadian Trusted Product Evaluation Criteria (CTPEC), while a number of European countries developed the Information Technology Security Evaluation Criteria (ITSEC). The idea was that a common evaluation scheme would help European defense contractors compete with US suppliers with their larger economies of scale; they would no longer have to undergo separate certification in the UK, France and Germany. ITSEC combined ideas from the Orange Book and from the IV&V processes in that there existed a number of different evaluation levels, and for all but the highest of these levels the work was outsourced. However, ITSEC introduced a pernicious innovation — the evaluation was paid for not by the government, but by the vendor seeking an evaluation of his product. This was an attempt to kill several birds with one stone: to save government money and at the same time to promote a more competitive market. However, the questions of incentives were not properly thought through.
This change of rules motivated the vendor to look for an evaluation contractor who would give his product the easiest ride, whether by asking fewer questions, charging less money, spending less time, or all of the above. To be fair, the potential for this was recognized, and schemes were created through which contractors could obtain approval as a Commercial Licensed Evaluation Facility (CLEF). The threat that a CLEF might lose its license was supposed to offset the commercial pressure to cut corners.

Common Criteria

This sets the stage for the Common Criteria. The Department of Defense began to realize that the Orange Book did not facilitate procurement, and contractors hated the need to obtain separate evaluations for their products in the USA, Canada and Europe. After the collapse of the Soviet Union in 1989 budgets began to be cut, and it was not clear where the able, motivated adversaries of the future would come from. The mood was in favor of a radical reduction of costs. In the end agreement was reached to abandon the national evaluation schemes and replace them with a single standard, the Common Criteria for Information Technology Security Evaluation.
The work was largely completed in 1994–1995, and the European model defeated the American and Canadian alternatives. As in ITSEC, evaluations at all levels except the highest are performed by CLEFs and must be recognized in all the participating countries (although any country may refuse to recognize an evaluation if it says that its national security is threatened), and the vendors pay for the evaluations.
There are some differences. Most importantly, the Common Criteria possess far greater flexibility than the Orange Book. Instead of expecting all systems to conform to Bell-LaPadula, a product is evaluated for conformity to a protection profile. There exist protection profiles for operating systems, access control systems, border control devices, intrusion detection systems, smart cards, key management systems, VPN clients and even dustbin identification systems — for transponders that determine when a household dustbin was last emptied. The tent is certainly far wider than with the Orange Book.
However, anyone may propose a protection profile and have it evaluated in a laboratory of his choice, so there are many profiles. The point is not so much that the Department of Defense abandoned multilevel security, as that it tried to make commercial IT vendors use the system for other purposes as well, and thus to defeat the pernicious incentives described above. The aspiration was to create a bandwagon effect that would lead the commercial world to some extent to adapt to the government way of doing things.
Its success has been mixed. The only major case that comes to mind, where a large firm used evaluations in its general marketing, was Oracle, which launched a marketing campaign after 9/11 describing its products as "Unbreakable", claiming that each of its fourteen security evaluations "represented an additional million dollars of investment in security". (This campaign petered out after a number of people found ways to hack their products.) But there were quite a few cases where evaluations were used in more specialized markets; we have already discussed bank card terminals (which turned out to be easy to hack) in the earlier parts of security engineering. The smart card industry was a particular devotee of evaluation: in 2007 there existed no fewer than 26 smart card protection profiles and a number of derivative profiles for smart-card-based devices such as TPMs and electronic signature creation devices. But perhaps this worked too well. There is a large choice, and it is not easy to understand what an evaluation certificate actually means. (We encountered this problem before, when some cryptographic processors were certified as secure hardware devices but were easy to hack, since they were used to run unsuitable software.)

To discuss the Common Criteria in detail, we need a little more jargon.

• The product undergoing testing is known as the target of evaluation (TOE). The rigor with which the examination is conducted is the evaluation assurance level (EAL) and can range from EAL 1, for which functional testing suffices, to EAL 7, for which not only thorough testing but also a formally verified design is required. The highest level of evaluation usually obtained for commercial products is EAL4, although there was one smart card operating system at the EAL6 level.
• A Protection Profile consists of security requirements, their justification and the EAL. It is supposed to be expressed in an implementation-independent way, so as to provide comparable evaluations for different products and versions. A Security Target (ST) is a refinement of the protection profile for a given target of evaluation.
• Besides evaluating a particular target, one can evaluate a protection profile (to make sure that it is complete, consistent and technically sound) and a security target (to check whether it correctly refines the given protection profile). When developing something from scratch, the idea is first to create a protection profile and have it evaluated (if a suitable one does not already exist), then to do the same for the security target and finally to evaluate the actual product. The end result of all this is a registry of protection profiles and a catalogue of evaluated products.

A protection profile must describe the assumptions about the environment, the objectives and the protection requirements (both in terms of function and of assurance) and break them down into components. There is a stylized way of doing this. For example, FCO NRO is a functionality component (hence the F) relating to communications (CO), and it relates to non-repudiation of origin (NRO). Other classes include FAU (audit), FCS (cryptographic support) and FDP, which means data protection (this is not data protection in the European legal sense, but means access control, Bell-LaPadula information flow control and related properties).
There exist catalogues of threats, such as T.Load Mal — "Data load failure: an attacker may maliciously generate errors in the setting data in order to compromise the security functions of the TOE", of assumptions, such as A.Role Man — "Role management: role management for the TOE is performed in a secure manner" (in other words, developers, operators and so on behave properly), of organizational policies, such as P.Crypt Std — "Cryptographic standards: cryptographic entities, data authentication and approval functions must conform to ISO standards and to related industry or organizational standards", of objectives, such as O.Flt Ins — "Fault insertion: the TOE must be resistant to repeated probing by means of the insertion of erroneous data", of assurance requirements, such as ADO DEL.2 — "Modification detection: the developer must document the procedures for delivery of the TOE or its parts to the user".
I mentioned that a protection profile will contain a rationale. Usually it consists of tables showing how each threat is controlled by one or several objectives, and in the reverse direction, how each objective is conditioned by some combination of threats or assumptions about the environment, together with accompanying explanations. It will also justify the choice of the assurance level and the requirements for mechanism strength.
The quickest way to understand the essence of this is probably to read a few existing profiles. You will understand that the quality varies greatly. For example, the Eurosmart protection profile for smart cards is largely concerned with maintaining the confidentiality of the chip design by imposing NDAs on contractors, destroying waste and so on, whereas in practice most attacks on smart cards used probing or power attacks, for which knowledge of the chip mask was of no importance. Another example of an unimpressive protection profile is the profile for automatic teller machines, which is written in management language, complete with clip art, "decided not to include any security policy" and misses even many of the problems that were well known and documented when it was written in 1999. Indeed, it says that it relies on the developer to document vulnerabilities and requires that "the evaluator determine that the TOE is resistant to penetration attacks performed by an attacker possessing a moderate attack potential". A better example is the later profile developed by the German government for medical practitioners' cards — cards used by doctors and nurses to log into hospital computer systems. This profile examines in detail the possible threats associated with physical interference, power analysis, abuse of functionality and so on, and systematically ties protection mechanisms to control objectives.
So the first question you should ask when you are told that some product has a Common Criteria evaluation is: "on the basis of which protection profile?" Is it a secret design that will be resistant to a "moderate attack", or was it evaluated on the basis of something thorough that was thought through by people who know what they are doing? The interesting thing is that all three profiles that I mentioned in the paragraph above are intended for evaluation at the EAL4+ level — so it is not the nominal CC level that tells you something, but the details of the PP.
The Criteria do say that "the fact that an IT product has been evaluated has meaning only in the context of the security properties that were evaluated and the evaluation methods that were used", but it goes on to say that both "Evaluation bodies must carefully examine products, properties and methods in order to determine that the evaluation will provide meaningful results" and "purchasers of evaluated products must carefully consider this context in order to determine whether the evaluated product is useful and applicable to their particular situation and needs". So who actually bears the responsibility? Well, we discover that "the CC does not address the administrative and legal framework within which the criteria may be applied by evaluation bodies" and that "the procedures for using evaluation results in accreditation are outside the scope of the CC".
The Common Criteria can sometimes be useful to the security engineer in the sense that they give you an extensive list of things to check, and the structure can also help in tracking all the various threats and ensuring that they are all considered systematically (otherwise it is very easy for one of them to be forgotten in the mass of detail and to slip past). What people often do, however, is find a protection profile that seems reasonably close to what they are trying to do and simply use it as a checklist without further thought. If you choose a profile that fits badly, or one of the old and vague ones, you are asking for trouble.

What the Common Criteria Do Not Do

It is also important to understand the limitations of the Criteria. The documents state that they do not concern measures of administrative security, nor "technical physical" aspects such as Emsec, nor cryptographic algorithms, nor evaluation methodology (although there is a companion volume on this), nor how the standards should be used. They state that they do not assume any particular development methodology (but then go on to assume a waterfall approach). There is a nod in the direction of the evolution of policy in response to experience, but the re-evaluation of products is declared to be out of scope. Oh, and there is no requirement for evidence that the protection profile corresponds to the real world, and I have seen several that studiously ignored published work on the relevant vulnerabilities. In other words, the Criteria avoid all the difficult and interesting parts of security engineering and can easily become a charter for cherry pickers.
The most common specific criticism (besides cost and bureaucracy) is that the Criteria are too focused on the technical aspects of design: things like usability are almost ignored, and the interaction of a firm's administrative procedures with the technical protection means is hushed up as being out of scope.
Another common criticism is that the Criteria cope badly with change. Many commercial products have an evaluation that is essentially meaningless: operating systems such as Windows and Linux have been evaluated, but in very limited configurations (typically a workstation without network connectivity or removable media — where all the evaluation says is that there is a login process and that access control to the file system works). Products with updates, such as Windows with its monthly security patches, are out of scope. This can lead directly to insecurity: it was noted that the first real distributed denial-of-service attack used hospital PCs, since the evaluation and accreditation process used at the time forced hospitals to use an obsolete and therefore insecure configuration.
In fact very often, when you read the small print of a PP, you discover that only some very basic functions of the product were evaluated. The vendor may have evaluated the boot code but left most of the operating system out of scope. This applies not only to CC evaluations but also to FIPS-140; it was told earlier how the IBM 4758 was evaluated by the US government to the highest standard of physical tamper resistance, and yet it was easy to hack. The evaluators looked only at the tamper-resistant platform, while the attackers also looked at the software running on it.
Yet another fundamental problem is that the Criteria are technology-oriented, whereas in most applications it is the business processes that must determine the protection decisions. Technical mechanisms should not be used when the impact is less than the cost of controlling it, or when procedural protection means are cheaper. Remember why Samuel Morse defeated dozens of other people who were racing to build electric telegraphs in the early nineteenth century. They were trying to build modems so that they could deliver text from one end to the other; Morse realized that with the technology available then it was cheaper to train people to be modems.
Over the last decade Ross Anderson must have been through half a dozen disputes about whether protection mechanisms had been correctly designed or implemented, and in which the Common Criteria were somehow involved. In not one of them did their approach turn out to be satisfactory. (Perhaps this is because he was called in only when something goes wrong — but his experience nevertheless points to a lack of robustness in the process.) There exist not only technical limitations and limitations of scope, there are also pernicious incentives and political failures.

Corruption, Manipulation and Inertia

Common Criteria evaluations are performed by CLEFs — contractors who are "Commercial Licensed Evaluation Facilities". They are licensed by the local signals intelligence agency (the NSA in America or its analogues in the UK, Canada, France, Germany, Spain, the Netherlands, Australia, New Zealand and Japan). One of the consequences of this is that their staff must have security clearances. As a result, CLEFs are quite heavily dependent on the local spies.
One egregious example occurred in the British National Health Service (NHS). The service agreed, under pressure from doctors, to encrypt traffic on the health service network; the signals intelligence service GCHQ made it clear that it wanted products with key escrow to be used. Trials were organized; one of them used commercial encryption software from a Danish supplier, which had no key escrow and cost £3,000, while another used software from a British defense contractor, which had key escrow and cost £100,000. To GCHQ's embarrassment, the Danish software worked, but the British supplier produced nothing that was fit for use. The situation was quickly rectified by having a company with a CLEF license conduct an evaluation of the trials. In their report they asserted exactly the opposite: that the escrow software worked excellently, while the foreign product had all sorts of problems. Perhaps the CLEF was simply told what to write, or perhaps the staff wrote what they knew GCHQ wanted to read.
The second problem is that, since it is the vendor who pays the CLEF, the vendor can look for a CLEF that will give him an easy ride technically, or that will follow his political line. In the context of the Icelandic health database [discussed] above, its promoters wanted to defuse criticism from doctors about its privacy problems, so they hired a British CLEF to write a protection profile for them. This profile simply repeated, in the jargon of the Criteria, the original design and the promoters' claims; it studiously avoided noticing the defects in that design that had already been documented and even discussed on Icelandic TV.
Sometimes protection profiles may be sound, but the way they are mapped onto the application is not. For example, smart card vendors lobbied European governments to adopt laws obliging business to recognize digital signatures made using smart cards; a number of protection profiles was duly written for a smart card functioning as a "Secure Signature-Creation Device". But the main problem in such an application is the PC, which displays to the citizen the material that he thinks he is signing. Since this problem is too complex, it is excluded, and the end result will be a "secure" (in the sense of non-repudiable) signature on whatever a virus or Trojan on your PC sent to your smart card. Electronic signatures did not take off; no sensible person will agree to be bound by any signature that appears to have been made by his smart card, regardless of whether he actually made it. (For comparison, in the world of paper a forged handwritten signature is completely invalid and cannot bind anyone.) In this case the greed of the vendors and the naivety of the legislators destroyed the market on which they hoped to profit.
Insiders devise even more sophisticated ways of manipulating the system. A good example here comes from how the French got round British and German objection to the smart-card-based electronic tachograph. The French wrote a relaxed protection profile and sent it to a British CLEF for evaluation. The CLEF was an army software company and, whatever their knowledge of MLS, they knew nothing about smart cards. But this did not lead them to turn down the business. They also did not know that the UK government was opposed to approving the protection profile (if they had known, they would no doubt have fallen into line). Thus the UK found itself faced with a choice between accepting defective road safety standards as a fait accompli or undermining confidence in the Common Criteria. In the end, no one in London had any desire to challenge the evaluation; eight years later the old paper tachographs began to be replaced by new, less secure, electronic ones.
As for the organizational aspects, assurance systems based on processes fail if accredited teams do not lose their accreditation when they lose their sparkle. This clearly applies to CLEFs. Even if CLEFs were licensed by a body independent of the intelligence community, many of them will deteriorate as key staff leave or skills fail to keep up with technology, and since customers look for easier evaluations, there will inevitably be both corruption and inflation of evaluations. As Ross Anderson said: "Nevertheless, at present I see no usable mechanism by which a practitioner possessing very weighty evidence of incompetence (or even dishonesty) can challenge a CLEF and have it removed from the list. In the absence of sanctions for misconduct, the incentive for CLEFs will be a race to the bottom and competition to provide vendors with an easy ride".
I described earlier how at the end of 2007 it was discovered that PIN entry devices certified as secure by both VISA and the Common Criteria were anything but secure; indeed, the protection profile on the basis of which they were evaluated could not be fulfilled. After reporting the vulnerability to the vendors, to VISA, to the bankers' trade association and to GCHQ, we pointed out that this was a failure not only of the laboratories that had evaluated these particular devices, but a systemic failure. What, we demanded, did VISA and GCHQ propose to do about it? The answer, as it turned out, was nothing. VISA refused to withdraw its certification, and the UK government said that, since the evaluation had not been registered with them, it was not their problem. Suppliers may continue to describe the defective terminal as "CC evaluated" and even "CC approved", for as long as they do not call it "CC certified" or "certified under the Common Criteria Recognition Arrangement (CCRA)". This seems a serious problem with the CC brand. It will continue to be stamped on lousy products for the foreseeable future.
So the best advice I can offer is as follows. When you are presented with a security product, you should always consider whether the salesman is lying or mistaken, and how. The Common Criteria were supposed to fix this problem, but they do not. When you are presented with an evaluated product, you should demand what vulnerabilities have been reported or discovered since the evaluation was carried out. (Get this in writing.) Then look closely at the protection profile: check whether it corresponds to what you really need. (Usually it does not — it protects the vendor, not you.) Do not limit your skepticism to the purely technical aspects: ask how it was manipulated and by whom, whether the CLEF that evaluated the profile was dishonest or incompetent, and what pressure from which government may have been applied behind the scenes.
You should also consider how your rights are eroded by a certificate. For example, if you use an unevaluated product to create digital signatures, and a forged signature appears that someone tries to use against you, you can reasonably expect to challenge the evidence by persuading the court to order full documentation to be disclosed to your expert witnesses. A Common Criteria certificate may make the court far less willing to order disclosure and thus may damage your rights.
A cynic might suggest that this is precisely why, in the commercial world, it is the vendors of products that are intended to transfer responsibility (such as smart cards), or to satisfy due diligence requirements (such as firewalls), who are most enthusiastic about the Common Criteria. A truly hardened cynic might point out that after the collapse of the Soviet Union the agencies justify their existence by economic espionage, and the Common Criteria participating countries provide most of the interesting targets. A fake evaluation of a product in the USA that is sold all over the world may compromise 250 million Americans, but since it will also compromise 400 million Europeans, the balance of advantage lies in deception. The balance is even stronger with small countries such as Britain and the Netherlands, which have fewer citizens to protect and more foreigners to attack. In addition, agencies receive praise (and budget) for the foreign secrets they steal, not for the domestic secrets that foreigners failed to steal.
So an economist is unlikely to trust a Common Criteria evaluation. It may be worth regarding them as a sort of rubber crutch. Such a device has all sorts of uses, from winning the sympathies of a judge to wheedling money out of a gullible government and hitting people over the head. (Just don't try to put any serious weight on it!)
Fortunately, economics should also limit the adoption of the Criteria to sectors where official certification, however irrelevant, erroneous or mendacious it may be, offers some competitive advantage.

Ways Forward

In his classic book "The Mythical Man-Month" Brooks convincingly argues that there is no "silver bullet" for solving the problems of software projects that run late and over budget. The easy parts of the problem, such as the development of high-level languages in which programmers are more productive, have been done. This removes much of the accidental complexity of programming, leaving the intrinsic complexity of the application. I discussed this in the earlier parts in the general context of systems development methodology; the discussion above should convince you that exactly the same applies to the problem of assurance and, especially, of evaluation.
A more realistic approach to evaluation and assurance would consider not only the technical functions of the product, but also how it behaves in real use. Usability is ignored by the Common Criteria, but in reality it all matters; the British government's email system, which required users to reboot their PCs whenever they changed compartments, so frustrated users that they made informal agreements to put everything into shared compartments — in effect wasting a nine-figure investment. (Official secrecy will no doubt continue to protect the guilty parties from punishment.) The critical functions that we discussed in the context of accounting systems in one of the parts, which are intended to limit the effects of human weakness, are also critical. In most applications it is necessary to assume that people are always careless, usually incompetent and sometimes dishonest.
It is also necessary to face the fact of large, feature-rich programs that are frequently updated. Economics cannot simply be wished away. Evaluation and assurance schemes such as the Common Criteria, ISO9001 and even CMM try to squeeze a highly volatile and competitive industry into a bureaucratic straitjacket in order to provide purchasers with the illusion of stability. But given the way the industry works, the best people can do is to rally around brands, such as IBM in the 70s and 80s, and Microsoft now. The creation and maintenance of these brands involves enormous market forces, and security plays a minor role.
I have given you enough hints as to how to cheat the system and pass off a lousy system as secure — at least long enough for the problem to become someone else's. In the remainder of this article I will assume that you are making an honest attempt to protect a system and want risk reduction, not due diligence or some other kind of responsibility shifting.
Nevertheless, there are still many systems where the owner of the system loses if security fails; we saw a number of them above (nuclear command and control, pay TV, prepaid utility meters and so on), and they provide many interesting engineering examples.

Adversarial Review

When you really want a protection property to hold, it is vital that the design undergo adversarial review. In the end it will happen anyway, and it is better if it is done before the system is deployed. As we saw in case after case, the motivation of the attacker is almost all-important; friendly reviews by people who want the system to pass are essentially worthless compared with the input of people who seriously try to hack it.
The classic ways of conducting an adversarial review are the contractual and the adversarial. An example of the contractual approach was the Independent Verification and Validation (IV&V) program used by NASA for manned space flight; contractors were hired to comb through the code and received a bonus for each error found. An example of the adversarial approach was the evaluation of nuclear command and control, where the Sandia National Laboratories and the NSA competed to find errors in each other's designs.
One way of combining the two is simply to hire several experts from different consulting firms or universities and to give the repeat business to whoever most conspicuously finds errors and improves the design. Another way is to have several different accreditation bodies: I mentioned how voting systems in the USA are checked independently in each state, and in those days, before standards were imposed by such organizations as VISA and SWIFT, banks built local payment networks, each of which had its own design, checked by its own auditors. However, neither approach is infallible; there are some really awful legacy voting systems and banking systems.

Free and Open Source Software

The free and open source software movement extends the philosophy of openness from the architecture to the details of the implementation. Many security products have publicly available source code; the first of these was probably the electronic mail encryption program PGP. The Linux operating system and the Apache web server are also open, and many people rely on them for the protection of information. There is also a drive to adopt open source in government.
Open source software is not a wholly recent invention; in the early days of computing most suppliers of system software published their source code. This openness began to disappear in the early 1980s, when the pressure of litigation led IBM to adopt an "object code only" policy for its mainframe software, despite bitter criticism from the user community. The pendulum has recently begun to swing back, and IBM is one of the pillars of open source.
There are a number of strong arguments in favor of open software and several against. Firstly, if everyone in the world can examine and play with the software, then bugs will probably be found and fixed; in Raymond's famous phrase, "Given enough eyeballs, all bugs are shallow". This is especially true if the software is maintained in a collaborative effort, like Linux and Apache. It may also be harder to insert backdoors into such a product.
The standard argument of the defense contractor against open source is that, when software becomes large and complex, there may be few or no able, motivated people studying it, and serious vulnerabilities may not be found for years. For example, a programming error in versions 5 and 6 of PGP allowed an attacker to add an extra key escrow without the knowledge of the key holder; this was around for years before it was noticed. There were also backdoor "service passwords" in products such as sendmail, which persisted for years before they were removed.
The worry is that there may be attackers who are motivated enough to spend more time looking for bugs or exploitable features in published code than the community of reviewers. Firstly, there may not be enough reviewers for many open products, since the typical volunteer finds developing code more rewarding than finding exploits. Much open source development is done by students, who find that it helps them get good jobs later if they can point to some component of an employer's systems that they helped develop; perhaps this would not be so useful if all they could point to was a collection of bug reports that forced security updates. Secondly, different testers find different bugs, since their testing focus differs, so it is quite possible that even after a product has withstood 10,000 hours of public examination, a foreign intelligence agency that has invested only 1000 hours may find a new vulnerability. Given the growth-of-reliability models quoted, the probabilities are easy enough to compute.
Other arguments include the observation that active open source projects add functionality and features at a dizzying rate compared with closed software, which may open up unpleasant feature interactions; that such projects may fail to reach consensus about what security is trying to achieve; and that there are special cases, such as the protection of smart cards against various attacks, where a proprietary encryption algorithm built into the hardware of the chip can force the attacker to spend significantly more effort on reverse engineering.
So where does the balance of advantage lie? Eric Raymond's influential analysis of the economics of open source software suggests that there are five criteria for whether a product will benefit from the open source approach: where it is based on common engineering knowledge rather than on proprietary techniques, where it is sensitive to failures, where it requires expert review for verification, where it is critical enough to the business that users will cooperate in finding and removing bugs, and where its economics involve strong network effects. Security passes all these tests.
Some people have argued that while openness helps defenders find bugs so that they can fix them, it will also help attackers find bugs so that they can exploit them. Will it help attackers or defenders more? In 2002 it was proved that, under the standard model of reliability growth, openness helps attack and defense equally. Thus whether the open or the proprietary approach will work better in a given application will depend on how far and in what way that application departs from the standard assumptions, for example about independent vulnerabilities. As an example, a study of the security bugs found in the OpenBSD operating system showed that these bugs were significantly correlated, which suggests that openness there was a good thing.
In fact there is a long history of security engineers in various disciplines converting to openness. The long-standing wisdom of Auguste Kerckhoffs was that cryptographic systems should be designed in such a way that they are not compromised if the adversary learns the technique used. The debate about whether locksmiths should discuss vulnerabilities in locks began in Victorian times. The scholar of law and economics Peter Swire explained why governments are by their nature less inclined to accept disclosure: although competitive forces make even Microsoft open much of its software for reasons of compatibility and trust, government agencies play other games (such as expanding their budgets and avoiding embarrassment). Nevertheless, the security arguments began to prevail in some circles: from cautious beginnings around 1999, the US Department of Defense began to accept open source, in particular through the SELinux project.
So, although open design is neither necessary nor sufficient, it will often be useful. The important questions are how much effort has been spent by able people on checking and testing what you have built — and whether they tell you everything they find.

Semi-Open Design

Where a fully open design is impossible, you can often still obtain the benefits by choosing a partially open one. For example, the architectural design may be published even if some implementation details are not. Examples we have seen include the bank smart card protocol, nuclear command and control systems and the SPDC mechanism in Blu-Ray.
Another approach to semi-open design is to use an open platform and build proprietary components on top of it. The best-known example here may be Apple's OS/X, which combines the OpenBSD operating system with proprietary multimedia components. In other applications a proprietary but widely used product such as Windows or Oracle may be "open enough". Suppose, for example, that you are worried about a legal attack. If there is a dispute in court about whether a system was secure at the time of the contested transaction, then instead of forcing opposing experts to comb through the code, you can rely on the history of disclosed vulnerabilities, patches and attacks. Thus we find that more and more ATMs use Windows as a platform (although the version they use has many unnecessary things cut out).

Penetrate-and-Patch, CERTs and Bugtraq

Penetrate-and-patch was the dismissive name given in the 1970s and 1980s to the evolutionary procedure of finding security bugs in systems and then fixing them; it was widely regarded at the time as inadequate, since more bugs were always being found. The hope at the time was that formal methods would allow error-free systems to be built. It is now well known that formal verification works only for systems that are too small and limited for most applications. Be that as it may, most software development is iterative, based on the build cycle.
Thus iterative approaches to assurance are necessary, and the question is how to manage them.
The interests of the various stakeholders in a system may diverge quite radically at this point.

1. The vendor would prefer bugs not to be found, in order to save the cost of fixing them.
2. The average customer may prefer the same; lazy customers often do not patch and get infected as a result. (Until their ISP disconnects them for sending spam, they may not notice or care.)
3. The typical security researcher wants a responsible way of disclosing his findings, so that he can give vendors a reasonable period of time to release a patch before he publishes his conference paper; therefore he usually sends a report to the local computer emergency response team (CERT), which in turn will notify the vendor and publish the vulnerability after 45 days.
4. Intelligence agencies want to learn about vulnerabilities quickly, so that they can be exploited before the patch is released. (Many CERTs are funded by agencies and have cleared staff.)
5. Some hackers disclose vulnerabilities in mailing lists such as bugtraq, which impose no delay; this can force software vendors to release emergency patches outside the normal cycle.
6. Security companies benefit from the existence of unpatched vulnerabilities in that they can use their firewalls to filter attacks using them, and the antivirus software on their customers' PCs can also often try to intercept such attacks. (Symantec hosts the bugtraq list.)
7. Large companies do not like emergency patches, and most government departments do not either, since the process of testing a new patch against the enterprise's critical systems and deploying it is expensive.

In the 1990s the debate was led by people who were frustrated that software vendors left their products unpatched for months or even years. This was one of the reasons the bugtraq list was created; this then led to debates about "responsible disclosure" with various proposals about how much breathing space a researcher should give a vendor. The CERT system with its 45-day delay arose out of this. It gives vendors a strong incentive to have an attentive bug-reporting facility; in return they get enough time to test the fix properly before releasing it, researchers get credits to put in their CVs, and users get bug fixes at the same time as bug reports, and large companies have regular updates or service packs for which their corporate customers can plan.
Is this system the best we could get? Recently the patch cycle has become a subject of study by security economists. At the Workshop on the Economics of Information Security in 2004 a debate unfolded between Eric Rescorla, who argued that, since bugs are numerous and uncorrelated, and since most exploits use vulnerabilities reverse-engineered from existing patches, there should be minimal disclosure, and Ashish Arora, who argued that, from both theoretical and empirical points of view, the threat of disclosure was necessary to force vendors to patch. There have been some innovations, in particular the introduction of essentially automatic updates for mass users and the creation of firms that make markets in vulnerabilities, and we have some further research data, in particular the fact that bugs in OpenBSD and some other systems are correlated. By and large, the current way of doing things seems reasonable and stable.

Education

Perhaps the problems and technologies of system protection ought to be far more widely understood. I have described case after case in which the wrong mechanisms were used, or the right mechanisms were used in the wrong way. It used to be the norm that protection got right only on the fifth or sixth attempt, whereas with a slightly better informed approach it could have been the second or third. Security specialists, unfortunately, tend to be either too specialized and focused on some tiny aspect of technology, or generalists who have never been exposed to many of the deeper technical issues. But blaming the problem on the education we currently give students — whether in computer science, business administration or law — is too easy; the difficult part is figuring out what to do about it. I hope my series of articles on security engineering will be useful.

Summary

Sometimes the most difficult part of a security engineering project is knowing when you are finished. A number of evaluation and assurance methodologies are available to help. In moderate quantities they can be very useful, especially for a startup firm whose development culture is still fluid and which strives to establish good working habits and build a reputation. But the help they can give has its limits, and excessive use of bureaucratic quality control tools can do serious harm. I think of them as salt; a few pinches on your french fries can be a good thing, but a few ounces definitely are not.
But although the picture is gloomy, this does not justify despondency. As people gradually acquire experience of what works, what is attacked and how, and as the requirements of protection and the mechanisms become more part of the skill set of the working engineer, things gradually improve. Security may be got right only on the fourth attempt, but that is better than never — which was typical fifteen years ago.

Life is complex. Success means coping with it. Complaining about it too much is the road to failure.