Protocols

In the previous section, we discussed who we are (psychology) and what we protect (policy). This section answers the question of how parties interact with each other securely. We define a security protocol as a set of rules governing communication between participants (principals) in the presence of an adversary.
The key idea is that system security is often broken not because of weaknesses in cryptography (encryption algorithms), but because of flaws in the design of the interaction protocol itself.

1. What Is a Security Protocol?

It is not merely the transmission of encrypted data. It is a message exchange scenario that must guarantee:

• Authentication — confidence in who you are communicating with.
• Integrity — confidence that the message has not been altered.
• Freshness — confidence that the message is not a replay of an old one (replay attack).
• Non-repudiation — the inability of the sender to deny having sent the message.

It should be emphasized that protocol design is “programming Satan’s computer.” The adversary will attempt to break the logic of the exchange by forging, intercepting, or modifying messages.

2. Simple Authentication and Challenge-Response

The most basic mechanism is password verification. However, transmitting a password in plaintext is dangerous (sniffing).
The solution is Challenge-Response.

• The server sends a random number (nonce).
• The client encrypts this number with its secret key (password) and sends it back.
• The server verifies the result.

Advantage: the password is never transmitted over the network.
Disadvantage: vulnerability to Man-in-the-Middle (MITM) attacks if mutual trust is implemented incorrectly.

3. Classic Attacks on Protocols

Let us examine in detail the sophisticated ways seemingly reliable schemes can be compromised.

Man-in-the-Middle Attack

The attacker positions themselves between Alice and Bob. They pretend to be Bob to Alice and Alice to Bob, relaying and modifying messages.
Example: the story of South African pilots during the war in Angola. Cuban MiGs intercepted Identification Friend or Foe (IFF) signals from SAAF aircraft, forwarded them to their air defense systems, obtained the correct response, and relayed it back, successfully penetrating the airspace.

Reflection Attack

This occurs during mutual authentication. An attacker initiates a session with the victim, receives a challenge, and instead of computing the response themselves, opens a second session with the same victim, forwarding the victim’s own challenge back to them. The victim answers their own challenge, and the attacker uses this response to complete the first session.
Defense: use different keys for different communication directions or include the sender’s identity in the encrypted message.

Replay Attacks

Recording a legitimate message and retransmitting it later.
Defense: timestamps or one-time values (nonces / counters). However, timestamps require clock synchronization, which is itself a potential vulnerability.

4. Key Management

How can keys be distributed securely among participants?

• Needham-Schroeder Protocol: a classic protocol that uses a trusted third party (TTP).
  Vulnerability: discovered 17 years after publication. The protocol allowed old session keys to be reused if a long-term key had been compromised in the past. This demonstrates how difficult protocol verification can be.

• Kerberos: a practical implementation of the Needham-Schroeder ideas, widely used in Windows and Unix networks. It uses tickets and timestamps to prevent replay attacks.

5. Formal Verification

Can we mathematically prove that a protocol is secure?

• BAN Logic (Burrows-Abadi-Needham): a formal method for analyzing the beliefs of protocol participants (“Alice believes that key K is fresh and belongs to Bob”).
• Limitations: the logic works only if the underlying assumptions are correct. It can prove the correctness of an abstract model, but it cannot account for implementation flaws or physical attacks. The Needham-Schroeder example showed that even “proven” protocols may contain hidden flaws.

Formal verification is a powerful filter for eliminating logical errors, but it does not replace careful engineering analysis of the implementation and its usage context.

6. Attacks on Composite Protocols (Chosen Protocol Attacks)

One of the most insidious problems in modern systems is the interaction of multiple protocols. A protocol that is secure in isolation may become vulnerable if its keys or messages are used in another context.

Mafia-in-the-Middle Attack

This is an extension of the MITM concept, but focused on the reuse of legitimate user actions for fraud.
Scenario: an attacker creates a fake website (a pornography site or a free service) that requires the user to verify their identity using the same smart card or token employed for banking operations.

Mechanism:

1. The victim visits the attacker’s website.
2. The website initiates an authentication protocol by forwarding a request to the bank on behalf of the victim.
3. The bank sends a challenge.
4. The website forwards the challenge to the victim.
5. The victim enters their PIN into their token / smart card and obtains a response.
6. The response is forwarded back to the bank.
7. The bank believes the victim has authorized a transaction (for example, a money transfer), while the victim believes they are simply logging into a website.

Cause of the vulnerability: lack of context binding. The protocol does not distinguish between “website login” and “payment authorization” if both use the same cryptographic operation with the same keys.
Defense: explicitly include the operation type, amount, recipient, and other contextual information in the signed message. Keys used for different purposes should be separated.

7. Problems in Real Implementations: EMV and Other Standards

Theory often diverges from practice due to backward compatibility requirements, standard complexity, and human factors.

EMV Vulnerabilities (Chip and PIN)

• No-PIN Attack: a compromised terminal may choose not to request a PIN from the card while reporting to the bank that the PIN was entered correctly. If the bank trusts the terminal more than the card, the transaction succeeds.
• Trust Interface Problem: the user enters a PIN into a terminal but has no way to verify that the terminal is actually secure. The card trusts the terminal, but the terminal itself may be malicious.
• Relay Attack: even if the PIN is protected, signals can be relayed between a card in the victim’s pocket and a terminal in a store, causing the card to believe it is physically near the terminal.

The Problem of Reversibility and Compatibility

Many systems maintain support for older, less secure operating modes (fallback modes) to remain compatible with legacy equipment. Attackers often target these modes by forcing the system into an insecure state. An example is forcing the use of a magnetic stripe instead of a chip in EMV systems when the terminal supposedly does not support chips.

8. Fundamental Principles of Secure Protocol Design

Based on the analysis of numerous failures, several principles emerge whose violation almost inevitably leads to vulnerabilities:

1. Explicitness. All significant parameters (participant identities, message type, nonce, timestamps, context) must be explicitly specified in the message. Do not rely on implicit context or field position within a packet.
2. Freshness. Always use reliable mechanisms to prevent replay attacks (nonces, timestamps with protection against desynchronization, counters). Ensure that freshness values cannot be predicted or reused.
3. Key Separation. Never use the same key for different purposes (encryption, authentication, signing) or across different protocols without explicit separation (for example, by adding a prefix to the data before encryption).
4. Least Privilege in Protocols. A protocol should disclose only the minimum information necessary to accomplish its task. Avoid transmitting excessive data that may be used for traffic analysis or future attacks.
5. Protection Against an Active Adversary. Design protocols under the assumption that an adversary can create, modify, delete, and delay any message on the network. Do not assume that the communication channel is reliable or that participants behave honestly.
6. Simplicity. The more complex a protocol, the greater the likelihood of errors. Avoid unnecessary complexity and excessive numbers of options. Every additional branch of logic is a potential vulnerability.

9. Evolution of Threats and Protocol Adaptation

Security protocols are part of an arms race.

• Computational Power. Algorithms that were considered secure 20 years ago (DES, RSA with short keys) can now be broken with relative ease. Protocols must support the flexible replacement and upgrading of cryptographic primitives.
• New Attack Vectors. The emergence of quantum computers threatens asymmetric cryptography (RSA, ECC). Future protocols must take post-quantum cryptography into account.
• Changing Threat Models. Protocols designed for closed corporate networks often prove insecure on the open Internet, where any node may be hostile.

Key Takeaway

Security cannot be achieved through strong encryption alone. If the message exchange logic contains flaws, an attacker can bypass cryptography by manipulating the flow of data. Engineers must think like adversaries, anticipating every possible way a dialogue between systems can be distorted.

A flaw in protocol design is catastrophic: it cannot be fixed with an application-level patch—the protocol itself must be replaced. Context is everything: the same protocol may be secure in one environment and completely vulnerable in another. Even a perfectly designed protocol can be compromised by implementation errors, poor key management, or social engineering.

Access Control

If we previously discussed who we are (psychology) and how we communicate (protocols), we now answer the question of what a specific user or process is allowed to do with system resources.

It should be emphasized that access control is the traditional center of gravity in computer security, where theory meets practice. However, the higher the level of abstraction (from hardware to applications), the more difficult access management becomes and the greater the likelihood of mistakes.

1. Levels of Access Control

Let us describe the hierarchy of protection mechanisms:

• Hardware Level: memory protection, address space segmentation (the processor prevents one process from reading another process’s memory).
• Operating System Level: management of files, ports, and devices (user alice can read file.txt but cannot write to /etc/passwd).
• Middleware Level: databases (who may execute queries against tables), message queue systems.
• Application Level: business logic (a manager may approve a payment up to $10,000, but anything above that requires a director’s signature).

The main problem: most real-world fraud occurs at the application level, where employees abuse trusted functions or discover loopholes in business logic that programmers have not protected as carefully as the operating system kernel.

2. Access Control Models

Access Matrix and Its Implementation

In theory, permissions can be represented as a matrix: rows are subjects (users), columns are objects (files), and cells contain permissions (read, write, execute). For large systems, this matrix becomes too large and sparse. Therefore, two primary compression methods are used.

Access Control Lists (ACLs). Rights are stored “by columns” (attached to the object). Example: file secret.doc has the list {Alice: rw, Bob: r}.

Advantages: easy to determine who has access to a specific file; easy to revoke access for everyone at once by deleting the file.
Disadvantages: difficult to determine which files a particular user can access; delegation is cumbersome.
Implementation: Unix (simplified owner/group/others model), Windows (full ACLs), NFS.

Capabilities (capabilities / tickets). Rights are stored “by rows” (attached to the subject). Example: user Alice possesses a “ticket” granting read access to file.txt. If she passes this ticket to Bob, Bob can read the file as well.

Advantages: easy delegation; high-performance authorization checks (no need to scan lists).
Disadvantages: difficult to revoke permissions (all issued tickets must be found); difficult to determine who else has access to a file.
Implementation: Kerberos (partially), certain microkernels, modern web application tokens (JWT).

Roles and Groups (RBAC – Role-Based Access Control)

To avoid managing permissions for each individual separately, groups and roles are used.

• Group: simply a list of users (for example, “Accountants”).
• Role: a set of functions a person may perform at a particular moment (for example, “On-Duty Physician”). One person may have multiple roles, but usually only one is active at a time.

It should be noted that RBAC has become the de facto standard in enterprise systems because it mirrors the organizational structure of a business.

3. Implementation in Popular Operating Systems

Unix / Linux

• Historically simple model: owner / group / others (rwx).
• Problem: overly coarse granularity.
• setuid mechanism: a program runs with the permissions of the file owner (often root) rather than the user who launched it. This is a powerful but dangerous mechanism and the source of many vulnerabilities (if a setuid program contains a bug, an attacker may gain root privileges).
• Modern extensions: SELinux, AppArmor (Mandatory Access Control, MAC), which allow the definition of complex policies (“the browser may write only to /tmp but may not read /home”).

Windows

• Complex ACLs with permission inheritance.
• The concepts of domains and trust relationships between them.
• UAC (User Account Control): introduced in Vista and Windows 7. Even administrators operate in a standard mode, while critical actions require confirmation (privilege elevation). This was an attempt to move away from the “always work as root” model that had been common in XP.

4. What Goes Wrong? Vulnerabilities

Let us spend some time examining how these systems fail.

Stack Overflow (Smashing the Stack)

A classic attack: an attacker overflows an input buffer with data containing executable code and overwrites the return address on the stack, forcing the process to execute malicious code with the program’s privileges.

Evolution of Defenses:

• StackGuard: a special canary value is placed before the return address. If a buffer overflow occurs, the canary changes and the program terminates immediately.
• NX-bit (No-Execute): hardware protection that marks memory regions (stack and heap) as non-executable.
• ASLR (Address Space Layout Randomization): random placement of memory regions makes it more difficult to predict addresses for code injection.

Race Conditions (TOCTTOU)

Time-of-Check to Time-of-Use (TOCTTOU) attacks exploit the time gap between verifying access permissions and actually using a resource.

Mechanism: an attacker creates a symbolic link (symlink) to a target file immediately after a program checks permissions on the original file but before it opens the file for writing.

Classic /tmp Example: if a privileged root program creates a temporary file with a predictable name (for example, /tmp/pid123.tmp) without using secure system calls (such as mkstemp with the O_EXCL flag), an attacker can pre-create a symlink with that name pointing to /etc/passwd. When the privileged program writes data, it overwrites the password file.

Defense: atomic kernel operations, random file names, opening files by file descriptors rather than names, and minimizing the time between checking and using a resource.

Human Factors and Interfaces

Users often grant programs excessive privileges (“Run as Administrator”) without understanding the consequences. Developers frequently avoid implementing fine-grained access control and instead grant full permissions “just in case.”

5. Sandboxing and Virtualization

How can we limit the damage caused by untrusted code?

• Sandboxes: isolated environments where code can execute, but its capabilities are strictly limited (no access to the network, disk, etc.). Examples include Java Applets and browser plugins.
• Virtualization: running entire operating systems inside another operating system. This allows applications to be isolated at the kernel level. If malware infects the guest OS, the host system remains clean.
• Trusted Computing: an initiative by Microsoft and Intel (TPM chips). An attempt to create a hardware-protected environment for executing critical code (DRM, key protection), isolated even from the operating system itself. One may view this skeptically, pointing to usability issues and questions about the actual level of trust.

6. Mandatory Access Control (MAC) and SELinux

While discretionary access control (DAC), used in standard Unix and Windows systems, relies on the file owner, mandatory access control (MAC) is enforced by the system based on a global policy that users cannot modify.

Type Enforcement and SELinux

One of the most powerful implementations of MAC is Type Enforcement, which forms the foundation of SELinux (Security-Enhanced Linux), developed by the NSA and integrated into Linux.

• Principle: each subject (process) and object (file, port) is assigned a type (security context). The policy defines a permission matrix specifying which process types may access which object types and in what manner.
• Advantage over DAC: even if an attacker gains root privileges (UID 0), they remain constrained by the SELinux policy. An Apache process, even when running as root, cannot read files from a user’s home directory or modify system configuration files unless explicitly permitted.
• Domain and Type Enforcement (DTE): an extension that allows processes to be grouped into domains and defines transition rules between them. This makes it possible to create sandboxes and secure data-processing pipelines.

The Complexity Problem

The main disadvantage of MAC (especially SELinux) is configuration complexity. An incorrect policy can block legitimate application behavior. As a result, many distributions run SELinux in permissive mode by default (logging violations only), reducing the actual level of protection.

7. The Trusted Interface Problem

Even the most reliable access control system is useless if the user cannot trust what they see on the screen.

The essence of the problem: the user enters a password or confirms a transaction, believing they are interacting with a legitimate system (a banking application or an operating system login dialog). However, malware (a Trojan) may forge that interface (window spoofing, UI redressing).

• Example with ATMs and smart cards: an attacker installs a keypad overlay or replaces the display. The user sees a PIN request from the “bank,” but is actually entering the PIN into the attacker’s device.
• Example on PCs (trusted path): in Windows, the Ctrl+Alt+Del key combination is implemented at the kernel level so that no application can intercept it and spoof the login screen. However, providing such a trusted path in complex graphical environments and web applications is extremely difficult.
• Consequences: without a trusted interface, mechanisms such as two-factor authentication or digital signatures can be compromised—the user signs or confirms something different from what they believe they are authorizing.

8. Economics and Incentives in Access Control

Once again, it is important to note that technical measures often fail because of poor economic incentives.

• Risk Dumping. Software vendors frequently design access control systems in a way that shifts responsibility onto the user. For example, applications request administrator privileges for nearly every action in order to simplify development. If a compromise occurs, the vendor can claim: “The user granted the permissions.”
• UAC in Windows. Although User Account Control attempts to restrict privileges by default, users have become accustomed to clicking “Allow” on every prompt, reducing the effectiveness of the mechanism to nearly zero.
• Conflict Between Convenience and Security. Systems with strict access controls (SELinux, rigid corporate policies) are often perceived as obstacles to productivity. This leads to the emergence of “shadow IT” solutions, policy circumvention, and the disabling of security mechanisms.
• Responsibility for Mistakes. Unlike physical security (where a lock manufacturer faces reputational risks if a lock is easily bypassed), access control flaws in software are often treated as “features” until a major incident occurs. The absence of legal liability for insecure code reduces the incentive to build truly secure systems.

9. Virtualization as a New Perimeter

With the rise of virtualization (VMware, Xen, Hyper-V), the concept of access control has expanded.

• Isolation Instead of Permissions: rather than carefully configuring file permissions within a single operating system, different trust levels are separated into different virtual machines.
• The Hypervisor Problem: the hypervisor becomes a new foundation of the Trusted Computing Base (TCB). If an attacker can perform a VM escape and gain control of the hypervisor, they gain access to all isolated systems.
• Covert Channels: virtualization creates new opportunities for hidden information transfer between isolated VMs through shared resources (CPU cache, memory usage), requiring new methods of control.

Key Takeaway

Access control is not merely a matter of configuring ACLs or assigning user permissions. It is a constant struggle among:

• the theoretical model (which may be mathematically rigorous, such as Bell-LaPadula or Biba);
• the implementation (vulnerable to race conditions, buffer overflows, and logical errors);
• the human factor (users bypass restrictions for convenience);
• economics (vendors do not want to bear the costs of security).

A reliable access control system requires a combination of:

• minimizing privileges (principle of least privilege);
• using mandatory mechanisms (MAC / SELinux) for critical components;
• protection against classes of memory vulnerabilities (StackGuard, ASLR);
• providing a trusted input/output path;
• proper incentives for developers and users.