Kubernetes Policy Comparison: OPA/Gatekeeper vs Kyverno

(Last Updated June 2023)

With the upcoming deprecation and subsequent removal of Pod Security Policies (PSPs) in Kubernetes, the time is near to find suitable alternatives. Those alternatives, it seems clear at present anyway, will need to be sourced from outside the Kubernetes project itself as there will be no replacement provided (edit: Pod Security Admission is the new, albeit incomplete, successor). The two leading CNCF projects which are prime candidates for PSP replacement are Open Policy Agent (OPA) via Gatekeeper and Kyverno, each with their own strengths and weaknesses. But it is confusing and daunting to compare and contrast them given that no comparisons presently exist. Not only are these projects viable replacements for PSPs, they are vastly more capable than simply acting on Pods alone--they are full Kubernetes policy engines. The goal of this article is to provide a neutral and impartial comparison between Gatekeeper and Kyverno so users may make the most informed selection decision for themselves. It will be an evaluation of each project's merits assuming only the Kubernetes angle.

Because this article is being written with Kubernetes in mind, the OPA/Gatekeeper project will simply be referred to as Gatekeeper henceforth.

This comparison article is organized into two main sections bookended by an introduction and closing. The first is an objective comparison, in tabular format, of Gatekeeper and Kyverno focusing on three main categories of attributes and capabilities reflecting, as best as possible, only the facts. The second is my subjective analysis given the earlier comparison and will provide personal opinion and commentary as to use case and recommendations of the ideal user.

I'd like to openly state my own personal bias in the interest of transparency. I am a Kyverno contributor and not a Gatekeeper one. I have authored a few blogs on Kyverno at this point and none on Gatekeeper. I have also been somewhat critical of OPA/Rego in the past. However, my goal here is to put all that plus any personal feelings aside and attempt to approach both projects fresh, with no bias and no preference for one over another.

This article has been written by equally engaging both the Kyverno and Open Policy Agent communities, giving the maintainers and contributors a fair and equal opportunity to comment on the comparison criteria as well as the final results of this article. No preference has been given to either project in terms of engagement or consideration either in regards to the time to comment or the methods in which to engage.

Introduction

What is a Kubernetes Policy

While Pod Security Policy in Kubernetes is a set of mechanisms for ensuring validating controls over Pods and their attributes, as the name would imply, it only operates on Pods and nothing else. Further, PSP can only block their creation; it does not perform any remediation thereof. Contrast that with policy engines such as Gatekeeper and Kyverno, and the capabilities are far more broad (i.e., applicable to more than just Pods) and deep (i.e., more than just simple validation). Having a policy engine for Kubernetes can be thought of as a way to more holistically control the Kubernetes environment and not just a single domain.

Gatekeeper Introduction

Gatekeeper is an open-source project and collaboration between a number of companies including Google and Microsoft, later donated to the CNCF. Now in its third iteration, Gatekeeper is the Kubernetes-specific implementation of Open Policy Agent (OPA), a general-purpose policy engine. Because of the relationship between Open Policy Agent with Gatekeeper, the project is often written "OPA/Gatekeeper" to acknowledge these ties. Gatekeeper enables request validation and, most recently, mutation. A major defining characteristic of Open Policy Agent is its reliance upon the use of a specialized programming language called Rego in order to implement the logic necessary for policy decisions. Through Rego, OPA is designed to be broadly applicable to disparate software and not Kubernetes exclusively, as well as achieve a high level of logic execution.

Kyverno Introduction

Kyverno is an open-source project originally from Nirmata and later donated to the CNCF where it is currently an Incubating project. Like Gatekeeper, Kyverno is a Kubernetes policy engine with validation and mutation abilities, however it adds other capabilities like image verification, resource generation, and cleanup capabilities. Unlike Gatekeeper, Kyverno was written from the start solely for use in Kubernetes. A major defining characteristic of Kyverno is its generation ability and the architecture which does not require specialized knowledge of a programming language to render policy decisions. Primarily as a reaction to this requirement of a specialized language, Kyverno differentiates itself from Gatekeeper through its model of simplicity.

Comparison

The following are a set of three tables which attempt to categorize and enumerate the features and qualities of each project in the most objective manner possible. The categories are features/capabilities which describe technical attributes; community/ecosystem which describe adoption and organizational attributes; and meta/misc which describe cognitive or miscellaneous attributes.

ᵃ Beta status

ᵇ Not available with all CRDs

ᶜ Separate CLI

ᵈ ConfigMaps only

* Not well defined. Gatekeeper clearly has more adoption from partners than Kyverno, but total penetration unknown.

** No objective measurement exists. Seems obvious that Gatekeeper has more traction from the community likely given its age.

* Not totally objective with direct comparison being difficult. Assessment made based on Gatekeeper project/functionality and not maturity level of Rego enablement materials/literature.

Analysis

Based upon the earlier table of feature comparisons between Gatekeeper and Kyverno, I have put together a concise list of the main advantages and disadvantages of each project. This is not an exhaustive list, only the large topics.

Gatekeeper Advantages

• Capable of expressing very complex policy
• More establishment in the community

Gatekeeper Disadvantages

• Requiring a programming language is a serious learning curve which generates large amounts of technical debt and increases time to value
• Mutation ability is in nascent stage
• No generation ability means its usefulness is limited to predominantly validation use cases
• Policy is complex, often lengthy, and requires multiple documents to implement

Kyverno Advantages

• Dramatic simplicity of policy expression allowing Kubernetes-style composition
• Solid mutation abilities
• Built-in image verification reduces external complexities
• Generation, syncing, cleanup, and other abilities are unique and allow new use cases
• Quick time to value and high degree of flexibility

Kyverno Disadvantages

• Highly complex policy may not be possible since no programming language is exposed, or may require external processing
• Young, still largely unproven due to lower establishment

Warning: The following is my opinionated analysis synthesized of the previous comparison table and advantages/disadvantages list in addition to my own experiences with both tools and time spent in the cloud-native community. If you are not interested in reading my opinion, this is the end of the article for you.

Kubernetes is a declarative system in which the user's desires are stated and Kubernetes, through its various controllers, goes to work reconciling the observed state in order to bring it into alignment with the desired state. This is the core value proposition of a cloud-native platform. In order to achieve this, the onus of the logic is shifted away from the user and into the platform itself. For each resource type, there exists some internal logic which houses the intelligence needed to remediate its state. With Gatekeeper, by far the largest weakness is its requirement of a purpose-built programming language called Rego, which aside from OPA, can be used nowhere else, in order to implement this logic. This is a reality because OPA is a general purpose policy engine. Only through Gatekeeper is its ability harnessed by adapting it into a Kubernetes form factor. Effectively then, the user is responsible for describing the object (policy) they wish reconciled as well as providing the logic necessary (Rego) to reconcile it. Using an external DSL to manage Kubernetes policies can become cumbersome and complex in a number of respects and add technical debt to projects. As a tradeoff, the distinct advantage is that it enables very powerful policies. After all, when one is required to write a programming language, one is only limited by the capabilities of that language and its inputs. This cost can be reduced if OPA can be leveraged elsewhere, however.

Kyverno is seen as a direct reaction to the complex technical demands required by Gatekeeper. Because it was built specifically for Kubernetes and expresses policy declaratively, its mental model is identical to the way Kubernetes objects are described and reconciled. The logic needed to carry out the policy decision is removed from the burden of the user and becomes the domain of the tool itself. This model results in an overwhelming simplification of how policy is written, which in turn makes policy easier in a number of respects. By having mutation and generation abilities, Kyverno is transfigured away from a simple admission controller and into the form of a true automation tool. By combining these many abilities, Kyverno is able to perform tasks that Gatekeeper simply cannot, but in addition has the power to eliminate other and disparate tools that might be employed in scattershot fashion throughout the cluster and/or organization. This simplicity coupled with its automation abilities and consolidation of other tools delivers tremendous value to new as well as experienced users and operators.

Based upon the information presented, it is my view that Kyverno should be a more natural choice for application of Kubernetes policy unless a user fits into either or both of two broad technical use cases, and then the selection of Gatekeeper is warranted.

1. There is a desire and specific intention to use a consistent core tool to apply policy to disparate systems (i.e., not just Kubernetes) within the organization.

Counter: In my experience, both within the cloud-native community and outside, most organizations are already using other tools today to apply policy to existing systems. This is typically because those systems, as well as the software implementing policy for said systems, predates Kubernetes and both OPA and Gatekeeper. Additionally, those existing tools commonly do not demand the use of a programming language to implement their policy. It is therefore seen as highly unlikely that most organizations will opt to discard those tools given the existing intellectual, operational, and capital investment in favor of a "new" tool which carries a high technical burden in order to realize the value brought by tooling consistency.

TL;DR: If you're looking for a single policy engine to use across Kubernetes and other systems, Kyverno is not for you.

2. The types of policies required are highly complex in nature.

Counter: In my experience, most Kubernetes organizations/users today are using no policy whatsoever, including PSPs. And an AWS 2020 survey of customers running containerized workloads on AWS found similar results with only 49% using policy. Of the ones who are, the vast majority are expressing the same things repeatedly. Examples of this repetition, in plain language, include desires such as "containers should not be privileged" or "ensure all Namespaces carry a given label" or "validate Pods are not using hostPath volumes". While the term "complex" is relative and somewhat subjective, such policy expressions are decidedly not complex. Kyverno allows policy to be written in the simplest form possible which in turn is easier to reason about and maintain. Paying an additional price for a tool which is more complex and difficult yields no benefit if it isn't being used to its fullest potential.

TL;DR: If you are not expressing highly complex policy, Gatekeeper gives you no advantage here.

Closing

Both the Gatekeeper and Kyverno projects are valuable and capable policy engines in their own right, each with strengths and weaknesses. Ultimately, users should evaluate and make the most informed decision given their use cases and constraints, but as a general recommendation, all production users should plan on using a policy engine to secure their clusters and simplify Kubernetes management.