Support namespace-scopted BpfNsApplication CRD

Signed-off-by: Andre Fredette <[email protected]>

PROJECT

@@ -136,4 +136,11 @@ resources:
   kind: BpfApplicationState
   path: github.com/bpfman/bpfman-operator/apis/v1alpha1
   version: v1alpha1
+- api:
+    crdVersion: v1
+  controller: true
+  domain: bpfman.io
+  kind: BpfNsApplicationState
+  path: github.com/bpfman/bpfman-operator/apis/v1alpha1
+  version: v1alpha1
 version: "3"

TODO.md

@@ -1,121 +1,24 @@
 # Intro
 
-The code has support for XDP, TCX, and Fentry programs in a BpfApplication.
+The code has support for:
+- XDP, TCX, and Fentry programs in the cluster-scoped BpfApplication, and 
+- XDP and TCX in the namespace-scoped BpfNsApplication
 
 It's written so that Dave's load/attach split code should drop in pretty easily,
 but it's not using it yet.  I'm simulating the attachments by reloading the code
 for each attachment (like we do today).
 
-# Observation/Question
-Fentry and Fexit programs break the mold.
-
-Fentry and Fexit programs need to be loaded separately for each attach point, so
-the user must specify the BPF function name and attach point together for each
-attachment.  The user can then attach or detach that program later, but if the
-user wants to attach the same Fentry/Fexit program to a different attach point,
-the program must be loaded again with the new attach point.
-
-For other program types, the user can load a program, and then attach or detach
-the program to/from multiple attach points at any time after it has been loaded.
-
-Some differences that result from these requirements:
-- Each Fentry/Fexit attach point results in a unique bpf program ID (even if
-  they all use the same bpf function)
-- For other types, a given bpf program can have one bpf program ID (assigned
-  when it's loaded), plus multiple attach IDs (assigened when it is attached)
-- We don't need an attach ID for Fentry/Fexit programs.
-
-We need to do one of the following:
-- Not support the user modifying Fentry/Fexit attach points after the initial
-  BpfApplication load.
-- Load the program if they add an attach point (which would result in an
-  independent set of global data), and unload the program if they remove an
-  attachment.
-
-Yaml options:
-
-**Option 1:**  The current design uses a map indexed by the bpffunction name, so
-we can only list a given bpffunction name once followed by a list of attach
-points as shown below.  This represents Fentry/Fexit programs the same way as
-others, but they would need to behave differently as outlined above.
-
-```yaml
-programs:
-  tcx_stats:
-    type: TCX
-    tcx:
-      attach_points:
-        - interfaceselector:
-            primarynodeinterface: true
-          priority: 500
-          direction: ingress
-        - interfaceselector:
-            interfaces:
-              - eth1
-          priority: 100
-          direction: egress
-  test_fentry:
-    type: Fentry
-    fentry:
-      attach_points:
-        - function_name: do_unlinkat
-          attach: true
-        - function_name: tcp_connect
-          attach: false
-```
-
-**Options 2:**  Use a slice, and allow the same Fentry/Fexit functions to be
-included multiple times. The is more like the bpfman api, but potentially more
-cumbersome for Fentry/Fexit programs.
-
-```yaml
-  programs:
-    - type: TCX
-      tcx:
-        bpffunctionname: tcx_stats
-        attach_points:
-          - interfaceselector:
-              primarynodeinterface: true
-            priority: 500
-            direction: ingress
-          - interfaceselector:
-              interfaces:
-                - eth0
-            priority: 100
-            direction: egress
-            containers:
-              namespace: bpfman
-              pods:
-                matchLabels:
-                  name: bpfman-daemon
-              containernames:
-                - bpfman
-                - bpfman-agent
-    - type: Fentry
-      fentry:
-        bpffunctionname: tcx_stats
-        function_name: do_unlinkat
-        attach: true
-    - type: Fentry
-      fentry:
-        bpffunctionname: tcx_stats
-        function_name: tcp_connect
-        attach: true
-```
-
 # New Code
 
 The new code is mainly in these directories:
 
 ## Updated & working APIs:
 
-- apis/v1alpha1/fentryProgram_types.go
-- apis/v1alpha1/xdpProgram_types.go
-- apis/v1alpha1/tcxProgram_types.go
-- apis/v1alpha1/bpfApplication_types.go
 - apis/v1alpha1/bpfApplicationState_types.go
+  - Program Types: TCX, XDP, Fentry
+- apis/v1alpha1/bpfNsApplicationState_types.go
+  - Program Types: TCX, XDP
 
-Note: the rest are partially updated.
 
 ## New Agent:
 
@@ -133,23 +36,23 @@ we run the operator.
 
 - Unit tests for the agent and the operator
 - The following working samples:
-    - config/samples/bpfman.io_v1alpha1_bpfapplication.yaml (XDP & TCX)
-    - config/samples/bpfman.io_v1alpha1_fentry_bpfapplication.yaml (Fentry)
+    - config/samples/bpfman.io_v1alpha1_bpfapplication.yaml (XDP, TCX and Fentry)
+    - config/samples/bpfman.io_v1alpha1_fentry_bpfnsapplication.yaml (XDP and TCX)
 
 # TODO:
 
 (In no particular order.)
 
-- Implement Fentry/Fexit solution.
+- ~~Implement Fentry/Fexit solution.~~
 - Integrate with the new bpfman code with load/attach split (of course)
-- Create a bpf.o file with all the application types for both cluster and
-  namespace scoped BpfApplicaitons.
+- ~~Create a bpf.o file with all the application types for both cluster and
+  namespace scoped BpfApplicaitons.~~
 - Redo the status/condition values.  I’m currently using the existing framework
   and some values for status/conditions, but I intend to create a new set of
   conditions/status values that make more sense for the new design.
 - Review all comments and logs.
-- Maybe make more code common.
-- Support the rest of the program types (including namespace-scoped CRDs).
+- ~~Support namespace-scoped BPF Application CRD~~
+- Support the rest of the program types.
 - Delete old directories.
 - Lots more testing and code cleanup.
 - Lots of other stuff (I'm sure).

apis/v1alpha1/bpfNsApplicationState_types.go

@@ -0,0 +1,135 @@
+/*
+Copyright 2023 The bpfman Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	metav1types "k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+)
+
+// BpfNsApplicationProgramState defines the desired state of BpfNsApplication
+// +union
+// +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'XDP' ?  has(self.xdp) : !has(self.xdp)",message="xdp configuration is required when type is XDP, and forbidden otherwise"
+// // +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'TC' ?  has(self.tc) : !has(self.tc)",message="tc configuration is required when type is TC, and forbidden otherwise"
+// +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'TCX' ?  has(self.tcx) : !has(self.tcx)",message="tcx configuration is required when type is TCX, and forbidden otherwise"
+// // +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'Uprobe' ?  has(self.uprobe) : !has(self.uprobe)",message="uprobe configuration is required when type is Uprobe, and forbidden otherwise"
+// // +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'Uretprobe' ?  has(self.uretprobe) : !has(self.uretprobe)",message="uretprobe configuration is required when type is Uretprobe, and forbidden otherwise"
+type BpfNsApplicationProgramState struct {
+	BpfProgramStateCommon `json:",inline"`
+	// Type specifies the bpf program type
+	// +unionDiscriminator
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:Enum:="XDP";"TC";"TCX";"Fentry";"Fexit";"Kprobe";"Kretprobe";"Uprobe";"Uretprobe";"Tracepoint"
+	Type EBPFProgType `json:"type,omitempty"`
+
+	// xdp defines the desired state of the application's XdpPrograms.
+	// +unionMember
+	// +optional
+	XDP *XdpNsProgramInfoState `json:"xdp,omitempty"`
+
+	// // tc defines the desired state of the application's TcPrograms.
+	// // +unionMember
+	// // +optional
+	// TC *TcProgramInfoState `json:"tc,omitempty"`
+
+	// tcx defines the desired state of the application's TcxPrograms.
+	// +unionMember
+	// +optional
+	TCX *TcxNsProgramInfoState `json:"tcx,omitempty"`
+
+	// // uprobe defines the desired state of the application's UprobePrograms.
+	// // +unionMember
+	// // +optional
+	// Uprobe *UprobeProgramInfoState `json:"uprobe,omitempty"`
+
+	// // uretprobe defines the desired state of the application's UretprobePrograms.
+	// // +unionMember
+	// // +optional
+	// Uretprobe *UprobeProgramInfoState `json:"uretprobe,omitempty"`
+}
+
+// BpfNsApplicationSpec defines the desired state of BpfNsApplication
+type BpfNsApplicationStateSpec struct {
+	// Node is the name of the node for this BpfNsApplicationStateSpec.
+	Node string `json:"node"`
+	// The number of times the BpfNsApplicationState has been updated.  Set to 1
+	// when the object is created, then it is incremented prior to each update.
+	// This allows us to verify that the API server has the updated object prior
+	// to starting a new Reconcile operation.
+	UpdateCount int64 `json:"updatecount"`
+	// AppLoadStatus reflects the status of loading the bpf application on the
+	// given node.
+	AppLoadStatus BpfProgramConditionType `json:"apploadstatus"`
+	// Programs is a list of bpf programs contained in the parent application.
+	// It is a map from the bpf program name to BpfNsApplicationProgramState
+	// elements.
+	Programs []BpfNsApplicationProgramState `json:"programs,omitempty"`
+}
+
+// +genclient
+// +kubebuilder:object:root=true
+// +kubebuilder:subresource:status
+// +kubebuilder:resource:scope=Namespaced
+
+// BpfNsApplicationState contains the per-node state of a BpfNsApplication.
+// +kubebuilder:printcolumn:name="Node",type=string,JSONPath=".spec.node"
+// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.conditions[0].reason`
+// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
+type BpfNsApplicationState struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   BpfNsApplicationStateSpec `json:"spec,omitempty"`
+	Status BpfAppStatus              `json:"status,omitempty"`
+}
+
+// +kubebuilder:object:root=true
+// BpfNsApplicationStateList contains a list of BpfNsApplicationState objects
+type BpfNsApplicationStateList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []BpfNsApplicationState `json:"items"`
+}
+
+func (an BpfNsApplicationState) GetName() string {
+	return an.Name
+}
+
+func (an BpfNsApplicationState) GetUID() metav1types.UID {
+	return an.UID
+}
+
+func (an BpfNsApplicationState) GetAnnotations() map[string]string {
+	return an.Annotations
+}
+
+func (an BpfNsApplicationState) GetLabels() map[string]string {
+	return an.Labels
+}
+
+func (an BpfNsApplicationState) GetStatus() *BpfAppStatus {
+	return &an.Status
+}
+
+func (an BpfNsApplicationState) GetClientObject() client.Object {
+	return &an
+}
+
+func (anl BpfNsApplicationStateList) GetItems() []BpfNsApplicationState {
+	return anl.Items
+}

apis/v1alpha1/bpfNsApplication_types.go

@@ -28,6 +28,7 @@ import (
 // +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'Uprobe' ?  has(self.uprobe) : !has(self.uprobe)",message="uprobe configuration is required when type is Uprobe, and forbidden otherwise"
 // +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'Uretprobe' ?  has(self.uretprobe) : !has(self.uretprobe)",message="uretprobe configuration is required when type is Uretprobe, and forbidden otherwise"
 type BpfNsApplicationProgram struct {
+	BpfProgramCommon `json:",inline"`
 	// Type specifies the bpf program type
 	// +unionDiscriminator
 	// +kubebuilder:validation:Required

apis/v1alpha1/tcxNsProgram_types.go

@@ -88,3 +88,35 @@ type TcxNsProgramList struct {
 	metav1.ListMeta `json:"metadata,omitempty"`
 	Items           []TcxNsProgram `json:"items"`
 }
+
+type TcxNsProgramInfoState struct {
+	// The list of points to which the program should be attached.
+	// TcxAttachInfoState is similar to TcxAttachInfo, but the interface and
+	// container selectors are expanded, and we have one instance of
+	// TcxAttachInfoState for each unique attach point. The list is optional and
+	// may be udated after the bpf program has been loaded.
+	// +optional
+	AttachPoints []TcxAttachInfoState `json:"attach_points"`
+}
+
+type TcxNsAttachInfoState struct {
+	AttachInfoCommon `json:",inline"`
+
+	// Interface name to attach the tcx program to.
+	IfName string `json:"ifname"`
+
+	// Optional container pid to attach the tcx program in.
+	ContainerPid uint32 `json:"containerpid"`
+
+	// Direction specifies the direction of traffic the tcx program should
+	// attach to for a given network device.
+	// +kubebuilder:validation:Enum=ingress;egress
+	Direction string `json:"direction"`
+
+	// Priority specifies the priority of the tcx program in relation to
+	// other programs of the same type with the same attach point. It is a value
+	// from 0 to 1000 where lower values have higher precedence.
+	// +kubebuilder:validation:Minimum=0
+	// +kubebuilder:validation:Maximum=1000
+	Priority int32 `json:"priority"`
+}

apis/v1alpha1/xdpNsProgram_types.go

@@ -88,3 +88,35 @@ type XdpNsProgramList struct {
 	metav1.ListMeta `json:"metadata,omitempty"`
 	Items           []XdpNsProgram `json:"items"`
 }
+
+type XdpNsProgramInfoState struct {
+	// The list of points to which the program should be attached.
+	// XdpAttachInfoState is similar to XdpAttachInfo, but the interface and
+	// container selectors are expanded, and we have one instance of
+	// XdpAttachInfoState for each unique attach point. The list is optional and
+	// may be udated after the bpf program has been loaded.
+	// +optional
+	AttachPoints []XdpAttachInfoState `json:"attach_points"`
+}
+
+type XdpNsAttachInfoState struct {
+	AttachInfoCommon `json:",inline"`
+
+	// Interface name to attach the xdp program to.
+	IfName string `json:"ifname"`
+
+	// Optional container pid to attach the xdp program in.
+	ContainerPid uint32 `json:"containerpid"`
+
+	// Priority specifies the priority of the xdp program in relation to
+	// other programs of the same type with the same attach point. It is a value
+	// from 0 to 1000 where lower values have higher precedence.
+	// +kubebuilder:validation:Minimum=0
+	// +kubebuilder:validation:Maximum=1000
+	Priority int32 `json:"priority"`
+
+	// ProceedOn allows the user to call other xdp programs in chain on this exit code.
+	// Multiple values are supported by repeating the parameter.
+	// +kubebuilder:validation:MaxItems=6
+	ProceedOn []XdpProceedOnValue `json:"proceedon"`
+}