Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Server Poc #231

Draft
wants to merge 50 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
9f5baef
Refactor of the Object interface to be a superset of the kubernetes i…
IfSentient Feb 9, 2024
76ee059
Implement New resource.Kind and resource.Object in Other Packages (#207)
IfSentient Feb 14, 2024
0c382e4
[Kind/Object Refactor] Codegen Updates (#211)
IfSentient Feb 21, 2024
b74f174
Update tutorial based on changes to Object interface and codegen. (#214)
IfSentient Feb 22, 2024
f574161
[Kind/Object Refactor] Misc Final Fixes (#216)
IfSentient Feb 23, 2024
653c4c9
[bugfix] thema codec: set the APIVersion and Kind correctly.
IfSentient Feb 23, 2024
8668730
Merge branch 'main' into kind-object-refactor
IfSentient Feb 23, 2024
3c3be6e
Merge branch 'main' into kind-object-refactor
radiohead Feb 27, 2024
3c7814a
add wip poc
toddtreece Mar 1, 2024
5d0cb2a
Some POC work for an apiserver package, based on work in the simple p…
IfSentient Mar 6, 2024
49a5553
Initial working POC of k8s openAPI codegen using kube-openapi in a Je…
IfSentient Mar 7, 2024
16d9766
Bump the all group with 1 update (#222)
dependabot[bot] Feb 29, 2024
4ca88b3
Fix calls to local scripts (#225)
mem Mar 4, 2024
c7f326a
Fix generated files' permissions (#224)
mem Mar 5, 2024
96cdf4b
simple version is almost working
toddtreece Mar 13, 2024
bcd0d30
Initial working POC of k8s openAPI codegen using kube-openapi in a Je…
IfSentient Mar 7, 2024
557c64e
working apiserver codegen
toddtreece Mar 14, 2024
c9e00bb
add example
toddtreece Mar 14, 2024
af7b038
add basic apiserver command
toddtreece Mar 14, 2024
07f5104
Update codegen to allow for prefixing of all go types with the kind n…
IfSentient Mar 14, 2024
39cae77
Minor clean-up of storage to dynamically decide on GetAttrs func for …
IfSentient Mar 14, 2024
8baeb00
Some more jenny updates to handle mutliple kinds in the same package,…
IfSentient Mar 14, 2024
062727b
Initial work on subresource routes.
IfSentient Mar 15, 2024
edade39
Small example updates.
IfSentient Mar 15, 2024
5ec9bee
Some code re-organization.
IfSentient Mar 15, 2024
8af0aab
In-progress apiserver.ResourceGroup work.
IfSentient Mar 18, 2024
af93241
add storage for subresources
toddtreece Mar 19, 2024
34edb3e
update codegen
toddtreece Mar 19, 2024
9f7148e
Added a second version to the ExternalName kind to demonstrate and te…
IfSentient Mar 20, 2024
61c0839
Removed unused examples/apiserver/apis/resource directory
IfSentient Mar 20, 2024
38069d8
Update codegen to better handle grouping, have codecs prefixed by kin…
IfSentient Mar 20, 2024
19ed13a
Moved APIServerOptions and NewCommandStartAPIServer from into , to g…
IfSentient Mar 20, 2024
5fd31a9
Tiny update to README and comments to main.
IfSentient Mar 21, 2024
46055b0
add admission support
toddtreece Mar 26, 2024
08f8b02
progress on admission hooks
toddtreece Mar 27, 2024
260d1c0
fix admission bug related to multiple versions
toddtreece Apr 8, 2024
6ab35c4
Merge branch 'main' into apiserver-poc
IfSentient Apr 10, 2024
5d764a2
Fix jennies/generators for tests.
IfSentient Apr 10, 2024
38e110a
downgrade client_go dependency
toddtreece Apr 11, 2024
28a1b0a
Set prometheus/common version to v0.46.0 to avoid test compile error.
IfSentient Apr 11, 2024
340cddf
Updated openapi jenny to correctly by-package codegen openAPI (instea…
IfSentient Apr 15, 2024
ac838a1
Merge branch 'main' of https://github.com/grafana/grafana-app-sdk int…
toddtreece Apr 16, 2024
e259e1d
Update apiserver example go.mod and parent go.work file so the exampl…
IfSentient Apr 17, 2024
dc7a517
Introduced APIGroupProvider interface and updated apiserver.ResourceG…
IfSentient Apr 19, 2024
334d964
Change apiserver.Resource.Reconciler field to a function that instant…
IfSentient Apr 19, 2024
da5b5ab
Remove accidental generated code from main branch CLI.
IfSentient Apr 19, 2024
41156de
Added convenience functions for GVK and GVR to resource.Kind
IfSentient May 1, 2024
f9c721d
Setup post-start hooks in apiserver.Config.Complete().
IfSentient May 1, 2024
bb9ae54
Updated resource conversion to be in the apiserver.Resource type inst…
IfSentient May 24, 2024
ed1af17
initial work on an app-centric workflow and ability to use a plugin a…
IfSentient Jul 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 63 additions & 114 deletions apiserver/resource.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,18 +37,6 @@ type Resource struct {
Reconciler operator.Reconciler // TODO: do we want this here, or only here for the simple package version?
}

func (r *Resource) RegisterAdmissionPlugin(plugins *admission.Plugins) {
if r.Validator == nil && r.Mutator == nil {
fmt.Printf("Resource %s has no admission plugins\n", r.Kind.Kind())
return
}
gvk := r.Kind.Group() + "/" + r.Kind.Version() + "/" + r.Kind.Kind()
fmt.Printf("Registering admission plugins for %s\n", gvk)
plugins.Register(gvk+"Admission", func(config io.Reader) (admission.Interface, error) {
return &AdmissionWrapper{resource: r}, nil
})
}

func (r *Resource) AddToScheme(scheme *runtime.Scheme) {
gv := schema.GroupVersion{
Group: r.Kind.Group(),
Expand All @@ -65,89 +53,79 @@ func (r *Resource) AddToScheme(scheme *runtime.Scheme) {
}
}

var _ admission.MutationInterface = &AdmissionWrapper{}
var _ admission.ValidationInterface = &AdmissionWrapper{}
var _ admission.Interface = &AdmissionWrapper{}

type AdmissionWrapper struct {
resource *Resource
func (r *Resource) RegisterAdmissionPlugin(plugins *admission.Plugins) {
if r.Validator == nil && r.Mutator == nil {
return
}
gvk := r.Kind.Group() + "/" + r.Kind.Version() + "/" + r.Kind.Kind()
plugins.Register(gvk+"Admission", func(config io.Reader) (admission.Interface, error) {
return r, nil
})
}

func (v *AdmissionWrapper) Handles(o admission.Operation) bool {
if v.resource.Validator == nil && v.resource.Mutator == nil {
func (r *Resource) Handles(o admission.Operation) bool {
if r.Validator == nil && r.Mutator == nil {
return false
}
return true
}

func (v *AdmissionWrapper) Validate(ctx context.Context, a admission.Attributes, o admission.ObjectInterfaces) error {
if v.resource.Validator == nil {
func (r *Resource) Validate(ctx context.Context, a admission.Attributes, o admission.ObjectInterfaces) error {
if r.Validator == nil {
return nil
}

// skip if the gvk doesn't match
if a.GetKind().Kind != v.resource.Kind.Kind() || a.GetKind().Group != v.resource.Kind.Group() || a.GetKind().Version != v.resource.Kind.Version() {
if !r.matchesResouce(a) {
return nil
}

userInfoExtra := make(map[string]any)
for k, v := range a.GetUserInfo().GetExtra() {
userInfoExtra[k] = v
req, err := buildAdmissionRequest(a)
if err != nil {
return err
}

userInfo := resource.AdmissionUserInfo{}
if a.GetUserInfo() != nil {
userInfoExtra := make(map[string]any)
for k, v := range a.GetUserInfo().GetExtra() {
userInfoExtra[k] = v
}
userInfo.Extra = userInfoExtra
userInfo.Groups = a.GetUserInfo().GetGroups()
userInfo.UID = a.GetUserInfo().GetUID()
userInfo.Username = a.GetUserInfo().GetName()
return r.Validator.Validate(ctx, req)
}

func (r *Resource) Admit(ctx context.Context, a admission.Attributes, o admission.ObjectInterfaces) error {
if r.Mutator == nil {
return nil
}

var (
obj, oldObj resource.Object
ok bool
)
if !r.matchesResouce(a) {
return nil
}

obj, ok = a.GetObject().(resource.Object)
if !ok {
return errors.NewInternalError(fmt.Errorf("new obj is not a valid resource.Object"))
req, err := buildAdmissionRequest(a)
if err != nil {
return err
}

if a.GetOldObject() != nil {
oldObj, ok = a.GetOldObject().(resource.Object)
if !ok {
return errors.NewInternalError(fmt.Errorf("old object is not a valid resource.Object"))
}
res, err := r.Mutator.Mutate(ctx, req)
if err != nil {
return err
}

req := &resource.AdmissionRequest{
Action: resource.AdmissionAction(a.GetOperation()),
Kind: a.GetKind().Kind,
Group: a.GetKind().Group,
Version: a.GetKind().Group,
UserInfo: userInfo,
Object: obj,
OldObject: oldObj,
if err := copyModifiedObjectToDestination(res.UpdatedObject, req.Object); err != nil {
return errors.NewInternalError(fmt.Errorf("unable to copy updated object to destination: %w", err))
}

return v.resource.Validator.Validate(ctx, req)
return nil
}

func (v *AdmissionWrapper) Admit(ctx context.Context, a admission.Attributes, o admission.ObjectInterfaces) error {
if v.resource.Mutator == nil {
return nil
}

// skip if the gvk doesn't match
if a.GetKind().Kind != v.resource.Kind.Kind() || a.GetKind().Group != v.resource.Kind.Group() || a.GetKind().Version != v.resource.Kind.Version() {
return nil
}
func (r *Resource) matchesResouce(a admission.Attributes) bool {
kindMatch := a.GetKind().Kind == r.Kind.Kind()
groupMatch := a.GetKind().Group == r.Kind.Group()
versionMatch := a.GetKind().Version == r.Kind.Version()
return kindMatch && groupMatch && versionMatch
}

userInfo := resource.AdmissionUserInfo{}
func buildAdmissionRequest(a admission.Attributes) (*resource.AdmissionRequest, error) {
var (
userInfo = resource.AdmissionUserInfo{}
obj, oldObj resource.Object
ok bool
)
if a.GetUserInfo() != nil {
userInfoExtra := make(map[string]any)
for k, v := range a.GetUserInfo().GetExtra() {
Expand All @@ -159,43 +137,27 @@ func (v *AdmissionWrapper) Admit(ctx context.Context, a admission.Attributes, o
userInfo.Username = a.GetUserInfo().GetName()
}

var (
obj, oldObj resource.Object
ok bool
)

obj, ok = a.GetObject().(resource.Object)
if !ok {
return errors.NewInternalError(fmt.Errorf("new obj is not a valid resource.Object"))
return nil, errors.NewInternalError(fmt.Errorf("new obj is not a valid resource.Object"))
}

if a.GetOldObject() != nil {
oldObj, ok = a.GetOldObject().(resource.Object)
if !ok {
return errors.NewInternalError(fmt.Errorf("old object is not a valid resource.Object"))
return nil, errors.NewInternalError(fmt.Errorf("old object is not a valid resource.Object"))
}
}

req := &resource.AdmissionRequest{
return &resource.AdmissionRequest{
Action: resource.AdmissionAction(a.GetOperation()),
Kind: a.GetKind().Kind,
Group: a.GetKind().Group,
Version: a.GetKind().Group,
UserInfo: userInfo,
Object: obj,
OldObject: oldObj,
}

res, err := v.resource.Mutator.Mutate(ctx, req)
if err != nil {
return err
}

if err := copyModifiedObjectToDestination(res.UpdatedObject, obj); err != nil {
return errors.NewInternalError(fmt.Errorf("unable to copy updated object to destination: %w", err))
}

return nil
}, nil
}

func copyModifiedObjectToDestination(updatedObj runtime.Object, destination runtime.Object) error {
Expand Down Expand Up @@ -248,30 +210,18 @@ type ResourceGroup struct {
// This SHOULD be supplied if multiple versions of the same GroupKind exist in the ResourceGroup.
// If not supplied, a GenericConverter will be used for all conversions.
// This can be empty or nil and specific MutatingAdmissionControllers can be set later with Operator.MutateKind
Converters map[metav1.GroupKind]Converter
converters map[metav1.GroupKind]Converter
}

func (g *ResourceGroup) Scheme() *runtime.Scheme {
scheme := runtime.NewScheme()
g.AddToScheme(scheme)
return scheme
func NewResourceGroup(name string, resources []Resource) *ResourceGroup {
g := &ResourceGroup{
Name: name,
Resources: resources,
}
return g
}

func (g *ResourceGroup) AddToScheme(scheme *runtime.Scheme) {
// we need to add the options to empty v1
// TODO fix the server code to avoid this
metav1.AddToGroupVersion(scheme, schema.GroupVersion{Version: "v1"})

// TODO: keep the generic API server from wanting this
unversioned := schema.GroupVersion{Group: "", Version: "v1"}
scheme.AddUnversionedTypes(unversioned,
&metav1.Status{},
&metav1.APIVersions{},
&metav1.APIGroupList{},
&metav1.APIGroup{},
&metav1.APIResourceList{},
)

func (g *ResourceGroup) AddToScheme(scheme *runtime.Scheme) error {
// TODO: this assumes items in the Resources slice are ordered by version
versions := make(map[metav1.GroupKind][]Resource)
for _, r := range g.Resources {
Expand All @@ -291,14 +241,15 @@ func (g *ResourceGroup) AddToScheme(scheme *runtime.Scheme) {
Group: latest.Kind.Group(),
Version: runtime.APIVersionInternal,
}

scheme.AddKnownTypeWithName(gv.WithKind(gk.Kind), latest.Kind.ZeroValue())
scheme.AddKnownTypeWithName(gv.WithKind(gk.Kind+"List"), latest.Kind.ZeroListValue())

// Get the converter for this GroupKind, or use a Generic one if none was supplied
var converter Converter = GenericConverter{}
if g.Converters != nil {
if g.converters != nil {
ok := false
converter, ok = g.Converters[gk]
converter, ok = g.converters[gk]
if !ok {
converter = GenericConverter{}
}
Expand All @@ -325,6 +276,7 @@ func (g *ResourceGroup) AddToScheme(scheme *runtime.Scheme) {
// Set version priorities based on the reverse-order list we built
scheme.SetVersionPriority(priorities...)
}
return nil
}

type StorageProviderFunc func(resource.Kind, *runtime.Scheme, generic.RESTOptionsGetter) (rest.Storage, error)
Expand All @@ -343,10 +295,7 @@ type StorageProvider2 interface {
StandardStorage(kind resource.Kind, scheme *runtime.Scheme) (StandardStorage, error)
}

func (g *ResourceGroup) APIGroupInfo(storageProvider StorageProvider2) (*server.APIGroupInfo, error) {
scheme := g.Scheme() // TODO: have this be an argument?
parameterCodec := runtime.NewParameterCodec(scheme) // TODO: have this be an argument?
codecs := serializer.NewCodecFactory(scheme) // TODO: codec based on kinds?
func (g *ResourceGroup) APIGroupInfo(scheme *runtime.Scheme, codecs serializer.CodecFactory, parameterCodec runtime.ParameterCodec, storageProvider StorageProvider2) (*server.APIGroupInfo, error) {
apiGroupInfo := genericapiserver.NewDefaultAPIGroupInfo(g.Name, scheme, parameterCodec, codecs)
for _, r := range g.Resources {
plural := strings.ToLower(r.Kind.Plural())
Expand Down
7 changes: 2 additions & 5 deletions examples/apiserver/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,11 @@ func main() {
// apiserver.Resource items must be added to an apiserver.ResourceGroup.
// Currently, there is no validation that the Name in the ResourceGroup matches the Group in each added Resource
// TODO: have a Validate() method on ResourceGroup to check that?
resourceGroup := apiserver.ResourceGroup{
Name: corev1.ExternalNameKind().Group(),
Resources: []apiserver.Resource{externalNameV1, externalNameV2},
}
resourceGroup := apiserver.NewResourceGroup(corev1.ExternalNameKind().Group(), []apiserver.Resource{externalNameV1, externalNameV2})

// APIServerOptions is used to create the API server from one or more ResourceGroups.
// TODO: this will be expanded upon
o := simple.NewAPIServerOptions([]apiserver.ResourceGroup{resourceGroup}, os.Stdout, os.Stderr)
o := simple.NewAPIServerOptions([]apiserver.ResourceGroup{*resourceGroup}, os.Stdout, os.Stderr)
o.RecommendedOptions.Authorization = nil
o.RecommendedOptions.Authentication = nil
o.RecommendedOptions.CoreAPI = nil
Expand Down
14 changes: 7 additions & 7 deletions simple/apiserver.go
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ type Server struct {
}

func (c completedConfig) NewServer() (*Server, error) {
//scheme := c.ExtraConfig.Scheme
//codecs := c.ExtraConfig.Codecs
scheme := c.ExtraConfig.Scheme
codecs := c.ExtraConfig.Codecs
openapiGetters := []common.GetOpenAPIDefinitions{}

for _, g := range c.ExtraConfig.ResourceGroups {
Expand All @@ -129,10 +129,10 @@ func (c completedConfig) NewServer() (*Server, error) {
GenericAPIServer: genericServer,
}

//parameterCodec := runtime.NewParameterCodec(scheme)
parameterCodec := runtime.NewParameterCodec(scheme)
provider := apiserver.NewRESTStorageProvider(c.GenericConfig.RESTOptionsGetter)
for _, g := range c.ExtraConfig.ResourceGroups {
apiGroupInfo, err := g.APIGroupInfo(provider)
apiGroupInfo, err := g.APIGroupInfo(scheme, codecs, parameterCodec, provider)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -166,7 +166,6 @@ func NewAPIServerOptions(groups []apiserver.ResourceGroup, out, errOut io.Writer
Version: r.Kind.Version(),
}
gvs = append(gvs, gv)

}
}

Expand All @@ -184,8 +183,9 @@ func NewAPIServerOptions(groups []apiserver.ResourceGroup, out, errOut io.Writer
}

o.RecommendedOptions.Admission.Plugins = admission.NewPlugins()
for _, g := range groups {
for _, r := range g.Resources {
for gid, g := range groups {
for rid, _ := range g.Resources {
r := &groups[gid].Resources[rid]
Copy link
Member Author

@toddtreece toddtreece Apr 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@IfSentient the issue with multiple versions was the pointer in the range loop changing. making a new pointer fixed the issue

r.RegisterAdmissionPlugin(o.RecommendedOptions.Admission.Plugins)
}
}
Expand Down
Loading