Multi-cluster Kubernetes Orchestration

  • By KubeWharf
  • Last update: Aug 15, 2023
  • Comments: 17

KubeAdmiral - Enhanced Kubernetes Federation

English | 简体中文

KubeAdmiral is a multi-cluster management system for Kubernetes, developed from Kubernetes Federation v2. Kubernetes Federation v2 allows users to manage Kubernetes resources across multiple clusters through the use of federated types such as FederatedDeployment, FederatedReplicaSet, FederatedSecret, etc. KubeAdmiral extends the Kubernetes Federation v2 API, providing compatibility with the Kubernetes native API and more powerful resource management capabilities. KubeAdmiral also adds new features such as:

  • A new scheduling framework with a rich set of scheduling plugins.
  • Override policies.
  • Automatic propagation of dependencies with follower scheduling.
  • Status aggregation of member cluster resources.
  • Scalability, stability and user experience enhancements

Getting started

KubeAdmiral supports Kubernetes versions from 1.16 up to 1.24. Using lower or higher Kubernetes versions may cause compatibility issues. For setup please refer to Quickstart.

Community

Contributing

If you are willing to be a contributor for the KubeAdmiral project, please refer to our CONTRIBUTING document for details.

Contact

If you have any questions or wish to contribute, you are welcome to communicate via GitHub issues or pull requests. Alternatively, you may reach out to our Maintainers.

License

KubeAdmiral is under the Apache 2.0 license. See the LICENSE file for details. KubeAdmiral is a continuation of Kubernetes Federation v2, and certain features in KubeAdmiral rely on existing code from Kubernetes — all credits go to the original Kubernetes authors. We also refer to Karmada for some of our architecture and API design, all relevant credits go to the Karmada Authors.

Download

kubeadmiral.zip

Comments(17)

  • 1

    feat: collect cluster resources without caching pods

    Overview

    Collecting member cluster resources is required for resource-based scheduling plugins, such as ClusterResourcesFit, ClusterResourcesBalancedAllocation, etc.

    Currently, we collect the available and allocatable member cluster resources by maintaining a Node and Pod informer for each member cluster. The number of schedulable nodes and the total allocatable resources is obtained from the Node informer. The total requested resources (which is subtracted from the total allocatable resources to get the total available resources) is obtained from the Pod informer.

    This naturally does not scale well as informers maintain a copy of each object in memory.

    This PR introduces a new mechanism for collecting cluster resources that does not require the caching of pod objects. With this new mechanism, we no longer require the use of a pod informer to collect the total requested resources in a cluster.

    This new mechanism works based on the premise that:

    1. Pod resource requests cannot be changed
    2. List pagination is consistent and returns a single list resource version

    How the new mechanism works

    1. We start with a paginated list and process each page immediately after we get it. We simply sum up the resources requested by each pod (excluding completed pods) to get a set of resource quantities.
    2. Once we are done listing, we start a watch request based on the resource version obtained from the list request above. Because pod resource requests cannot change, we only need to care about when a pod is created, completed, or deleted. When a pod is created, we add its requested resources, and when a pod is completed/deleted, we subtract its requested resources (the requested resources can be obtained from the object in the watch event).
    3. To avoid double counting when a completed pod gets deleted, we maintain the set of uids for completed pods that we have accounted for but are not yet deleted.
    4. When a watch connection drops, we freeze the current set of resource quantities, and start a (paginated) relist where we add up the requested resources again from 0 to obtain a new set of resource quantities. Once the list is complete, we simply replace the old set with the new set and continue from step 2.

    Memory improvements

    (In relation to collecting total requested resources)

    Previously, a copy of every pod in each member cluster was stored in memory. During relists, where we have to deserialize a complete list of pods again, the memory might even double.

    With this new mechanism, the amount of memory used would be O(number of member clusters * page size). The amount of memory used for maintaining the set of completed-but-not-terminated-pods' uids should be insignificant.

    Results

    I ran two versions of the KubeAdmiral controller-manager against test clusters where I simulated the creation and deletion of 10000 pods at regular intervals. The first version is the main branch, while the second version uses the new cacheless collector. Periodic relists were triggered by pausing the apiservers to ensure that relists do not affect memory usage and correctness

    I also hooked up some temporary metrics for evaluating the memory usage and correctness. Below are the results:

    Heap alloc of the 2 versions: image

    Reported CPU available (both correspond) image

    Reported memory available (both correspond) image

  • 2

    feat: add commands for building docker images and new kubeadmiral local startup

    What this PR does?

    1. Modify the script to support compiling binary files for multiple platforms at one time(i.e., linux/darwin, amd64/arm64);
    2. Add scripts to quickly generate images for different platfroms(i.e., linux/amd64, linux/arm64);
    3. Add scripts to help users quickly start a local kubeadmiral;
    4. Add scripts to help users quickly clean up local kind clusters related to kubeadmiral;
    5. Restructure the kubeadmiral/hack directory structure to make the script easier to read.

    Does this PR introduce a user-facing change?

    • Users can use the following make command to compile binary files for multiple platforms:
    # Production build
#
# Args:
#   BUILD_PLATFORMS: platforms to build. e.g.: linux/amd64,darwin/amd64. The default value is the host platform.
#
# Example:
#   # compile kubeadmiral-controller-manager with the host platform
#   make build
#   # compile kubeadmiral-controller-manager with specified platforms
#   make build BUILD_PLATFORMS=linux/amd64,darwin/amd64
$ make build
    • Users can use the following make command to generate docker images for multiple architectures:
    # Build binaries and docker images.
# The supported OS is linux, and user can specify the arch type (only amd64,arm64,arm are supported)
#
# ARGS:
#   REGISTRY:    image registry, the default value is "ghcr.io/kubewharf"
#   TAG:         image tag, the default value is "latest"
#   ARCHS:       list of target architectures, e.g.:amd64,arm64,arm. The default value is host arch
#
# Examples:
#   # build images with the host arch, it will generate "ghcr.io/kubewharf/kubeadmiral-controller-manager:latest"
#   make images
#
#   # build images with amd64,arm64 arch
#   # note: If you specify multiple architectures, the image tag will be added with the architecture name,e.g.:
#   # 		"ghcr.io/kubewharf/kubeadmiral-controller-manager:latest-amd64"
#   # 		"ghcr.io/kubewharf/kubeadmiral-controller-manager:latest-arm64"
#   make images ARCHS=amd64,arm64
$ make images
    • Users can use the following make command to quickly start and experience kubeadmiral locally
    # Local up the KubeAdmiral.
#
# Args:
#   NUM_MEMBER_CLUSTERS: The number of member clusters you want to startup, default is 3, at least 1.
#   REGION:              region to build. e.g.: 'cn' means china mainland(the default value)
#
# Example:
#   # It will locally start a meta-cluster to deploy the kubeadmiral control-plane, and three member cluster clusters
#   make local-up
$ make local-up
    • Users can clean up the local kind clusters started by the make local-up command with the following command
    # Clean up all the local kind clusters and related kubeconfigs
$ make clean-local-cluster
  • 3

    chore(deps): bump k8s.io/klog/v2 from 2.80.1 to 2.90.1

    Bumps k8s.io/klog/v2 from 2.80.1 to 2.90.1.

    Release notes

    Sourced from k8s.io/klog/v2's releases.

    Prepare klog release for Kubernetes v1.27 (Take 2)

    What's Changed

    Full Changelog: https://github.com/kubernetes/klog/compare/v2.90.0...v2.90.1

    Prepare klog release for Kubernetes v1.27 (Take 1)

    What's Changed

    Full Changelog: https://github.com/kubernetes/klog/compare/v2.80.1...v2.90.0

    There are some API differences from previous version

    k8s.io/klog/v2/klogr contains incompatible changes:
 - klogger.Enabled: removed
 - klogger.Error: removed
 - klogger.Info: removed

    k8s.io/klog/v2/test contains incompatible changes:
    

      

    • InitKlog: changed from func() to func(testing.TB) *flag.FlagSet
Commits
  • d7fc505 Merge pull request #363 from pohly/textlogger-backend
  • e37f9fe Merge pull request #368 from pohly/ktesting-config-api
  • a0fea0c textlogger: verbosity changes through public API
  • e092d89 ktesting: support verbosity changes at runtime
  • 90cff0f Merge pull request #365 from pohly/ktesting-timestamps
  • af72dbd Merge pull request #366 from pohly/buffer-size-limit
  • 1b27ee8 ktesting: allow overriding default formatter
  • 2582956 buffer: restore dropping of too large buffers
  • d113925 ktesting: capture log entries only if requested
  • 8b4cfd2 ktesting: use klog-like header
  • Additional commits viewable in compare view

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 4

    feat: support pod propagation

    This PR supports the federation of pods using KubeAdmiral by introducing the following:

    1. Add a new FTC for pods
    2. Implement drop and retain logic for pod fields before syncing

    Additionally:

    1. The kube-scheduler is now disabled by the dev-up.sh script for the host cluster.
    2. dev-up.sh now supports running 0 member clusters
  • 5

    chore(deps): bump sigs.k8s.io/kind from 0.17.0 to 0.18.0

    Bumps sigs.k8s.io/kind from 0.17.0 to 0.18.0.

    Release notes

    Sourced from sigs.k8s.io/kind's releases.

    v0.18.0 - Thanks Docker! 🐳

    KIND v0.18.0 Comes with a big shoutout to Docker, Inc. for accepting us into the updated Docker Sponsored OSS Program. Thanks Docker! 🎉

    Images should no longer have pull rate limits as a result.

    The project will still consider mirroring on or switching primarily to registry.k8s.io in the future, after determining an updated immutable tagging scheme to comply with requirements there.

    Otherwise of particular note are a fix for iptables nf_tables v1.8.8+, updated dependencies including runc v1.1.5 with CVE fixes, and a new networking option to control the DNS search list.

    • The default node image is a Kubernetes v1.26.3 image: kindest/node:v1.26.3@sha256:61b92f38dff6ccc29969e7aa154d34e38b89443af1a2c14e6cfbd2df6419c66f
    • Dropped support for PPC64LE and S390x, which only had limited support previously
      • These platforms had very slow and flaky builds despite attempts at fixing and very limited demand. We've dropped these to focus on the vast majority of our users. These platforms never reached the point of having official node images.
    • Removed registry mirror config for k8s.gcr.io => registry.k8s.io
    • New networking.dnsSearch config field for overriding the cluster nodes' DNS search list
    • Documented how to use KIND on chromeOS
    • Automated builds for most images
    • Improved output for kind delete cluster

    New Node images have been built for kind v0.18.0, please use these exact images (IE like kindest/node:v1.26.3@sha256: 61b92f38dff6ccc29969e7aa154d34e38b89443af1a2c14e6cfbd2df6419c66f including the digest) or build your own as we may need to change the image format again in the future :sweat_smile:

    Images pre-built for this release:

    • 1.26: kindest/node:v1.26.3@sha256:61b92f38dff6ccc29969e7aa154d34e38b89443af1a2c14e6cfbd2df6419c66f
    • 1.25: kindest/node:v1.25.8@sha256:00d3f5314cc35327706776e95b2f8e504198ce59ac545d0200a89e69fce10b7f
    • 1.24: kindest/node:v1.24.12@sha256:1e12918b8bc3d4253bc08f640a231bb0d3b2c5a9b28aa3f2ca1aee93e1e8db16
    • 1.23: kindest/node:v1.23.17@sha256:e5fd1d9cd7a9a50939f9c005684df5a6d145e8d695e78463637b79464292e66c
    • 1.22: kindest/node:v1.22.17@sha256:c8a828709a53c25cbdc0790c8afe12f25538617c7be879083248981945c38693
    • 1.21: kindest/node:v1.21.14@sha256:27ef72ea623ee879a25fe6f9982690a3e370c68286f4356bf643467c552a3888

    See also:

    NOTE: These node images support amd64 and arm64, both of our supported platforms. You must use the same platform as your host, for more context see kubernetes-sigs/kind#2718

    • Fixed iptables rules when the host has iptables 1.8.8+ in nf_tables (not legacy) mode

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 6

    feat: controller health checks

    This PR removes the existing healthz endpoint and adds more specific readyz and livez endpoints, see https://kubernetes.io/docs/reference/using-api/health-checks/#api-endpoints-for-health.

    In addition, it adds a readyz check for each controller run by the ControllerManager and FTCManager. The root readyz check aggregates the results from each individual controller.

    Examples:

    % curl "localhost:11257/readyz?verbose"
[+]configmaps-automigration ok
[+]configmaps-federate ok
[+]configmaps-scheduler ok
[+]cronjobs.batch-automigration ok
[+]cronjobs.batch-federate ok
[+]cronjobs.batch-scheduler ok
[+]daemonsets.apps-automigration ok
[+]daemonsets.apps-federate ok
[+]daemonsets.apps-scheduler ok
[+]deployments.apps-automigration ok
[+]deployments.apps-federate ok
[+]deployments.apps-scheduler ok
[-]federatedcluster failed: reason withheld
[+]follower ok
[+]ingresses.networking.k8s.io-automigration ok
[+]ingresses.networking.k8s.io-federate ok
[+]ingresses.networking.k8s.io-scheduler ok
[+]jobs.batch-automigration ok
[+]jobs.batch-federate ok
[+]jobs.batch-scheduler ok
[+]limitranges-automigration ok
[+]limitranges-federate ok
[+]limitranges-scheduler ok
[+]namespaces-automigration ok
[+]namespaces-federate ok
[+]namespaces-scheduler ok
[+]persistentvolumeclaims-automigration ok
[+]persistentvolumeclaims-federate ok
[+]persistentvolumeclaims-scheduler ok
[+]persistentvolumes-automigration ok
[+]persistentvolumes-federate ok
[+]persistentvolumes-scheduler ok
[+]resourcequotas-automigration ok
[+]resourcequotas-federate ok
[+]resourcequotas-scheduler ok
[+]secrets-automigration ok
[+]secrets-federate ok
[+]secrets-scheduler ok
[+]serviceaccounts-automigration ok
[+]serviceaccounts-federate ok
[+]serviceaccounts-scheduler ok
[+]services-automigration ok
[+]services-federate ok
[+]services-scheduler ok
[+]statefulsets.apps-automigration ok
[+]statefulsets.apps-federate ok
[+]statefulsets.apps-scheduler ok
[+]storageclasses-automigration ok
[+]storageclasses-federate ok
[+]storageclasses-scheduler ok
[+]typeconfig ok
healthz check failed

    % curl -i "localhost:11257/readyz/configmaps-automigration"
HTTP/1.1 200 OK
Date: Thu, 04 May 2023 05:42:31 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8

    % curl -i "localhost:11257/readyz/federatedcluster" 
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Thu, 04 May 2023 05:43:52 GMT
Content-Length: 61

internal server error: controller federatedcluster not ready

    Livez checks consist of a simple ping check and a leader election check.

    Examples

    % curl "localhost:11257/livez?verbose" 
[+]leaderElection ok
[+]ping ok
healthz check passed
  • 7

    feat: customize pod informer to reduce memory usage

    Use a custom Pod Informer to limit the number of concurrent list pods from member clusters, and simplify the metadata of the pods before we store them into the cache.

    The reference conversations: https://github.com/kubewharf/kubeadmiral/pull/14

    As result: image 4 clusters, 100 kwok-node/cluster, 10 deployment/cluster, 1000 pod/deployment, total: 10k pod/cluster

    • case1: simplify the pod (In the last three rounds, kwok failed to delete pods of a cluster so the pods of cluster came to 20k; and some kwok nodes got not ready, so the available cpu,gpu came to zero, but the pods still in work. The overall impact is not significant.)
    • case2: regular pod informer
    • case3: simplify the pod and limit 2 available listers
    • case4: simplify the pod, ignore modified event of watch, and limit 2 available listers

    summary: It can save at least half of the memory, and reduce the rate of change of memory usage

  • 8

    chore(deps): bump k8s.io/apiextensions-apiserver from 0.26.0 to 0.27.1

    Bumps k8s.io/apiextensions-apiserver from 0.26.0 to 0.27.1.

    Commits
    • 70c3191 Update dependencies to v0.27.1 tag
    • 15ea0ae Merge pull request #117305ncdc/automated-cherry-pick-of-#117301
    • c701fe6 Revert "Merge pull request #113151 from ncdc/refactor-crd-conversion"
    • 3735b65 Revert "CR conversion: protect from converter input edits"
    • 2c580f6 Merge remote-tracking branch 'origin/master' into release-1.27
    • 072e6ee .*: update vendor dir and cleanup
    • eebba51 allow multiple sources to add/remove from discovery without clobbering each o...
    • 508f83b Merge pull request #116539 from pohly/ginkgo-gomega-update
    • 6a644f1 dependencies: ginkgo v2.9.1, gomega v1.27.4
    • 9b2bf1f Merge pull request #116490 from tallclair/docs-urls2
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 9

    chore(deps): bump k8s.io/api from 0.26.0 to 0.27.1

    Bumps k8s.io/api from 0.26.0 to 0.27.1.

    Commits
    • 6c11c9e Update dependencies to v0.27.1 tag
    • 5b93db5 Merge remote-tracking branch 'origin/master' into release-1.27
    • 1258df6 .*: update vendor dir and cleanup
    • 9d61da7 remove kubernetes.io/grpc standard protocol
    • 130e058 Merge pull request #116743 from thockin/docs-clarify-publish-not-ready-endpoints
    • c5c9df1 Clarify EPSlice docs wrt the Ready conditions
    • bad7d34 Merge pull request #113218 from ahmedtd/kep-3257
    • 7d2e0cc ClusterTrustBundles: make update
    • 27a32d4 ClusterTrustBundles: Define types
    • c80582e Custom match criteria (#116350)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 10

    chore(deps): bump github.com/onsi/ginkgo/v2 from 2.8.2 to 2.9.2

    Bumps github.com/onsi/ginkgo/v2 from 2.8.2 to 2.9.2.

    Release notes

    Sourced from github.com/onsi/ginkgo/v2's releases.

    v2.9.2

    2.9.2

    Maintenance

    • Bump github.com/go-task/slim-sprig (#1167) [3fcc5bf]
    • Bump github.com/onsi/gomega from 1.27.3 to 1.27.4 (#1163) [6143ffe]

    v2.9.1

    2.9.1

    Fixes

    This release fixes a longstanding issue where ginkgo -coverpkg=./... would not work. This is now resolved and fixes #1161 and #995

    • Support -coverpkg=./... [26ca1b5]
    • document coverpkg a bit more clearly [fc44c3b]

    Maintenance

    • bump various dependencies
    • Improve Documentation and fix typo (#1158) [93de676]

    v2.9.0

    2.9.0

    Features

    • AttachProgressReporter is an experimental feature that allows users to provide arbitrary information when a ProgressReport is requested [28801fe]

    • GinkgoT() has been expanded to include several Ginkgo-specific methods [2bd5a3b]

      The intent is to enable the development of third-party libraries that integrate deeply with Ginkgo using GinkgoT() to access Ginkgo's functionality.

    v2.8.4

    2.8.4

    Features

    • Add OmitSuiteSetupNodes to JunitReportConfig (#1147) [979fbc2]
    • Add a reference to ginkgolinter in docs.index.md (#1143) [8432589]

    Fixes

    • rename tools hack to see if it fixes things for downstream users [a8bb39a]

    Maintenance

    • Bump golang.org/x/text (#1144) [41b2a8a]
    • Bump github.com/onsi/gomega from 1.27.0 to 1.27.1 (#1142) [7c4f583]

    v2.8.3

    2.8.3

    Released to fix security issue in golang.org/x/net dependency

    Maintenance

    ... (truncated)

    Changelog

    Sourced from github.com/onsi/ginkgo/v2's changelog.

    2.9.2

    Maintenance

    • Bump github.com/go-task/slim-sprig (#1167) [3fcc5bf]
    • Bump github.com/onsi/gomega from 1.27.3 to 1.27.4 (#1163) [6143ffe]

    2.9.1

    Fixes

    This release fixes a longstanding issue where ginkgo -coverpkg=./... would not work. This is now resolved and fixes #1161 and #995

    • Support -coverpkg=./... [26ca1b5]
    • document coverpkg a bit more clearly [fc44c3b]

    Maintenance

    • bump various dependencies
    • Improve Documentation and fix typo (#1158) [93de676]

    2.9.0

    Features

    • AttachProgressReporter is an experimental feature that allows users to provide arbitrary information when a ProgressReport is requested [28801fe]

    • GinkgoT() has been expanded to include several Ginkgo-specific methods [2bd5a3b]

      The intent is to enable the development of third-party libraries that integrate deeply with Ginkgo using GinkgoT() to access Ginkgo's functionality.

    2.8.4

    Features

    • Add OmitSuiteSetupNodes to JunitReportConfig (#1147) [979fbc2]
    • Add a reference to ginkgolinter in docs.index.md (#1143) [8432589]

    Fixes

    • rename tools hack to see if it fixes things for downstream users [a8bb39a]

    Maintenance

    • Bump golang.org/x/text (#1144) [41b2a8a]
    • Bump github.com/onsi/gomega from 1.27.0 to 1.27.1 (#1142) [7c4f583]

    2.8.3

    Released to fix security issue in golang.org/x/net dependency

    Maintenance

    • Bump golang.org/x/net from 0.6.0 to 0.7.0 (#1141) [fc1a02e]
    • remove tools.go hack from documentation [0718693]
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 11

    feat: implement apply later for propagation policy

    Three new annotations are introduced on fedObj to implement the ApplyLater feature. Specifically: if a resource is associated with a policy, three annotations will be marked on its fedObj, namely:

    • kubeadmiral.io/last-applied-propagation-policy-generation Its value indicates the generation version of the policy that took effect last time.
    • kubeadmiral.io/last -applied-template-hash Its value represents the hash value of the fedObj.spec.template content that took effect last time.
    • kubeadmiral.io/observed-propagation-policy-generation Its value represents the generation version of the policy observed by the scheduler.

    If policy.spec.applyPolicy = ApplyLater, then the change of the policy will not cause the change of the PolicyGeneration in schedulingTriggers, only the change of the original resource will make the PolicyGeneration change, thus triggering the rescheduling based on the policy. Therefore, kubeadmiral.io/last-applied-template-hash is introduced to observe whether resources change, and kubeadmiral.io/last-applied-propagation-policy-generation is introduced to retain last applied PolicyGeneration in schedulingTriggers.

    And kubeadmiral.io/observed-propagation-policy-generation is introduced to easily observe whether the current resource is already the latest policy result from fedObj: last-applied-propagation-policy-generation == observed-propagation-policy-generation, the latest policy has been applied, otherwise the latest policy has not been applied.

    If policy.spec.applyPolicy = ApplyNow (this is the default behavior), the change of policy will change the PolicyGeneration in schedulingTriggers and then trigger the scheduler rescheduling.

    When a policy is removed from a resource, the above three annotations will be removed on its fedObj too.

    Note: The first time a resource is associated with a new policy, scheduling will be triggered immediately anyway.

    Here is an example when policy.spec.applyPolicy = ApplyLater:

    apiVersion: types.kubeadmiral.io/v1alpha1
kind: FederatedConfigMap
metadata:
  annotations:
    federate.controller.kubeadmiral.io/observed-annotations: '|kubeadmiral.io/scheduling,kubeadmiral.io/syncing'
    federate.controller.kubeadmiral.io/observed-labels: kubeadmiral.io/propagation-policy-name|aa,aaa
    federate.controller.kubeadmiral.io/template-generator-merge-patch: '{"metadata":{"annotations":{"kubeadmiral.io/scheduling":null,"kubeadmiral.io/syncing":null},"creationTimestamp":null,"finalizers":null,"labels":{"kubeadmiral.io/propagation-policy-name":null},"managedFields":null,"resourceVersion":null,"uid":null}}'
    internal.kubeadmiral.io/enable-follower-scheduling: "true"
    kubeadmiral.io/federated-object: "1"
    kubeadmiral.io/last-applied-propagation-policy-generation: "7"       # policy generation
    kubeadmiral.io/last-applied-template-hash: "3942948866"              # template hash
    kubeadmiral.io/observed-propagation-policy-generation: "8"           # latest policy generation
    kubeadmiral.io/pending-controllers: '[]'
    kubeadmiral.io/scheduling-trigger-hash: "2359891339"
    kubeadmiral.io/syncing: '{"generation":null,"fedGeneration":14,"clusters":[{"name":"kubeadmiral-member-1","status":"OK"},{"name":"kubeadmiral-member-2","status":"OK"}]}'
    lastSyncSuccessGeneration: "14"
    syncSuccessTimestamp: "2023-06-28T09:40:38.208825293Z"
...
  • 12

    Pass the resourceQuest into schedulingUnit

    Pass the resourceQuest into schedulingUnit so that we can filter and score the cluster based on resource quest of workloads. At present, the interpret of the PodTemplate path of K8s native resources is built in, and it can be put into FTC for subsequent consideration.

    Refer to https://github.com/kubewharf/kubeadmiral/blob/7ede6f4b715d2b3861ce8a6b4f681cc229d86cfe/pkg/util/pod/pod.go#L38

  • 13

    ClusterResourcesFit does not work because cannot get resourceQuest of object

    Currently, ClusterResourcesFit filter plugin does not work because it cannot get resourceQuest of object. We need to interpret the logic of calculating resourceQuest according to GVK. We used ftc's pathDefinition to interpret replicas before, but for CRD, the process of calculating resources may be a bit complicated. We may need hook functions to allow users to customize the calculation logic of resource requests.

  • 14

    chore(deps): bump k8s.io/client-go from 0.26.6 to 0.26.7

    Bumps k8s.io/client-go from 0.26.6 to 0.26.7.

    Commits
    • 30088cd Update dependencies to v0.26.7 tag
    • cea5ee9 Merge pull request #118970champtar/automated-cherry-pick-of-#117791
    • c4c506f update serial number to a valid non-zero number in ca certificate
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 15

    chore(deps): bump k8s.io/apiextensions-apiserver from 0.26.6 to 0.26.7

    Bumps k8s.io/apiextensions-apiserver from 0.26.6 to 0.26.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 16

    chore(deps): bump k8s.io/apimachinery from 0.26.6 to 0.26.7

    Bumps k8s.io/apimachinery from 0.26.6 to 0.26.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • 17

    Clean up kubebuilder marker comments

    The APIs use a few deprecated kubebuilder marker comments, which are confusing and verbose. We should remove/replace them with currently supported alternatives.

    Ref: https://book.kubebuilder.io/migration/legacy/migration_guide_v1tov2.html

    • Popular Tag

    Related