Release Notes v1.77
23KE release notes and upgrade guideβ
Due to Gardener upstream changes some secrets in the virtual Garden cluster are not deployed via the gardener-controlplane helmchart anymore. Therefore, we took over these secrets with the 23ke internal garden-content helmchart. The affected secrets are the
default-domainsecretinternal-domainsecretopenvpn-diffie-hellman-keysecretalertingsecret
An example for the migration of these secrets with respect to the internal-domain secrete is given below:
kubectl label -n garden secret internal-domain-... helm.toolkit.fluxcd.io/name-
kubectl label -n garden secret internal-domain-... helm.toolkit.fluxcd.io/namespace-
kubectl annotate -n garden secret internal-domain-... meta.helm.sh/release-name=garden-content --overwrite
with a kubeconfig pointing to the virtual Garden cluser. Note that ... needs to be replaced by the name of the Secret in your environment.
You can print commands that do so by running this loop
export KUBECONFIG=your-vgarden-kubeconfig
for name in $(kubectl -n garden get secrets -o=name | grep 'default-domain\|internal-domain\|openvpn-diffie-hellman-key\|alerting'); do
echo kubectl label -n garden "$name" helm.toolkit.fluxcd.io/name-
echo kubectl label -n garden "$name" helm.toolkit.fluxcd.io/namespace-
echo kubectl annotate -n garden "$name" meta.helm.sh/release-name=garden-content --overwrite
done
Related upstream release notes / changelogsβ
Update gardener-controlplane to 1.76.1
[gardener/gardener]
π Bug Fixesβ
[OPERATOR]gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed. by @gardener-ci-robot [#8286]
Docker Images
admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.76.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.76.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.76.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.76.1
operator: eu.gcr.io/gardener-project/gardener/operator:v1.76.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.76.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.76.1
Update gardener-controlplane to 1.76.1
[gardener/gardener]
π Bug Fixesβ
[OPERATOR]gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed. by @gardener-ci-robot [#8286]
Docker Images
admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.76.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.76.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.76.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.76.1
operator: eu.gcr.io/gardener-project/gardener/operator:v1.76.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.76.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.76.1
Update gardenlet to 1.76.1
[gardener/gardener]
π Bug Fixesβ
[OPERATOR]gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed. by @gardener-ci-robot [#8286]
Docker Images
admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.76.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.76.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.76.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.76.1
operator: eu.gcr.io/gardener-project/gardener/operator:v1.76.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.76.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.76.1
Update gardener-controlplane to 1.76.2
[gardener/gardener]
π Bug Fixesβ
[USER]An issue has been fixed for highly-availableShoots whoseetcdclusters didn't get ready in theCompletingphase of a CA credentials rotation. by @gardener-ci-robot [#8306]
π Othersβ
[OPERATOR]A bug preventingplutonoingress to usewildcard-certificateis fixed. by @gardener-ci-robot [#8318][OPERATOR]gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed. by @gardener-ci-robot [#8314]
Docker Images
admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.76.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.76.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.76.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.76.2
operator: eu.gcr.io/gardener-project/gardener/operator:v1.76.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.76.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.76.2
Update gardener-controlplane to 1.76.2
[gardener/gardener]
π Bug Fixesβ
[USER]An issue has been fixed for highly-availableShoots whoseetcdclusters didn't get ready in theCompletingphase of a CA credentials rotation. by @gardener-ci-robot [#8306]
π Othersβ
[OPERATOR]A bug preventingplutonoingress to usewildcard-certificateis fixed. by @gardener-ci-robot [#8318][OPERATOR]gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed. by @gardener-ci-robot [#8314]
Docker Images
admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.76.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.76.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.76.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.76.2
operator: eu.gcr.io/gardener-project/gardener/operator:v1.76.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.76.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.76.2
Update gardenlet to 1.76.2
[gardener/gardener]
π Bug Fixesβ
[USER]An issue has been fixed for highly-availableShoots whoseetcdclusters didn't get ready in theCompletingphase of a CA credentials rotation. by @gardener-ci-robot [#8306]
π Othersβ
[OPERATOR]A bug preventingplutonoingress to usewildcard-certificateis fixed. by @gardener-ci-robot [#8318][OPERATOR]gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed. by @gardener-ci-robot [#8314]
Docker Images
admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.76.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.76.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.76.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.76.2
operator: eu.gcr.io/gardener-project/gardener/operator:v1.76.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.76.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.76.2
Update external-dns-management to 0.15.8
[gardener/external-dns-management]
π Othersβ
[OPERATOR]Reduce memory footprint for secrets. by @MartinWeindel [#312][OPERATOR]Bump builder image fromgolang:1.20.5togolang:1.20.7by @MartinWeindel [#312]
Update provider-azure to 1.37.1
[gardener/gardener-extension-provider-azure]
π Bug Fixesβ
[USER]The node-controller-manager is now set to keep setting deprecated node labels for k8s clusters of version>=1.26.0, <1.28.0to ensure pods using persistent volumes with node affinities are scheduled in the cluster. by @kon-angelo [#718]
Update shoot-dns-service to 1.37.0
[gardener/external-dns-management]
π Bug Fixesβ
[OPERATOR]Update controller-manager-library dependency to fix panic on api-resources discovery. by @MartinWeindel [gardener/external-dns-management#310]
π Othersβ
[OPERATOR]Bump builder image fromgolang:1.20.5togolang:1.20.7by @MartinWeindel [gardener/external-dns-management#312][OPERATOR]Reduce memory footprint for secrets. by @MartinWeindel [gardener/external-dns-management#312]
[gardener/gardener-extension-shoot-dns-service]
β οΈ Breaking Changesβ
[OPERATOR]extension-shoot-dns-serviceno longer supports Shoots with Πubernetes version < 1.22. by @shafeeqes [#218][OPERATOR]Thesecurity.gardener.cloud/pod-security-enforceannotation in the ControllerRegistration is set tobaseline. With this, the pods running in the extension namespace should comply withbaselinepod-security standard. by @shafeeqes [#222]
π Othersβ
[OPERATOR]Bumps golang from 1.20.5 to 1.21.0. by @dependabot[bot] [#227][OPERATOR]Ensure dns-controller-manager is restarted on CA rotation for remote-access server by @MartinWeindel [#223][OPERATOR]Bumps github.com/gardener/gardener from 1.75.0 to 1.76.2. by @dependabot[bot] [#228][OPERATOR]Fix set_dependency_version forworkerlessSupportedby @MartinWeindel [#226]
Update gardener-controlplane to 1.77.0
[gardener/etcd-backup-restore]
π° Noteworthyβ
[OPERATOR]Etcd-backup-restore now uses a distroless image as its base image. It is no longer compatible with etcd-custom-image, and must be used with etcd-wrapper instead. by @aaronfern [gardener/etcd-backup-restore#637][OPERATOR]Etcd-backup-restore now uses the user home directory to create files. by @aaronfern [gardener/etcd-backup-restore#637]
π Othersβ
[OPERATOR]While scaling up a non-HA etcd cluster to HA skipping the scale-up checks for first member of etcd cluster as first member can never be a part of scale-up scenarios. by @ishan16696 [gardener/etcd-backup-restore#649][OPERATOR]Backup-restore waits for its etcd to be ready before attempting to update peerUrl by @aaronfern [gardener/etcd-backup-restore#628][DEVELOPER]Add CVE categorization for etcd-backup-restore. by @shreyas-s-rao [gardener/etcd-backup-restore#644]
[gardener/gardener]
β οΈ Breaking Changesβ
[DEVELOPER]If you are usingprovider-extensionsetup you should adapt your files inexample/provider-extensions/garden/controlplanebecausedefault-domainandinternal-domainsecrets are removed fromgardener-controlplaneHelm chart. by @oliver-goetz [#8308][DEVELOPER]Packagepkg/utils/managedresourcesnow works with immutable secrets for managed resources under the hood. Existing secrets will be marked for garbage collection and replaced with immutable ones during the first reconciliation of the managed resource. by @dimityrmirchev [#8116][DEVELOPER]TheSecretstype as well as theDeletefunctions for secrets were removed frompkg/utils/managedresources/buildersince their usage was prone to errors. The higher level packagepkg/utils/managedresourcesshould be used instead. by @dimityrmirchev [#8116][DEPENDENCY]hack/generate.shhas been renamed tohack/generate-sequential.sh. by @shafeeqes [#8289][DEPENDENCY]The deprecatedextensions/pkg/controller/worker.{Options,ApplyMachineResources{ForConfig}}symbols have been dropped sincegardenlettakes over management of themachine.gardener.cloud/v1alpha1API CRDs sincegardener/gardener@v1.73. by @rfranzke [#8280][OPERATOR]Thevirtual-garden-kube-apiserverservice (for thevirtual-gardencluster) was switched from typeLoadBalancertoClusterIP. Please make sure to migrate all DNS records from thevirtual-garden-kube-apiserverto theistio-ingressgatewayendpoint before upgrading to this Gardener version. by @timuthy [#8302][OPERATOR]gardenletno longer reports theBootstrappedcondition onSeeds. Instead, it now reports the progress in.status.lastOperation, similar to how it's done forShoots. by @rfranzke [#8290][OPERATOR]default-domain,internal-domain,alertingandopenvpn-diffie-hellmansecrets are removed fromgardener-controlplaneHelm chart. Please ensure to update them in a different way before upgrading Gardener. If you would like to prevent Helm from deleting these secret during the upgrade, you could annotate them with"helm.sh/resource-policy": keep. by @oliver-goetz [#8308]
π° Noteworthyβ
[DEVELOPER]Thecharts/images.yamlfile was moved toimagevector/images.yaml. by @rfranzke [#8250][DEPENDENCY]pkg/utils/chartdoes now support embedded charts. The already deprecated methods in theChartApplierandChartRendererwill be removed in a few releases, so extensions should adapt to embedded charts. by @rfranzke [#8250][OPERATOR]Gardenlet can now set feature gates foretcd-druid. They can be specified via the gardenlet configurationGardenletConfiguration.EtcdConfig.FeatureGatesby @gardener-ci-robot [#8335]
β¨ New Featuresβ
[OPERATOR]The garbage collection controller now also considers managed resources when deciding if secrets/configmaps should be garbage collected. by @dimityrmirchev [#8116][OPERATOR]Gardener Scheduler's Minimal Distance strategy can take scheduling decisions based on region distances configured by operators. This especially improves the allocation for shoots of providers regions for which the standard Levenshtein distance is inappropriate. Please seedocs/concepts/scheduler.mdfor more information. by @timuthy [#8277][OPERATOR]Operators can now view and manage dashboards for compaction jobs running in shoot control plane. by @abdasgupta [#8206][OPERATOR]maintenance-controllernow disablesPodSecurityPolicyadmission controller when forcefully upgrading the Kubernetes version of aShoottov1.25. It also ensures maximum workers of each for group is greater or equal to its number of zone for forceful upgrades tov1.27. by @oliver-goetz [#8281][OPERATOR]kubectl get gardennow features additional printer columns providing more information about the substantial configuration values and statuses. by @rfranzke [#8279][OPERATOR]Thegardener-apiservernow drops expiredKubernetesandMachineImageversions fromCloudprofiles during creation. by @shafeeqes [#8297][OPERATOR]gardener-operatornow takes over management offluent-operatorandvali. by @vlvasilev [#8240][USER]Two additional labelsworker.gardener.cloud/image-nameandworker.gardener.cloud/image-versionare attached to worker nodes to identify which operating system they are running. This can then be used in selectors that target only workers with a specific operating system and is helpful for e.g. driver deployment. by @MrBatschner [#8295][USER]A new feature gate namedContainerdRegistryHostsDiris introduced to gardenlet. When enabled, the/etc/containerd/certs.ddirectory is created on the Node and containerd is configured to look up for registries/mirrors configuration in this directory (if there is any configuration applied). In future, the registry-cache extension will add such registries/mirrors configuration under this directory (via OperatingSystemConfig mutation). by @ialidzhikov [#8094][USER]TheShootmaintenance controller now updates the CRI of worker pools fromdockertocontainerdwhen force-upgrading from Kubernetesv1.22tov1.23. by @oliver-goetz [#8272][DEVELOPER]Extensions running on seed clusters can get access to the garden cluster by using the injected kubeconfig specified by theGARDEN_KUBECONFIGenvironment variable. You can read about the details in this doc. by @timebertt [#8264]
π Bug Fixesβ
[OPERATOR]WhenShoots were updated from non high-availability tozonehigh-availability, it could happen that the control-plane was scheduled to two instead of three zones. This issue is relevant for cloud providers with an inconsistent zone naming (Azureis currently the only candidate to our knowledge). Existing shoots with the before mentioned problem must be fixed manually be operators if required. An automatic move ofetcds and their volumes is not part of this fix due to availability reasons. by @gardener-ci-robot [#8345][OPERATOR]gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed. by @istvanballok [#8284][USER]An issue has been fixed for highly-availableShoots whoseetcdclusters didn't get ready in theCompletingphase of a CA credentials rotation. by @timuthy [#8303]
π Othersβ
[OPERATOR]A bug preventingprometheusingress to usewildcard-certificateis fixed. by @acumino [#8319][OPERATOR]A bug preventingplutonoingress to usewildcard-certificateis fixed. by @acumino [#8317][OPERATOR]gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed. by @istvanballok [#8310][DEVELOPER]Thegithub.com/golang/mock/gomockdependency is replaced bygo.uber.org/mock. by @afritzler [#8269][DEVELOPER]Add failure tolerance option to theCreateShoottest. by @hendrikKahl [#8298]
[gardener/etcd-druid]
β οΈ Breaking Changesβ
[OPERATOR]β οΈetcd.Status.ClusterSize,etcd.Status.ServiceName,etcd.Status.UpdatedReplicashave been marked as deprecated and users should refrain from depending on these fields. by @unmarshall [gardener/etcd-druid#594][OPERATOR]File ownership forvar/etcd/datawill be changed to non-root user (65532). by @aaronfern [gardener/etcd-druid#620][OPERATOR]Etcd-druid will now deploy distrolessetcd-wrapperandetcd-backup-restoreimages. Please refer to etcd-wrapper for more information. by @aaronfern [gardener/etcd-druid#620][OPERATOR]Etcd-related secrets will now be mounted onto the/var/directory instead of/root/. by @aaronfern [gardener/etcd-druid#620][DEVELOPER]Developer Action Required: Themake deploycommand has been replaced withmake deploy-via-kustomize. Please update your deployment workflows accordingly. by @seshachalam-yv [gardener/etcd-druid#599]
β¨ New Featuresβ
[DEVELOPER]Makefile has been updated to useSkaffoldfor deployingetcd-druidwith themake deploytarget, simplifying the deployment process and eliminating the need to push the image to the container registry for each local development testing. by @seshachalam-yv [gardener/etcd-druid#599][OPERATOR]Feature gates have been introduced in etcd-druid, and can be specified using CLI flag--feature-gate. by @aaronfern [gardener/etcd-druid#646][OPERATOR]Druid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance. by @abdasgupta [gardener/etcd-druid#569][OPERATOR]UseEtcdWrapperfeature gate has been introduced to allow users to opt for the new etcd-wrapper image. by @aaronfern [gardener/etcd-druid#646]
π Bug Fixesβ
[OPERATOR]A bug causing incorrect volume mount path forEtcds andEtcdCopyBackupsTasks usingLocalsnapshot storage provider while using distroless etcd-backup-restore imagev0.25.xhas been resolved. by @aaronfern [gardener/etcd-druid#662][OPERATOR]AllMembersReadycondition has now been fixed to eventually show the correct overall readiness of an etcd cluster. by @unmarshall [gardener/etcd-druid#594][OPERATOR]A bug causingEtcdCopyBackupsTaskjobs to fail to create temp snapshot directory while using distroless etcd-backup-restore imagev0.25.xhas been resolved. by @aaronfern [gardener/etcd-druid#662]
π Othersβ
[OPERATOR]Print build version and go runtime info. by @shreyas-s-rao [gardener/etcd-druid#636][OPERATOR]Bumped up the custom image version to v3.4.13-bootstrap-11 by @abdasgupta [gardener/etcd-druid#623][OPERATOR]When scaling from single-node to multi-node etcd cluster, Etcd Druid will now first ensure that any change to the peer URL (e.g TLS enablement) is seen by the existing etcd process running within the etcd member pod. Once that is confirmed then it will scale up the Etcd StatefulSet and add relevant annotations. by @unmarshall [gardener/etcd-druid#598][DEVELOPER]Refactoredstatefulset,service,poddisruptionbudget,lease, andconfigmapcomponents to use default labels and owner references frometcd. by @seshachalam-yv [gardener/etcd-druid#559][DEVELOPER]Add CVE categorization for etcd-druid. by @shreyas-s-rao [gardener/etcd-druid#634]
[gardener/vpn2]
π° Noteworthyβ
[OPERATOR]Bump builder image golang from1.20.4to1.20.6by @axel7born [gardener/vpn2#33]
[gardener/hvpa-controller]
π Bug Fixesβ
[OPERATOR]Fixed a bug that caused HVPA reconciliation to fail withexpected pointer, but got v2beta1.MetricSpec typewhen the HPA spec had changed. by @voelzmo [gardener/hvpa-controller#125]
Update gardener-controlplane to 1.77.0
[gardener/etcd-backup-restore]
π° Noteworthyβ
[OPERATOR]Etcd-backup-restore now uses a distroless image as its base image. It is no longer compatible with etcd-custom-image, and must be used with etcd-wrapper instead. by @aaronfern [gardener/etcd-backup-restore#637][OPERATOR]Etcd-backup-restore now uses the user home directory to create files. by @aaronfern [gardener/etcd-backup-restore#637]
π Othersβ
[OPERATOR]While scaling up a non-HA etcd cluster to HA skipping the scale-up checks for first member of etcd cluster as first member can never be a part of scale-up scenarios. by @ishan16696 [gardener/etcd-backup-restore#649][OPERATOR]Backup-restore waits for its etcd to be ready before attempting to update peerUrl by @aaronfern [gardener/etcd-backup-restore#628][DEVELOPER]Add CVE categorization for etcd-backup-restore. by @shreyas-s-rao [gardener/etcd-backup-restore#644]
[gardener/gardener]
β οΈ Breaking Changesβ
[DEVELOPER]If you are usingprovider-extensionsetup you should adapt your files inexample/provider-extensions/garden/controlplanebecausedefault-domainandinternal-domainsecrets are removed fromgardener-controlplaneHelm chart. by @oliver-goetz [#8308][DEVELOPER]Packagepkg/utils/managedresourcesnow works with immutable secrets for managed resources under the hood. Existing secrets will be marked for garbage collection and replaced with immutable ones during the first reconciliation of the managed resource. by @dimityrmirchev [#8116][DEVELOPER]TheSecretstype as well as theDeletefunctions for secrets were removed frompkg/utils/managedresources/buildersince their usage was prone to errors. The higher level packagepkg/utils/managedresourcesshould be used instead. by @dimityrmirchev [#8116][DEPENDENCY]hack/generate.shhas been renamed tohack/generate-sequential.sh. by @shafeeqes [#8289][DEPENDENCY]The deprecatedextensions/pkg/controller/worker.{Options,ApplyMachineResources{ForConfig}}symbols have been dropped sincegardenlettakes over management of themachine.gardener.cloud/v1alpha1API CRDs sincegardener/gardener@v1.73. by @rfranzke [#8280][OPERATOR]Thevirtual-garden-kube-apiserverservice (for thevirtual-gardencluster) was switched from typeLoadBalancertoClusterIP. Please make sure to migrate all DNS records from thevirtual-garden-kube-apiserverto theistio-ingressgatewayendpoint before upgrading to this Gardener version. by @timuthy [#8302][OPERATOR]gardenletno longer reports theBootstrappedcondition onSeeds. Instead, it now reports the progress in.status.lastOperation, similar to how it's done forShoots. by @rfranzke [#8290][OPERATOR]default-domain,internal-domain,alertingandopenvpn-diffie-hellmansecrets are removed fromgardener-controlplaneHelm chart. Please ensure to update them in a different way before upgrading Gardener. If you would like to prevent Helm from deleting these secret during the upgrade, you could annotate them with"helm.sh/resource-policy": keep. by @oliver-goetz [#8308]
π° Noteworthyβ
[DEVELOPER]Thecharts/images.yamlfile was moved toimagevector/images.yaml. by @rfranzke [#8250][DEPENDENCY]pkg/utils/chartdoes now support embedded charts. The already deprecated methods in theChartApplierandChartRendererwill be removed in a few releases, so extensions should adapt to embedded charts. by @rfranzke [#8250][OPERATOR]Gardenlet can now set feature gates foretcd-druid. They can be specified via the gardenlet configurationGardenletConfiguration.EtcdConfig.FeatureGatesby @gardener-ci-robot [#8335]
β¨ New Featuresβ
[OPERATOR]The garbage collection controller now also considers managed resources when deciding if secrets/configmaps should be garbage collected. by @dimityrmirchev [#8116][OPERATOR]Gardener Scheduler's Minimal Distance strategy can take scheduling decisions based on region distances configured by operators. This especially improves the allocation for shoots of providers regions for which the standard Levenshtein distance is inappropriate. Please seedocs/concepts/scheduler.mdfor more information. by @timuthy [#8277][OPERATOR]Operators can now view and manage dashboards for compaction jobs running in shoot control plane. by @abdasgupta [#8206][OPERATOR]maintenance-controllernow disablesPodSecurityPolicyadmission controller when forcefully upgrading the Kubernetes version of aShoottov1.25. It also ensures maximum workers of each for group is greater or equal to its number of zone for forceful upgrades tov1.27. by @oliver-goetz [#8281][OPERATOR]kubectl get gardennow features additional printer columns providing more information about the substantial configuration values and statuses. by @rfranzke [#8279][OPERATOR]Thegardener-apiservernow drops expiredKubernetesandMachineImageversions fromCloudprofiles during creation. by @shafeeqes [#8297][OPERATOR]gardener-operatornow takes over management offluent-operatorandvali. by @vlvasilev [#8240][USER]Two additional labelsworker.gardener.cloud/image-nameandworker.gardener.cloud/image-versionare attached to worker nodes to identify which operating system they are running. This can then be used in selectors that target only workers with a specific operating system and is helpful for e.g. driver deployment. by @MrBatschner [#8295][USER]A new feature gate namedContainerdRegistryHostsDiris introduced to gardenlet. When enabled, the/etc/containerd/certs.ddirectory is created on the Node and containerd is configured to look up for registries/mirrors configuration in this directory (if there is any configuration applied). In future, the registry-cache extension will add such registries/mirrors configuration under this directory (via OperatingSystemConfig mutation). by @ialidzhikov [#8094][USER]TheShootmaintenance controller now updates the CRI of worker pools fromdockertocontainerdwhen force-upgrading from Kubernetesv1.22tov1.23. by @oliver-goetz [#8272][DEVELOPER]Extensions running on seed clusters can get access to the garden cluster by using the injected kubeconfig specified by theGARDEN_KUBECONFIGenvironment variable. You can read about the details in this doc. by @timebertt [#8264]
π Bug Fixesβ
[OPERATOR]WhenShoots were updated from non high-availability tozonehigh-availability, it could happen that the control-plane was scheduled to two instead of three zones. This issue is relevant for cloud providers with an inconsistent zone naming (Azureis currently the only candidate to our knowledge). Existing shoots with the before mentioned problem must be fixed manually be operators if required. An automatic move ofetcds and their volumes is not part of this fix due to availability reasons. by @gardener-ci-robot [#8345][OPERATOR]gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed. by @istvanballok [#8284][USER]An issue has been fixed for highly-availableShoots whoseetcdclusters didn't get ready in theCompletingphase of a CA credentials rotation. by @timuthy [#8303]
π Othersβ
[OPERATOR]A bug preventingprometheusingress to usewildcard-certificateis fixed. by @acumino [#8319][OPERATOR]A bug preventingplutonoingress to usewildcard-certificateis fixed. by @acumino [#8317][OPERATOR]gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed. by @istvanballok [#8310][DEVELOPER]Thegithub.com/golang/mock/gomockdependency is replaced bygo.uber.org/mock. by @afritzler [#8269][DEVELOPER]Add failure tolerance option to theCreateShoottest. by @hendrikKahl [#8298]
[gardener/etcd-druid]
β οΈ Breaking Changesβ
[OPERATOR]β οΈetcd.Status.ClusterSize,etcd.Status.ServiceName,etcd.Status.UpdatedReplicashave been marked as deprecated and users should refrain from depending on these fields. by @unmarshall [gardener/etcd-druid#594][OPERATOR]File ownership forvar/etcd/datawill be changed to non-root user (65532). by @aaronfern [gardener/etcd-druid#620][OPERATOR]Etcd-druid will now deploy distrolessetcd-wrapperandetcd-backup-restoreimages. Please refer to etcd-wrapper for more information. by @aaronfern [gardener/etcd-druid#620][OPERATOR]Etcd-related secrets will now be mounted onto the/var/directory instead of/root/. by @aaronfern [gardener/etcd-druid#620][DEVELOPER]Developer Action Required: Themake deploycommand has been replaced withmake deploy-via-kustomize. Please update your deployment workflows accordingly. by @seshachalam-yv [gardener/etcd-druid#599]
β¨ New Featuresβ
[DEVELOPER]Makefile has been updated to useSkaffoldfor deployingetcd-druidwith themake deploytarget, simplifying the deployment process and eliminating the need to push the image to the container registry for each local development testing. by @seshachalam-yv [gardener/etcd-druid#599][OPERATOR]Feature gates have been introduced in etcd-druid, and can be specified using CLI flag--feature-gate. by @aaronfern [gardener/etcd-druid#646][OPERATOR]Druid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance. by @abdasgupta [gardener/etcd-druid#569][OPERATOR]UseEtcdWrapperfeature gate has been introduced to allow users to opt for the new etcd-wrapper image. by @aaronfern [gardener/etcd-druid#646]
π Bug Fixesβ
[OPERATOR]A bug causing incorrect volume mount path forEtcds andEtcdCopyBackupsTasks usingLocalsnapshot storage provider while using distroless etcd-backup-restore imagev0.25.xhas been resolved. by @aaronfern [gardener/etcd-druid#662][OPERATOR]AllMembersReadycondition has now been fixed to eventually show the correct overall readiness of an etcd cluster. by @unmarshall [gardener/etcd-druid#594][OPERATOR]A bug causingEtcdCopyBackupsTaskjobs to fail to create temp snapshot directory while using distroless etcd-backup-restore imagev0.25.xhas been resolved. by @aaronfern [gardener/etcd-druid#662]
π Othersβ
[OPERATOR]Print build version and go runtime info. by @shreyas-s-rao [gardener/etcd-druid#636][OPERATOR]Bumped up the custom image version to v3.4.13-bootstrap-11 by @abdasgupta [gardener/etcd-druid#623][OPERATOR]When scaling from single-node to multi-node etcd cluster, Etcd Druid will now first ensure that any change to the peer URL (e.g TLS enablement) is seen by the existing etcd process running within the etcd member pod. Once that is confirmed then it will scale up the Etcd StatefulSet and add relevant annotations. by @unmarshall [gardener/etcd-druid#598][DEVELOPER]Refactoredstatefulset,service,poddisruptionbudget,lease, andconfigmapcomponents to use default labels and owner references frometcd. by @seshachalam-yv [gardener/etcd-druid#559][DEVELOPER]Add CVE categorization for etcd-druid. by @shreyas-s-rao [gardener/etcd-druid#634]
[gardener/vpn2]
π° Noteworthyβ
[OPERATOR]Bump builder image golang from1.20.4to1.20.6by @axel7born [gardener/vpn2#33]
[gardener/hvpa-controller]
π Bug Fixesβ
[OPERATOR]Fixed a bug that caused HVPA reconciliation to fail withexpected pointer, but got v2beta1.MetricSpec typewhen the HPA spec had changed. by @voelzmo [gardener/hvpa-controller#125]
Update gardenlet to 1.77.0
[gardener/etcd-backup-restore]
π° Noteworthyβ
[OPERATOR]Etcd-backup-restore now uses a distroless image as its base image. It is no longer compatible with etcd-custom-image, and must be used with etcd-wrapper instead. by @aaronfern [gardener/etcd-backup-restore#637][OPERATOR]Etcd-backup-restore now uses the user home directory to create files. by @aaronfern [gardener/etcd-backup-restore#637]
π Othersβ
[OPERATOR]While scaling up a non-HA etcd cluster to HA skipping the scale-up checks for first member of etcd cluster as first member can never be a part of scale-up scenarios. by @ishan16696 [gardener/etcd-backup-restore#649][OPERATOR]Backup-restore waits for its etcd to be ready before attempting to update peerUrl by @aaronfern [gardener/etcd-backup-restore#628][DEVELOPER]Add CVE categorization for etcd-backup-restore. by @shreyas-s-rao [gardener/etcd-backup-restore#644]
[gardener/gardener]
β οΈ Breaking Changesβ
[DEVELOPER]If you are usingprovider-extensionsetup you should adapt your files inexample/provider-extensions/garden/controlplanebecausedefault-domainandinternal-domainsecrets are removed fromgardener-controlplaneHelm chart. by @oliver-goetz [#8308][DEVELOPER]Packagepkg/utils/managedresourcesnow works with immutable secrets for managed resources under the hood. Existing secrets will be marked for garbage collection and replaced with immutable ones during the first reconciliation of the managed resource. by @dimityrmirchev [#8116][DEVELOPER]TheSecretstype as well as theDeletefunctions for secrets were removed frompkg/utils/managedresources/buildersince their usage was prone to errors. The higher level packagepkg/utils/managedresourcesshould be used instead. by @dimityrmirchev [#8116][DEPENDENCY]hack/generate.shhas been renamed tohack/generate-sequential.sh. by @shafeeqes [#8289][DEPENDENCY]The deprecatedextensions/pkg/controller/worker.{Options,ApplyMachineResources{ForConfig}}symbols have been dropped sincegardenlettakes over management of themachine.gardener.cloud/v1alpha1API CRDs sincegardener/gardener@v1.73. by @rfranzke [#8280][OPERATOR]Thevirtual-garden-kube-apiserverservice (for thevirtual-gardencluster) was switched from typeLoadBalancertoClusterIP. Please make sure to migrate all DNS records from thevirtual-garden-kube-apiserverto theistio-ingressgatewayendpoint before upgrading to this Gardener version. by @timuthy [#8302][OPERATOR]gardenletno longer reports theBootstrappedcondition onSeeds. Instead, it now reports the progress in.status.lastOperation, similar to how it's done forShoots. by @rfranzke [#8290][OPERATOR]default-domain,internal-domain,alertingandopenvpn-diffie-hellmansecrets are removed fromgardener-controlplaneHelm chart. Please ensure to update them in a different way before upgrading Gardener. If you would like to prevent Helm from deleting these secret during the upgrade, you could annotate them with"helm.sh/resource-policy": keep. by @oliver-goetz [#8308]
π° Noteworthyβ
[DEVELOPER]Thecharts/images.yamlfile was moved toimagevector/images.yaml. by @rfranzke [#8250][DEPENDENCY]pkg/utils/chartdoes now support embedded charts. The already deprecated methods in theChartApplierandChartRendererwill be removed in a few releases, so extensions should adapt to embedded charts. by @rfranzke [#8250][OPERATOR]Gardenlet can now set feature gates foretcd-druid. They can be specified via the gardenlet configurationGardenletConfiguration.EtcdConfig.FeatureGatesby @gardener-ci-robot [#8335]
β¨ New Featuresβ
[OPERATOR]The garbage collection controller now also considers managed resources when deciding if secrets/configmaps should be garbage collected. by @dimityrmirchev [#8116][OPERATOR]Gardener Scheduler's Minimal Distance strategy can take scheduling decisions based on region distances configured by operators. This especially improves the allocation for shoots of providers regions for which the standard Levenshtein distance is inappropriate. Please seedocs/concepts/scheduler.mdfor more information. by @timuthy [#8277][OPERATOR]Operators can now view and manage dashboards for compaction jobs running in shoot control plane. by @abdasgupta [#8206][OPERATOR]maintenance-controllernow disablesPodSecurityPolicyadmission controller when forcefully upgrading the Kubernetes version of aShoottov1.25. It also ensures maximum workers of each for group is greater or equal to its number of zone for forceful upgrades tov1.27. by @oliver-goetz [#8281][OPERATOR]kubectl get gardennow features additional printer columns providing more information about the substantial configuration values and statuses. by @rfranzke [#8279][OPERATOR]Thegardener-apiservernow drops expiredKubernetesandMachineImageversions fromCloudprofiles during creation. by @shafeeqes [#8297][OPERATOR]gardener-operatornow takes over management offluent-operatorandvali. by @vlvasilev [#8240][USER]Two additional labelsworker.gardener.cloud/image-nameandworker.gardener.cloud/image-versionare attached to worker nodes to identify which operating system they are running. This can then be used in selectors that target only workers with a specific operating system and is helpful for e.g. driver deployment. by @MrBatschner [#8295][USER]A new feature gate namedContainerdRegistryHostsDiris introduced to gardenlet. When enabled, the/etc/containerd/certs.ddirectory is created on the Node and containerd is configured to look up for registries/mirrors configuration in this directory (if there is any configuration applied). In future, the registry-cache extension will add such registries/mirrors configuration under this directory (via OperatingSystemConfig mutation). by @ialidzhikov [#8094][USER]TheShootmaintenance controller now updates the CRI of worker pools fromdockertocontainerdwhen force-upgrading from Kubernetesv1.22tov1.23. by @oliver-goetz [#8272][DEVELOPER]Extensions running on seed clusters can get access to the garden cluster by using the injected kubeconfig specified by theGARDEN_KUBECONFIGenvironment variable. You can read about the details in this doc. by @timebertt [#8264]
π Bug Fixesβ
[OPERATOR]WhenShoots were updated from non high-availability tozonehigh-availability, it could happen that the control-plane was scheduled to two instead of three zones. This issue is relevant for cloud providers with an inconsistent zone naming (Azureis currently the only candidate to our knowledge). Existing shoots with the before mentioned problem must be fixed manually be operators if required. An automatic move ofetcds and their volumes is not part of this fix due to availability reasons. by @gardener-ci-robot [#8345][OPERATOR]gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed. by @istvanballok [#8284][USER]An issue has been fixed for highly-availableShoots whoseetcdclusters didn't get ready in theCompletingphase of a CA credentials rotation. by @timuthy [#8303]
π Othersβ
[OPERATOR]A bug preventingprometheusingress to usewildcard-certificateis fixed. by @acumino [#8319][OPERATOR]A bug preventingplutonoingress to usewildcard-certificateis fixed. by @acumino [#8317][OPERATOR]gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed. by @istvanballok [#8310][DEVELOPER]Thegithub.com/golang/mock/gomockdependency is replaced bygo.uber.org/mock. by @afritzler [#8269][DEVELOPER]Add failure tolerance option to theCreateShoottest. by @hendrikKahl [#8298]
[gardener/etcd-druid]
β οΈ Breaking Changesβ
[OPERATOR]β οΈetcd.Status.ClusterSize,etcd.Status.ServiceName,etcd.Status.UpdatedReplicashave been marked as deprecated and users should refrain from depending on these fields. by @unmarshall [gardener/etcd-druid#594][OPERATOR]File ownership forvar/etcd/datawill be changed to non-root user (65532). by @aaronfern [gardener/etcd-druid#620][OPERATOR]Etcd-druid will now deploy distrolessetcd-wrapperandetcd-backup-restoreimages. Please refer to etcd-wrapper for more information. by @aaronfern [gardener/etcd-druid#620][OPERATOR]Etcd-related secrets will now be mounted onto the/var/directory instead of/root/. by @aaronfern [gardener/etcd-druid#620][DEVELOPER]Developer Action Required: Themake deploycommand has been replaced withmake deploy-via-kustomize. Please update your deployment workflows accordingly. by @seshachalam-yv [gardener/etcd-druid#599]
β¨ New Featuresβ
[DEVELOPER]Makefile has been updated to useSkaffoldfor deployingetcd-druidwith themake deploytarget, simplifying the deployment process and eliminating the need to push the image to the container registry for each local development testing. by @seshachalam-yv [gardener/etcd-druid#599][OPERATOR]Feature gates have been introduced in etcd-druid, and can be specified using CLI flag--feature-gate. by @aaronfern [gardener/etcd-druid#646][OPERATOR]Druid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance. by @abdasgupta [gardener/etcd-druid#569][OPERATOR]UseEtcdWrapperfeature gate has been introduced to allow users to opt for the new etcd-wrapper image. by @aaronfern [gardener/etcd-druid#646]
π Bug Fixesβ
[OPERATOR]A bug causing incorrect volume mount path forEtcds andEtcdCopyBackupsTasks usingLocalsnapshot storage provider while using distroless etcd-backup-restore imagev0.25.xhas been resolved. by @aaronfern [gardener/etcd-druid#662][OPERATOR]AllMembersReadycondition has now been fixed to eventually show the correct overall readiness of an etcd cluster. by @unmarshall [gardener/etcd-druid#594][OPERATOR]A bug causingEtcdCopyBackupsTaskjobs to fail to create temp snapshot directory while using distroless etcd-backup-restore imagev0.25.xhas been resolved. by @aaronfern [gardener/etcd-druid#662]
π Othersβ
[OPERATOR]Print build version and go runtime info. by @shreyas-s-rao [gardener/etcd-druid#636][OPERATOR]Bumped up the custom image version to v3.4.13-bootstrap-11 by @abdasgupta [gardener/etcd-druid#623][OPERATOR]When scaling from single-node to multi-node etcd cluster, Etcd Druid will now first ensure that any change to the peer URL (e.g TLS enablement) is seen by the existing etcd process running within the etcd member pod. Once that is confirmed then it will scale up the Etcd StatefulSet and add relevant annotations. by @unmarshall [gardener/etcd-druid#598][DEVELOPER]Refactoredstatefulset,service,poddisruptionbudget,lease, andconfigmapcomponents to use default labels and owner references frometcd. by @seshachalam-yv [gardener/etcd-druid#559][DEVELOPER]Add CVE categorization for etcd-druid. by @shreyas-s-rao [gardener/etcd-druid#634]
[gardener/vpn2]
π° Noteworthyβ
[OPERATOR]Bump builder image golang from1.20.4to1.20.6by @axel7born [gardener/vpn2#33]
[gardener/hvpa-controller]
π Bug Fixesβ
[OPERATOR]Fixed a bug that caused HVPA reconciliation to fail withexpected pointer, but got v2beta1.MetricSpec typewhen the HPA spec had changed. by @voelzmo [gardener/hvpa-controller#125]