Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread #1330

dgrisonnet · 2021-08-20T07:27:00Z

We are now setting the upgradeable status based on whether the workflows
with PVCs are correctly spread between multiple nodes in HA topology.
This will prevent 4.8 clusters with soft-affinity on hostname to
encounter scheduling issues when updating to 4.9 where hard-affinity is
defined on hostname.

I completely decoupled setting the Upgradeable status from the usual
status flow and it is now handled completely at the end of the sync.

I added CHANGELOG entry for this change.
No user facing changes, so no entry in CHANGELOG was needed.

pkg/operator/operator.go

openshift-ci · 2021-08-20T07:40:51Z

@dgrisonnet: This pull request references Bugzilla bug 1995924, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @juzhao

In response to this:

Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

simonpasquier · 2021-08-24T12:47:34Z

/test e2e-agnostic-upgrade

simonpasquier · 2021-08-24T14:08:12Z

/test e2e-agnostic

simonpasquier · 2021-08-24T15:51:43Z

/test e2e-agnostic-operator

simonpasquier · 2021-08-25T07:35:28Z

/test e2e-agnostic
/hold
waiting on openshift/release#21258 to avoid failing CI jobs because CMO might go Upgradeable: false. As an example, this run of e2e-agnostic failed because of this.

Some cluster operators are not ready: monitoring (Upgradeable=False HighlyAvailableWorkloadIncorrectlySpread: Highly-available workload openshift-monitoring/prometheus-k8s is incorrectly spread across multiple nodes
Manual intervention is needed to upgrade to the next minor version. Please refer to the following documentation to fix this issue: https://github.com/openshift/runbooks/blob/master/alerts/HighlyAvailableWorkloadIncorrectlySpread.md.)

Indeed the prometheus-k8s pods were all scheduled on the same node. From oc get pods -A:

openshift-monitoring prometheus-k8s-0 7/7 Running [...] ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-99rqg
openshift-monitoring prometheus-k8s-1 7/7 Running [...] ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-99rqg

Correlating from the kube-scheduler logs, it seems that when the prometheus operator provisions the prometheus-k8s statefulset, ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-99rqg is the only node being ready for scheduling.

I0824 14:52:23.590898       1 node_tree.go:65] Added node "ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-99rqg" in group "westus:\x00:0" to NodeTree
[...]
I0824 14:54:56.185289       1 scheduler.go:672] "Successfully bound pod to node" pod="openshift-monitoring/prometheus-k8s-0" node="ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-99rqg" evaluatedNodes=4 feasibleNodes=1
I0824 14:54:56.234721       1 scheduler.go:672] "Successfully bound pod to node" pod="openshift-monitoring/prometheus-k8s-1" node="ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-99rqg" evaluatedNodes=4 feasibleNodes=1
[...]
I0824 14:57:18.790086       1 node_tree.go:65] Added node "ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-mr8j9" in group "westus:\x00:0" to NodeTree
[...]
I0824 14:57:18.951128       1 node_tree.go:65] Added node "ci-op-gzbktzw9-3cef8-jtnd7-worker-westus-kjp7m" in group "westus:\x00:0" to NodeTree

simonpasquier · 2021-08-25T15:49:33Z

/skip

simonpasquier · 2021-08-25T15:49:38Z

/retest

simonpasquier · 2021-08-26T07:22:54Z

/test e2e-agnostic

jan--f · 2021-08-30T13:35:50Z

/lgtm

pkg/operator/operator.go

simonpasquier · 2021-10-11T14:29:55Z

/lgtm

openshift-ci · 2021-10-11T14:32:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, jan--f, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dgrisonnet,jan--f,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2021-10-11T15:43:34Z

@dgrisonnet: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-single-node	`d15cb46`	link	false	`/test e2e-aws-single-node`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dgrisonnet · 2021-10-11T19:37:45Z

/retest-required

dgrisonnet · 2021-10-11T19:38:30Z

Unholding since openshift/release#21564 has been merged.

/unhold

openshift-bot · 2021-10-11T19:38:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-11T20:43:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-11T21:09:35Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-10-11T21:34:09Z

@dgrisonnet: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#1330

Bugzilla bug 1995924 has been moved to the MODIFIED state.

In response to this:

Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…ble" This reverts commit a62b8fe, reversing changes made to d5dfc39.

Bug 1995924: Revert "Merge pull request #1330 from dgrisonnet/ha-upgradeable"

Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread

… spread (#1431) * Merge pull request #1330 from dgrisonnet/ha-upgradeable Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread * pkg/rebalancer: sort resources for deletion The resources that are marked for deletion are sorted by their PVC creation timestamp, from the newest to the oldest to make the deletion consistent. Signed-off-by: Damien Grisonnet <[email protected]> * pkg: improve rebalancer logging Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: sort based on pod names Sort PVCs by their creation timestamps, from the newest to the oldest to make sure that the oldest PVC is retained in case all of them are annotated. If some PVCs have the same creation timestamp, they will be sorted based on their pod name. Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: split annotation removal Add EnsurePVCsAreNotAnnoted function that makes sure that none of the PVCs of the given workload have the openshift.io/cluster-monitoring-drop-pvc annotation after the rebalancing is done. In case one of the PVC has the annotation, it will be removed to prevent deleting the PVC in a future cycle. Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: sort resources to delete by age Signed-off-by: Damien Grisonnet <[email protected]> * test/e2e: fix framework Signed-off-by: Damien Grisonnet <[email protected]> Co-authored-by: OpenShift Merge Robot <[email protected]>

… spread (openshift#1431) * Merge pull request openshift#1330 from dgrisonnet/ha-upgradeable Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread * pkg/rebalancer: sort resources for deletion The resources that are marked for deletion are sorted by their PVC creation timestamp, from the newest to the oldest to make the deletion consistent. Signed-off-by: Damien Grisonnet <[email protected]> * pkg: improve rebalancer logging Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: sort based on pod names Sort PVCs by their creation timestamps, from the newest to the oldest to make sure that the oldest PVC is retained in case all of them are annotated. If some PVCs have the same creation timestamp, they will be sorted based on their pod name. Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: split annotation removal Add EnsurePVCsAreNotAnnoted function that makes sure that none of the PVCs of the given workload have the openshift.io/cluster-monitoring-drop-pvc annotation after the rebalancing is done. In case one of the PVC has the annotation, it will be removed to prevent deleting the PVC in a future cycle. Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: sort resources to delete by age Signed-off-by: Damien Grisonnet <[email protected]> * test/e2e: fix framework Signed-off-by: Damien Grisonnet <[email protected]> Co-authored-by: OpenShift Merge Robot <[email protected]> (cherry picked from commit 8500e0f)

…ble" This reverts commit a62b8fe, reversing changes made to d5dfc39.

… spread (openshift#1431) * Merge pull request openshift#1330 from dgrisonnet/ha-upgradeable Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread * pkg/rebalancer: sort resources for deletion The resources that are marked for deletion are sorted by their PVC creation timestamp, from the newest to the oldest to make the deletion consistent. Signed-off-by: Damien Grisonnet <[email protected]> * pkg: improve rebalancer logging Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: sort based on pod names Sort PVCs by their creation timestamps, from the newest to the oldest to make sure that the oldest PVC is retained in case all of them are annotated. If some PVCs have the same creation timestamp, they will be sorted based on their pod name. Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: split annotation removal Add EnsurePVCsAreNotAnnoted function that makes sure that none of the PVCs of the given workload have the openshift.io/cluster-monitoring-drop-pvc annotation after the rebalancing is done. In case one of the PVC has the annotation, it will be removed to prevent deleting the PVC in a future cycle. Signed-off-by: Damien Grisonnet <[email protected]> * pkg/rebalancer: sort resources to delete by age Signed-off-by: Damien Grisonnet <[email protected]> * test/e2e: fix framework Signed-off-by: Damien Grisonnet <[email protected]> Co-authored-by: OpenShift Merge Robot <[email protected]>

dgrisonnet commented Aug 20, 2021

View reviewed changes

pkg/operator/operator.go Outdated Show resolved Hide resolved

openshift-ci bot requested review from arajkumar and fpetkovski August 20, 2021 07:28

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2021

dgrisonnet changed the title ~~Set Upgradeable: false when HA workloads are incorrectly spread~~ Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread Aug 20, 2021

openshift-ci bot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Aug 20, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Aug 20, 2021

openshift-ci bot requested a review from juzhao August 20, 2021 07:40

dgrisonnet force-pushed the ha-upgradeable branch from 2eafad1 to c00189a Compare August 20, 2021 08:33

dgrisonnet mentioned this pull request Aug 20, 2021

Rebalance workloads before running the upgrades tests openshift/release#21258

Closed

simonpasquier force-pushed the ha-upgradeable branch 2 times, most recently from fbc7374 to d2b5a64 Compare August 24, 2021 12:38

simonpasquier mentioned this pull request Aug 24, 2021

Bug 1933847: enable hard affinity + PodDisruptionBudget for Prometheus and Thanos Ruler pods #1341

Merged

2 tasks

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 25, 2021

simonpasquier force-pushed the ha-upgradeable branch from d2b5a64 to 0817169 Compare August 25, 2021 10:05

openshift-ci bot assigned jan--f Aug 30, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 30, 2021

dgrisonnet commented Sep 1, 2021

View reviewed changes

pkg/operator/operator.go Outdated Show resolved Hide resolved

openshift-ci bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed lgtm Indicates that a PR is ready to be merged. labels Sep 1, 2021

dgrisonnet force-pushed the ha-upgradeable branch from c0a06d5 to 94372a1 Compare September 2, 2021 14:53

dgrisonnet force-pushed the ha-upgradeable branch from 3768c5d to d15cb46 Compare October 11, 2021 14:28

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 11, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 11, 2021

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 11, 2021

openshift-merge-robot merged commit a62b8fe into openshift:master Oct 11, 2021

dgrisonnet deleted the ha-upgradeable branch October 12, 2021 08:37

dgrisonnet added a commit to dgrisonnet/cluster-monitoring-operator that referenced this pull request Oct 12, 2021

Revert "Merge pull request openshift#1330 from dgrisonnet/ha-upgradea…

a702102

…ble" This reverts commit a62b8fe, reversing changes made to d5dfc39.

openshift-ci bot mentioned this pull request Oct 12, 2021

Bug 1995924: Revert "Merge pull request #1330 from dgrisonnet/ha-upgradeable" #1425

Merged

2 tasks

openshift-merge-robot added a commit that referenced this pull request Oct 12, 2021

Merge pull request #1425 from dgrisonnet/revert-1330

f766545

Bug 1995924: Revert "Merge pull request #1330 from dgrisonnet/ha-upgradeable"

dgrisonnet mentioned this pull request Oct 12, 2021

Annotate monitoring PVCs for deletion before running the upgrade tests openshift/release#22659

Merged

dgrisonnet pushed a commit to dgrisonnet/cluster-monitoring-operator that referenced this pull request Oct 13, 2021

Merge pull request openshift#1330 from dgrisonnet/ha-upgradeable

d758788

Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread

dgrisonnet mentioned this pull request Oct 13, 2021

Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread #1431

Merged

2 tasks

dgrisonnet pushed a commit to dgrisonnet/cluster-monitoring-operator that referenced this pull request Oct 15, 2021

Merge pull request openshift#1330 from dgrisonnet/ha-upgradeable

b396fca

Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread

simonpasquier pushed a commit to dgrisonnet/cluster-monitoring-operator that referenced this pull request Oct 28, 2021

Merge pull request openshift#1330 from dgrisonnet/ha-upgradeable

9992cb7

Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread

jan--f mentioned this pull request Nov 8, 2021

Bug 2021097: Set Upgradeable: false when HA workloads are incorrectly spread #1472

Merged

2 tasks

dgrisonnet added a commit to dgrisonnet/cluster-monitoring-operator that referenced this pull request Sep 5, 2023

Revert "Merge pull request openshift#1330 from dgrisonnet/ha-upgradea…

027e1eb

…ble" This reverts commit a62b8fe, reversing changes made to d5dfc39.

Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread #1330

Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread #1330

Uh oh!

Conversation

dgrisonnet commented Aug 20, 2021

Uh oh!

Uh oh!

openshift-ci bot commented Aug 20, 2021

Uh oh!

simonpasquier commented Aug 24, 2021

Uh oh!

simonpasquier commented Aug 24, 2021

Uh oh!

simonpasquier commented Aug 24, 2021

Uh oh!

simonpasquier commented Aug 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier commented Aug 25, 2021

Uh oh!

simonpasquier commented Aug 25, 2021

Uh oh!

simonpasquier commented Aug 26, 2021

Uh oh!

jan--f commented Aug 30, 2021

Uh oh!

Uh oh!

simonpasquier commented Oct 11, 2021

Uh oh!

openshift-ci bot commented Oct 11, 2021

Uh oh!

openshift-ci bot commented Oct 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dgrisonnet commented Oct 11, 2021

Uh oh!

dgrisonnet commented Oct 11, 2021

Uh oh!

openshift-bot commented Oct 11, 2021

Uh oh!

openshift-bot commented Oct 11, 2021

Uh oh!

openshift-bot commented Oct 11, 2021

Uh oh!

openshift-ci bot commented Oct 11, 2021

Uh oh!

Uh oh!

Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread #1330

Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread #1330

simonpasquier commented Aug 25, 2021 •

edited

Loading

openshift-ci bot commented Oct 11, 2021 •

edited

Loading