Garbage collector behavior on invalid ownerReferences for existing uids across namespaces and across kinds is non-deterministic

[caesarxuchao]

If an object with a given uid is already in the garbage collector uid map, a child object created with an ownerReference pointing to that uid is not treated as having a non-existent parent, even if:

1. the ownerReference refers to a cluster-scoped kind and no cluster-scoped kind with that name exists
2. the child is namespaced, and the ownerReference refers to a namespaced kind and no namespace-scoped kind with that name exists in the same namespace
3. the child is cluster-scoped, and the ownerReference refers to a namespaced kind (since the namespace of the parent is ambiguous, it should be treated as not existing)

Original description follows

Forked from #63386 (comment).

Garbage collector should work for three cases: (a) cluster-scoped owner with namespaced dependents, (b) namespaced owner and namespaced dependents that are in the same namespace, and (c) cluster-scoped owner with cluster-scoped dependents.

Garbage collector should NOT work for the other two cases: (c) namespaced owner with cluster-scoped dependents. (d) owner and dependents that are in different namespaces. Today, GC sometimes work in these two cases. It's a bug for two reasons:

• This weakens our security guarantees. Users might exploit this bug to grant other users delete permissions.
• Users might rely on the unsupported behavior.

We can add extra checks in the GC controller to make it never work for case (c) and (d).

cc @lavalamp @liggitt @deads2k

[tpepper]

The discussion in #63386 is unclear to me. While that is 1.11 milestone material, is the content of this issue here a lower priority and deferred to the next milestone and perhaps later cherry-picking? Given the delay, I'd like to assume it is not critical-urgent, but rather than assume...

@kubernetes/sig-api-machinery-bugs can folks comment?

[liggitt]

This is longstanding and not a 1.11 blocker.

[wenjiaswe]

/cc @janetkuo @roycaihw

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[caesarxuchao]

/remove-lifecycle stale
/lifecycle frozen

[roytman]

I work with K8s 1.13 and observed that GC works when

(d) owner and dependents that are in different namespaces

If required, I can provide a simple test example.

[liggitt]

I work with K8s 1.13 and observed that GC works when

(d) owner and dependents that are in different namespaces

It does not work reliably. If you set up a relationship like that, then restart the controller manager, the dependent will usually be deleted (because the uid-indexed owner cache is not populated yet)

[roytman]

I mean that in my case the dependent is deleted without restart the controller manager. It is quite reproducible.

[liggitt]

I mean that in my case the dependent is deleted without restart the controller manager. It is quite reproducible.

That is the expected/desired behavior, but is racy, so sometimes the cross-namespace relationship does not result in deletion until restart. Because it is racy, environmental factors like load and performance (both of the API server and controller manager) can affect results.

[roytman]

I see, thanks

[liggitt]

xref #72192

[tedyu]

In GarbageCollector#attemptToDeleteItem :

		for _, dep := range deps {
			if dep.isDeletingDependents() {
...
				if _, err := gc.patch(item, patch, gc.unblockOwnerReferencesJSONMergePatch); err != nil {
					return err
				}
				break

I think we should continue with other dependents performing the unblocking (instead of breaking out of the loop).

[liggitt]

I'm working on a fix for this for the 1.20 timeframe

[sayanchowdhury]

👋🏽 Hey @liggitt ! I'm from the Bug Triage team. This issue has not been updated for a while, so I'd like to check on the status. The code freeze is starting November 12th (about 3 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time.

As the issue is tagged for 1.20, is it still planned for this release?

[liggitt]

Yes, work on this is in progress for 1.20

[liggitt]

Fixed in 1.20 by #92743