AudioNode Lifetime section seems to attempt to make garbage collection observable

[karlt]

Attempting to describe observable behavior on garbage collection wouldn't make
sense because garbage collection is the removal of objects that are no longer
needed. If their removal would cause changes in behaviour, then that implies
that the objects are still needed.

https://w3ctag.github.io/design-principles/#js-gc states the requirement in
reverse: "There must not be a way for author code to deduce when/if garbage
collection of JavaScript objects has run."

https://webaudio.github.io/web-audio-api/#lifetime-AudioNode
describes conditions that would keep an AudioNode alive and
the deletion of AudioNodes when none of these conditions are present.

The section appears to be describing garbage collection. The first condition
is "A normal reference obeying normal garbage collection rules". I'm not
clear exactly what this means. Garbage collection attempts to detect objects
that are no longer needed. Typically, implementations ensure that objects
that are still needed have "normal references" reachable from roots.

All other "types of references" listed seem to be explicit conditions where the
AudioNode is still needed.

One situation that is not explicitly described is when an AudioNode has
connections on its outputs. In this situation non-unit output channel counts
can affect downstream input mixing, and the behavior of some kinds of
downstream nodes (e.g. AudioWorkletNode) depends on whether or not these
connections exist. An implementation however is able to consider an otherwise
unneeded upstream AudioNode as unneeded if it maintains the (now unchangeable)
observable effects on downstream nodes when garbage collection the upstream node.

The section causing the most confusion is in

When an AudioNode has no references it will be deleted. Before it is deleted,
it will disconnect itself from any other AudioNodes which it is connected to.
In this way it releases all connection references (3) it has to other nodes.

The releasing of "connection references" to other nodes is kind of consistent
with the model that permits otherwise unneeded upstream nodes to be deleted,
but "disconnect itself from any other AudioNodes which it is connected to"
could be interpreted to mean a disconnection similar to an explicit
disconnect(). Such a disconnection can produce observable behavior changes on
downstream nodes.
#1453 (comment) at least is interpreting this to mean that the effects of the connections are
removed together with the connection references when the AudioNode is deleted.

If this section is trying to define an object lifetime model that is different
from that of JS objects then that is problematic. AudioNodes are JS objects,
and efficient and effective garbage collection algorithms cannot immediately
detect when an object becomes otherwise unneeded. There is a indeterminate
delay before the object is deleted.

Requiring behavior changes when the object is otherwise unneeded leads to
unpredictable behavior, as the behavior may either continue as if the object
is otherwise needed for a long period of time or not at all. Such variations
may occur either between implementations or within implementations.

[joeberkovitz]

I agree, this does seem to be a problem leading to the observability of GC.

[joeberkovitz]

Thinking out loud here... possibly not very soundly... it seems we can go in at least two different directions:

1. Purposely hide all observable information resulting from disconnection of GC'ed nodes, whether automatic or GC-related. At the least, disconnection would have to be prevented from decreasing the number of input channels visible to downstream nodes -- the input channel count could only go up as new nodes were added. But it would make the dynamic mixing behavior asymmetric and weird, as removals (even explicit ones) would no longer cause the corresponding downstream adjustments.

2. Disallow GC from having any effect on the graph at all. Nodes could no longer be pinned into the graph by merely maintaining a JS reference to them: they'd have to be "active" in some sense. This probably requires a very careful look at how we've defined "activity". However, the concepts of tail-time, the "playing" state of a AudioScheduledSourceNode and AudioWorkletProcessor's "active source flag" feel like they are sufficient, or almost so.

In many ways Option 2 feels like the more rational way to go here, but it would obviously be a big change. The hugest piece of this change seems to be that nodes could become disconnected from the graph due to their own deterministic changes in state. Since JS references to the node could still exist (because they would no longer prevent such disconnection), the node would have to make its own disconnected state visible to applications. Such nodes would either be no longer usable (and many of our nodes do become unusable once they are finished with their job), or there would have to be a codified way to re-use them.

[karlt]

I hadn't considered Option 1. It would work and is simple, but having explicit
disconnect() calls remove only some of the effects I think would be quite
unexpected. I'm unsure how much that matters, but option 3 below would
make this change unnecessary.

For option 2, I assume that an inactive node that became active again would be
automatically "reconnected" to the graph where it was before? If not I fear
option 2 would be too much a change in direction. I suspect there are plenty
of existing use cases where a processing graph is one-off pre-constructed and
one shot source nodes are added on demand.

Regardless, I expect this would be simpler than option 2:

3. Require that the downstream effects of connected nodes are
   maintained if and when the implementation garbage collects.

   Nodes are not "disconnected" when garbage collected, but their
   "connection references" can be removed.

This would be equivalent to not having any AudioNode lifetime section at all.
The implementation is free to garbage collect any resources provided that they
are no longer needed to maintain specified behavior.

[joeberkovitz]

The WG discussed this today. We came up with the following proposal along the lines of Option 2:

• Clarify the Node Lifetime section https://webaudio.github.io/web-audio-api/#lifetime-AudioNode to define that the lifetime in question is the lifetime of a AudioNode's effects on the graph, not of the JS AudioNode object itself.

• Remove condition 1 of this section, i.e. delete the language: "A normal reference obeying normal garbage collection rules." As a result of this change, GC can no longer have any effect on the audio graph which fixes this issue.

• Modify condition 2 of this section to clarify that AudioScheduledSourceNodes are retained in the graph not only while playing, but also during the period preceding a scheduled start time.

As Karl and I pointed out, the big consequence of this change is that a node can now be discarded from the graph due to inactivity even though it has JS references. If the holder of the JS reference could reactivate the node somehow (e.g. by connecting new inputs to it), this would present a problem. There are no such source node types, though, except AudioWorkletNode. Consequently we proposed to resolve #1463 by stating that returning false from process() causes the node to become permanently inactive.

On reflection this approach seems to me to have a big flaw: Consider an app that puts a GainNode into the graph, and holds a JS reference to that node with the intention of connecting other source nodes to its input later -- a very common pattern for a "master gain control". If this node automagically disappears from the graph (since it has no incoming connections yet, and it's not "active" as defined by the Lifetime section) then we broke this application. On the other hand, if we say that GainNodes always stick around until explicitly removed, we'll break all the applications that dynamically schedule a source S routed through a source-specific GainNode G: these apps create S and G on the fly, with the expectation that when S goes away, G will disappear too.

If that logic is correct, we're left with these choices:

• Allow "resuscitated" nodes to be automatically reconnected to the same place in the graph (which might require all downstream nodes to be also automatically reconnected to their former spots)

• Option 1 (effects of all disconnections are not observable from downstream)

• Option 3 (effects of disconnections "due to GC" are not observable downstream)

[joeberkovitz]

I'll mention one other infeasible approach which is to require app developers to explicitly mark nodes that ought to magically go away when disconnected. Incompatible.

[rtoy]

I think, independent of what the lifetime section says, GC is actually observable because we added AudioWorkletNode.

Consider a default ChannelMergerNode connected to an AudioWorklet. By default the merger has 6 channels on the single output. The worklet would see 6 channels on its input. If the reference to the merger node is dropped and collected, then there will suddenly be no inputs to the worklet. As currently spec'ed, input[0] will be an array with zero elements. Thus, you can tell that GC happened sometime before this change in the channel count.

I don't know how to fix this while also preserving the very, very useful feature of the worklet having dynamic channel count support.

[joeberkovitz]

Here is an outline of a possible fix to the situation. I do not think we can have perfect dynamic channel upmixing and GC-blindness at the same time, so this approach makes the channel upmixing a little less dynamic in order to eliminate the GC problem.

• Add a variable named deactivated to the AudioNode interface, initially having the value false. This variable cannot be accessed programmatically.

• Any node with a definite lifetime that is independent of GC (e.g. Oscillator, ABSN, etc.) must set deactivated to true when its lifetime ends. (Conversely, GC has no effect on the value of deactivated. So if a node's lifetime ends because of GC only, deactivated will be false.)

• Add a variable named input channel floors to the AudioNode interface, initially having an array containing elements with value 0, with one element for each of the node's inputs. This variable is taken into account by dynamic channel count up-mixing: an input to a given node will always be up-mixed to include at least as many channels as the value of the corresponding element of input channel floors.

• When a node N1 whose lifetime has ended has its outgoing connection to downstream node N2 automatically removed:

  • If N1's deactivated variable is false, then set the value of the corresponding element of N2's input channel floors to N1's output channel count. This causes N2 to continue reflecting the channel mixing effects of N1, even after N1 is gone. (For AudioWorkletNode, this implies that input[0] will not be empty, but shows a bunch of empty frames for the number of channels being "faked".)

This approach limits the cases where dynamic channel mixing is affected. In particular, dynamic channel count behavior will continue working, except in a small number of cases where developers rely on GC alone to eliminate nodes from the graph and these nodes have varying output channel counts.

[hoch]

It is Interesting that we had the similar thought, but consider this case:

N1 (6ch) -> N3
N2 (2ch) -> N3

Here N3's input channel floors will be 6, but what would be the input channel floors of N3 when N1 is removed by GC? 2 or 6? Either case poses a problem of its own:

1. If the value is still 6, 2 channel input from N2 will be up-mixed to 6.
2. If the value becomes 2, that means GC is still observable by inspecting the input channel count in N3.

Correct me if I misunderstood your proposal.

[joeberkovitz]

Let me explain what I think would happen here, I think maybe you misunderstood me (and I'm sure I could have been clearer!):

At the time the graph is constructed, N3's input channel floors will be set to [0], not [6]. A floor value for an input only gets set when a GCed node is disconnected from that input.

Thus, when N1 is removed by GC, then N3's input channel floors will be set to [6], making the GC unobservable. Correspondingly the input of N2 will continue being upmixed to 6.

Note that if N1 had been explicitly removed by calling N1.disconnect(), or if N1 was a source node that explicitly finished playing and was thus deactivated, input channel floors would still remain as [0]. So it's always possible to retain the dynamic mixing behavior, when there is an disconnection that is driven by an explicit, non-GC node lifetime.

[joeberkovitz]

[previous hasty comment deleted]

We also need a way for explicitly deactivated source nodes to communicate their state, by setting the deactivated flag on downstream nodes that then become eligible for release. This handles the case where subgraphs of sources chained through filter-type nodes like GainNode etc become inactive. I'm mulling this over and hope to post an update tomorrow after further thought.

[hoch]

Uh oh. Now I am confused.

At the time the graph is constructed, N3's input channel floors will be set to [0], not [6].

This sounds okay. How about during the rendering? What's the N3's input channel count?

A floor value for an input only gets set when a GCed node is disconnected from that input.

So it'll be 6 when N1 is GCed. Then it'll be 2 when N2 GCed? Then can't we get a signal when the number goes from 6 to 2? It won't match the exact timing of GC, but still this is a piece of information that developers can act upon.

Correspondingly the input of N2 will continue being upmixed to 6.

Well, this defeats the whole purpose of dynamic up/down mixing. The feels like sacrifice the most useful bit of WebAudio only to hide GC.

In my opinion, the source nodes are the least of our problems. The GainNode (N2 in the example) and the ChannelMerger (N1 in the example) going into an AudioWorkletNode (N3 in the example) would be our litmus test for GC observability.