Language worker "User" logs are ignored after execution is complete

[ejizba]

Back when @yojagad added a LogCategory for language workers to pass either System or User in this PR to the node worker, @pragnagopa said

With in the execution context, we should not be logging system logs

However, some of the logs in question are like this:

Error: 'done' has already been called. Please check your script for extraneous calls to 'done'.

Which actually happens after execution completes. As of today, those logs are silently ignored by the host here (nothing in app insights, nothing in the console, etc.). Which leads me to a few questions:

• What exactly is the difference between System and User logs? (We might want to document in the protobuf definition)
• How should language workers categorize logs that are related to an execution, but after an execution?
• Should the host really be ignoring these logs? These logs could be coming directly from the user, and perhaps they don't realize they log after execution completes because they never see the logs.

cc @alrod

[pragnagopa]

What exactly is the difference between System and User logs? (We might want to document in the protobuf definition)

User logs - Cx have access to these logs via App Insights - are super set for Function Execution logs + Host / Extension logs - Any logs coming from Host are considered as system logs and can be logged to Kusto. User logs cannot be logged to Kusto as they are considered PII

How should language workers categorize logs that are related to an execution, but after an execution?

Logs between invocation request and invocation complete should be considered user logs- logs after execution completes are orphaned logs
https://github.com/Azure/azure-functions-language-worker-protobuf/blob/6635670b136d565f2a196ce1a64fee1ea5c9c1c9/src/proto/FunctionRpc.proto#L59-L62

Should the host really be ignoring these logs? These logs could be coming directly from the user, and perhaps they don't realize they log after execution completes because they never see the logs.

Host should not log from orphaned executions - but you bring up a good point specifically for errors - as of now we do not have a good solution for this.
Tagging @fabiocav @liliankasem for design discussion on how to address this.

[ejizba]

Logs between invocation request and invocation complete should be considered user logs

If the language worker is crafting the message itself, can't we consider it system logs regardless of when it happens? There's no risk of PII as long as it's a simple hard-coded string, which is why I assume it's okay for the host to write system logs during this time. Having those worker logs in our telemetry would be very helpful for supportability of language workers.

[liliankasem]

ich actually happens after execution completes

If the error is coming from the worker itself, I would consider it a system log, especially if it doesn't include the user's actual function logs (which is the PII part). i.e. like Eric suggested, the worker is seeing some sort of error and is crafting a log for it

[fabiocav]

This item requires design and we need to gather requirements for worker logs not associated with an invocation.

Adding to the backlog for the design work.

[fabiocav]

@AnatoliB it would be good to sync on this to make sure we're not duplicating anything you're already planning for this area.