Basketball Game Analysis using an LLM

I asked OpenAI's ChatGPT and Google Gemini to analyze some game snapshots, and it's incredible how well they break down a game situation.

When ChatGPT said, 'It could be a pick-and-roll situation,' that was spot on 😮

It even recognises player positions, jersey numbers, and the sponsor boards along the walls.

Now, imagine running an AI analysis every few seconds, with responses structured in a JSON schema. You'd essentially have an expert AI coach explaining every move throughout the game.

Here's the response from ChatGPT on what it replied based on the above picture 🤯 🧠

Now let's try Google Gemini with another picture and see if the Pro version 1.5 is any good...

I listed the player names and jersey number in the system prompt as follows:

Then I uploaded the above image and asked the following question:

"Explain in great detail what you see"

Just crazy!

Both OpenAI and Google Gemini support JSON schema responses, so we could easily provide a template answer where the score, clock, but also possible plays are listed and the LLM needs to tell use what it thinks it sees.

In the Google AI Studio you can actually define the JSON schema using a visual editor.

Now when we prompt the model the answer will always be in the format of the above JSON schema.

which is exactly what the video stream was showing in the picture.

Unfortunately looking at the pricing of these cloud based LLM's it seems the explored approach is not yet feasible considering the token cost.

I did try using a local model named "LLaVa" using Ollama but the result was not as good as the commercial cloud based LLM's. Here's the code and chat response:

Currently, the main barrier with LLM cloud providers is pricing, not the technology itself. And probably also the inference execution time.

If we have an unlimited (LLM token) budget we could perfectly develop a basketball analyzer that mentions when certain plays are happening (layup, dunk, three point, mid-range, free throw etc), the player names involved etc.

Exciting times! 🤩

Stephan Janssen