I love watching sports games… when I'm with good company. Sup Player lets viewers watch entertainment alongside their favorite players and characters.

I designed this prototype with the concept that watching sports is about more than just observing the game; it's about immersing yourself in the vibrant atmosphere of cheers, shouts, and lively fan interactions. My aim was to elevate the experience of watching sports, even when you're enjoying it solo.

Sup Player is currently built through a manual process of rigging 3D character models and triggering their animations / dialogues to play at specified timestamps of the AVPlayer. I created the voice cloning with Eleven Labs and wrote the script myself.

After creating the above prototype, I considered what other genres of media could be enhanced with this type of experience. While I’m not an avid TV watcher, I've observed many friends and family members who frequently revisit their favorite shows and series. Providing fresh and interactive content to old episodes feels like low hanging fruit inside Apple Vision Pro, sort of like offering DLC for TV. Thus, I prototyped what a similar experience would look like when watching a rerun of a Rick and Morty episode (inspiration from MST3000).

The above is built with the same Eleven Labs voice cloning method, but leverages a transparent video texture with a .lookAt modifier to render Rick, along with a particle emitter anchored to the video entity to depict the hover material.

To prototype a scaled version of this, I plan to expand on this by implementing the following steps:

• Enter URL to capture video from any public link

• Transcribe the video and audio

• Run the video and audio through a classification model to generate array of timestamps with high probability that an event occurred (currently thinking of using a mix of Twelve Labs for video and Lab Studio for audio)

• Trigger animation and dialogue from character models by iterating through timestamp array

Lastly, I believe an experience like this needs to use 3D models that are either hyper-realistic, or clearly stylized. I'm inclined to use transparent video textures as hyper-realistic content in early iterations of this prototype to get more expected visualizations of characters.