Character Puppeteering & Co-Creation in Resonite

Two months of building. Years of seeing where this could go. We’ve built a framework where Python drives characters in real-time, inside VR, inside Blender, across platforms. No motion capture suits. No animation studios. Just hands, a camera, sensors, and code.

The characters aren’t avatars. You don’t become them. You’re beside them. Your hand curls, a wing flaps. Your heartbeat pulses, a gecko grips tighter. Water touches a sensor, a whale starts to sing. Fire burns in Bolivia, an orchid wilts.

Each creature has its own brain, neural networks reading your face through MediaPipe, local AI models deciding how they feel, environmental sensors connecting them to the real world. The intelligence layer is the same Python whether the creature lives in Blender or in a shared VR world in Resonite.

That’s the breakthrough: the brain is platform-agnostic. Write the animation logic once. Run it anywhere. In VR, we can run as many characters as we want because we’re animating them in real-time, not playing back keyframes.

This is cinema. This is neural networks. This is co-creation between human bodies and digital beings. Built from Cochabamba. Rooted in Indigenous knowledge systems. Running on open-source tools.

15 creatures. 14 pipelines. Water sensors, pulse sensors, fire satellites, hand tracking, face tracking, AI emotion. And we’re just getting started, months of building ahead.

We won’t let others design the rules of the world we will live in.

unitednotions.film

This fortnight: the water lily learned to breathe, the sundew learned to count fingers, and the pipeline moved to WebSocket.

Two weeks ago the tree was shy. Now there are fourteen creatures, and they’re starting to know each other.

The Victoria Regia came first. A waterproof sensor dropped into a glass of water, an ESP32 reading capacitance every 40 milliseconds, and a giant Amazonian water lily that opens when the water is still and closes when you disturb it. Dip your finger in, the leaves contract. Pour a little more water, the whole pad tilts. It took a day to get the baseline calibration right, the sensor drifts with temperature, so we made it re-zero every time the system boots. Pipeline 12. The lily breathes now.

The Drosera was harder. The sundew has thirteen tentacles and each one needs to curl toward a specific fingertip, which meant finally fixing the MediaPipe hand-landmark mapping we’d been half-ignoring for months. Index finger to tentacle 3, thumb to tentacle 1, and so on. When your fingers splay, the sundew opens. When you pinch, it snaps shut around an invisible insect. Brian rigged it so the curl comes from the base, not the tip, which is how real sundews actually move. You can feel the difference immediately.

The Colibri got her head back. The old version tracked only the body, so the hummingbird’s head pointed wherever the torso pointed, which is not how hummingbirds work. We moved to MediaPipe FaceMesh, 468 landmarks, and now the head tracks independently, tilts, follows your gaze. She looks at you when you look at her. It’s unsettling the first time.

And then the big architectural decision. For weeks we’d been running everything through UDP into Blender, which worked for the studio but wouldn’t survive deployment into Resonite VR. So we rewrote the spine. Python handles all the external sensor logic now, cameras, ESP32s, satellites, proximity, and instead of sending raw bone rotations we send about fifty creature-state values through a ResoniteLink WebSocket. “Tree fear level: 0.7.” “Orchid grief: 0.4.” “Llama stillness: 1.0.” The creatures decide for themselves how to express those states inside VR. It’s the difference between puppeteering and relationship. Dan built a standalone hand-tracking client for his PC that plugs straight into Resonite, no Blender in the middle, so we can finally test in headset without the full studio rig.

On the side, I’ve been getting ComfyUI and Hunyuan3D running locally on the M5 Max, 48GB of unified memory, generating creature assets without sending a single frame to a server. The first test was a moth. It was a terrible moth. The second was better.

Eleven pipelines became fourteen. The tree is still shy. The orchid is still mourning. The llama still stands at rest when you stand at rest. But now the water lily breathes, the sundew counts fingers, and the hummingbird looks you in the eye.

Todo nace desde lo pequeño. Y lo pequeño está empezando a hablar entre sí.​​​​​​​​​​​​​​​​

This week Violeta Ayala taught a tree to be shy.

A Sharp infrared sensor on an Arduino, $3 of hardware, measures how close you are. When you’re far away, 6,317 bones dance independently, each leaf moving to its own rhythm like fingers underwater. When you approach, the tree notices. It tenses. Get closer and it shakes, folds inward, trembles. It’s afraid of you. Step back and it slowly, slowly exhales back into its private dance.

The same week, NASA told our orchid the world was on fire. Real satellite data, MODIS sensors at 705km altitude scanning the planet every six hours, feeding into a Maxillaria orchid that wilts, curls, changes color from green to charred black depending on how many fires are burning in Bolivia right now. The orchid doesn’t know it’s an orchid, but its petals droop with real grief pulled from real coordinates where real forests are disappearing. I built an interactive world map where you can click any country and watch the fires load in. The numbers are always worse than you expect.

And then there’s the llama. Brian Condori rigged 56 bones and I spent two days solving this problem: when a human stands up straight, a llama should stand on four legs doing nothing. Sounds obvious. But every tracking system sends non-zero data when you’re standing still, your shoulders have an angle, your elbows have an angle, your knees have an angle. The llama was standing like a person. Lifting its chest, crossing its forelegs. So we built a calibration system: stand still for two seconds, the system learns what “you doing nothing” looks like, and from then on it only sends the difference. Standing equals zero. Zero equals a llama at rest on four hooves with a horizontal spine.

We talked with Eilif B. Muller and Yasmeen Hitti about what happens when the data goes the other way. Not just body to creature, but creature back to body. If the tree is afraid, can the performer feel the fear? Haptics. Vibration. Texture. Brain cortex stimulation.

Eleven pipelines running now. Five plants, four animals, a fire, and a frightened tree. Every creature responds to something different, a camera, a gyroscope, a pulse sensor, a satellite, a proximity sensor and none of them know about each other yet. They just receive UDP packets and become alive.

The llama still needs work. The leaves still don’t fully update from timer callbacks. The IMU sends garbage at the wrong baud rate. But the tree is shy, the orchid mourns, and when Brian moves his arms in front of the OAK-D camera the llama’s legs follow on the screen beside him.

Todo nace desde lo pequeño.

Daniel Fallshaw is back with a lots of new sensors and more computers, hard drives, mics, cameras… :-)

This week at our lab we hit a wall and broke through it with a $2 sensor.

For two weeks we tried to make a single camera track a performer’s full 360° body rotation. We tested 9 different algorithms. MediaPipe (3.5M parameters), depth silhouette PCA, shoulder swap detection, face visibility tracking, every approach failed past 180°.

After discussing with Eilif B. Muller, we strapped an MPU-6050 gyroscope to a belt. Full 360°..:Continuous. No blind spots. The gyroscope gives the creature what the camera can’t: a sense of proprioception, a felt sense of which way it faces, even when facing away. When the performer turns back toward the camera, MediaPipe face tracking gently corrects the gyroscope’s drift. Two contradictory sensing systems, each filling the other’s gaps.

This is sensor fusion applied to real-time non-humanoid character animation. Eight creature pipelines active. Music leads the performance, the performer’s body translates it, the sensors capture that translation, and the creatures embody it live.

The $2 gyroscope solved what the neural network couldn’t. Sometimes the answer isn’t a bigger model, it’s a different sense.

Working from our small lab in the mountains in Cochabamba, we are developing new approaches to animate non-humanoid digital characters.

Most computer vision frameworks were trained on human pose datasets. When working with creatures, plants, or unconventional bodies, these systems quickly reach their limits. There is also little data about spatial behavior — how bodies actually move through depth, rotation, and real physical space. We optimized vision systems to recognize humans in images, but not necessarily to understand how movement unfolds in the real world. W ere missing embodied intelligence.

To address this, we are integrating MediaPipe tracking with stereo depth sensing from a robotic camera. This allows us to estimate spatial orientation and body rotation using real-world measurements in millimeters rather than relying purely on 2D image heuristics.

Much of the work happens through hands-on experimentation: prototyping new pipelines, combining sensing systems, and testing ideas through iteration.

At United Notions Film the last two weeks have unfolded across two fronts that keep feeding each other, technology in motion and documentary in real time.

In the lab we expanded motion systems inside Blender, developed hand–command interaction through MediaPipe and TensorFlow, and brought our IoT pixel surfaces closer to a nervous system of their own. The panels respond to movement like organisms made of light. Not fully sentient yet, but no longer passive. They react. They learn. They almost feel.

On the documentary front the atmosphere remains sharp and alive. We filmed abuelas y abuelos holding their vigil for more than 93 days. We recorded the Comteco worker streaming on TikTok and revealing how speech fractures when a city is tense. Protesters continue to update the system from the street and you can sense the collective brain pulsing; human logic, cellular networks, shared footage circulating like neurons.

Social platforms have become part of the film itself. Not only a window but a circulation system. Vi’s TikTok surpassed one million views and her Instagram reels keep carrying testimonies further and faster than traditional distribution ever could. The documentary is no longer waiting for the edit. It is happening with us and through us. Alive, unstable, unfiltered, collaborative.

Maybe this is the evolution of nonfiction. A documentary that acts like cognition in real time. Cameras as nodes. Crowds as processors. Stories breathing mutating responding like systems made of people and pixels.

We return to one question again and again.

What happens when documentary begins to think?

When it responds?

When it becomes sentient before we notice?

For now UNF continues in two rhythms.

One hand in code teaching machines to listen.

The other in the street listening to the people who refuse to disappear.

The work moves. The world moves with it.

Technologies of distribution, Bolivia opening a new political chapter, and the urgent fight for transparency in our telecom cooperative Comteco.

While there’s growing hype around Starlink entering the country, Bolivia still doesn’t have a data protection law, digital rights remain undefined, and our existing infrastructure lacks transparency. Satellite internet isn’t a substitute for governance, accountability, or sovereignty.

For 88 days, senior citizens, the OG (original) shareholders who built this cooperative — have been holding vigil, demanding elections, audits, and answers. Much of the media remains silent, so I’ve been using TikTok as a counterbalance, turning citizen journalism into a living archive of this moment.

Bolivia is shifting.

Technology is shifting.

And we must build systems; political, digital, and narrative, that honour truth, memory, and the communities that carry them.

Violeta in Itoshima performs for the robotic camera as Huk, reacts and voices perceptions, a live, multilingual experience to Dan in Sydney.