From macOS Limits to Browser-Based VTubing

The Beginnings: Building mac-avatar-layer on macOS

We initially developed mac-avatar-layer as a lightweight avatar host for macOS. The goal was to display a transparent avatar over desktop applications without bringing in a full game engine.

The project is a Rust-first Live2D host: Cubism Core evaluates the model, and a custom Rust/Metal renderer draws it. That is different from Virmii's 3D VRM workspace, not an earlier version of the same 3D engine. But the experience of working on avatar display and controls helped shape Virmii.

Encountering Native macOS Platform Limitations

Turning a local desktop tool into something other people can run means handling more than rendering. Window behavior, permissions, graphics support, packaging, and updates all affect the experience. Working on one development Mac is different from working across users' machines.

A borderless overlay needs deliberate window ordering, focus, and click-through behavior. Screen capture and camera input have their own permission requirements; displaying a transparent window does not itself mean that screen-recording permission is required. These boundaries need to be explained clearly.

A renderer tied to Metal cannot simply be carried over unchanged to Windows or Linux. Signing, notarization, and update requirements also depend on how the app is distributed. Local development and public distribution are different workflows.

None of this makes native software the wrong choice. For a small project, though, it raises a practical question: how much time should go into avatar features, and how much into maintaining each platform's runtime and distribution setup?

The Cross-Platform Dilemma: Fragmentation Across Operating Systems

Displaying a transparent real-time avatar over the desktop involves different windowing and input behavior on each operating system. A shared framework can reduce the work, but it does not remove the need to test those differences.

• On macOS, native window and layer work involves Cocoa APIs such as NSWindow and CALayer.
• On Windows, Win32 and DWM provide windowing and composition mechanisms with their own styles and message handling.
• On Linux, behavior depends on X11 or Wayland, the compositor, and the protocols it supports.

Graphics backends also vary. Metal, Direct3D, Vulkan, and OpenGL have different availability and integration paths. An operating system does not necessarily require one particular API, but each supported backend and driver combination adds testing work.

Camera capture, audio loopback where needed, window input, and update distribution widen that scope. In planning Virmii, we reconsidered which differences to handle ourselves and which to delegate to a shared runtime.

Rethinking Avatar Engines: The Web Browser as a Universal Runtime

Browsers can run real-time 3D applications, not just display documents. WebGL and WebAssembly make it possible to render avatars and process camera input inside a web application.

The reason for choosing a browser-centered workspace was access across operating systems. Users can open a URL to inspect a model instead of installing a dedicated Virmii desktop client. We can work with browser APIs rather than build every operating system's windowing layer ourselves.

The differences do not disappear. GPU drivers, browser versions, camera permissions, and hardware still affect behavior and performance. Broadcasting through OBS also requires setting up OBS separately.

The goal is to reduce installation and connection steps, not promise identical performance everywhere. Explaining failures and limitations is part of that work.

Leveraging WebGL, WebGPU, and WebAssembly for Real-Time 3D

WebGL provides GPU-backed rendering in the browser. Virmii's current avatar viewer uses three.js, React Three Fiber, and three-vrm to load and display meshes, materials, bones, and the expressions available in a model.

WebAssembly can bring tracking and other processing modules into a web application. Virmii's face tracking uses MediaPipe Tasks Vision and its WASM resources. That does not mean all avatar physics, bone transforms, or blendshape calculations run in a custom WebAssembly engine.

WebGPU offers newer GPU capabilities, including compute shaders, but browser and device support must be checked. Virmii's current avatar renderer is not a WebGPU implementation. A possible future technology choice should not be presented as a feature already in use.

Shared buffers and moving calculations into WASM are not automatic performance fixes either. Data transfer, memory management, GPU load, and other page activity need measurement. Running in a browser does not guarantee a stable 60 FPS viewport or freedom from dropped frames.

For users, understandable loading errors, permission prompts, and connection states often matter more than the API names. The runtime needs to make clear what is available in their environment.

Architecting Virmii: From Desktop Prototype to Web Workspace

Where mac-avatar-layer started with displaying an existing avatar on the desktop, Virmii connects character preparation, model review, and broadcast output. That required more than moving a native viewer into a web page.

The current workflow is:

1. Start a 3D avatar generation with a character reference image you are authorized to use.
2. Review the generated model in the browser.
3. Generate AI expression images separately if needed, then test the eye and mouth responses.
4. Adjust camera tracking and model response in Live Studio.
5. Add the private live-stage URL to an OBS Browser Source.
6. Download the available VRM and check compatibility before using it in another app.

The browser loads models with GLTFLoader and VRMLoaderPlugin. Avatar display and animation run on the user's device, while generation, storage, authentication, and other service functions also involve backend systems. Generating a model and displaying the completed model are separate stages.

The OBS connection is not a virtual-camera video feed. Live Studio sends movement data through a WebRTC data channel, and the OBS Browser Source renders the model on the stage page. A connection may use a relay, but this path does not send raw camera footage to OBS.

Separating the tracking controller from the output page lets creators add an avatar to OBS without a dedicated Virmii virtual-camera driver. They must keep the tracking page open during use and keep the private stage URL secure.

Democratizing VTubing: Lowering Technical Barriers for Creators

Preparing an avatar can involve modeling, textures, rigging, expressions, file formats, tracking, and OBS settings. Virmii brings the path from generating a first model to testing its movement and displaying it in OBS into one workspace. It does not replace every part of specialist modeling or manual refinement.

Virmii-generated models can use a separate set of 13 AI expression images. Those images work in Virmii previews, Live Studio, and the OBS stage; they are not embedded in downloaded VRM files. Uploaded VRMs use their existing expression bindings. The AI expression guide explains the scope.

Generation and expression features use credits or plan allowances. Check the current pricing before starting, and allow for the possibility that a result will need another attempt or external editing.

A browser workflow reduces installation steps, not computational load. Whether a budget laptop works well depends on model complexity, resolution, the browser, and other applications such as games. Test a recording with the actual broadcast setup and lower settings where needed.

For a first setup, the VTuber beginner guide covers the preparation and testing sequence.

Separating Camera Tracking from Audio Processing

Camera footage is sensitive data. Virmii's current tracking processes frames in the browser and derives head orientation and eye, mouth, and other movement values. Those values, rather than the original video, are sent to the private stage.

MediaDevices.getUserMedia() obtains camera or microphone streams with permission. Virmii requests camera input with audio: false. Face-tracking results drive the model's supported expressions; there is no microphone frequency-analysis or phoneme-recognition pipeline providing lip sync.

Using Web Audio's AudioContext, AnalyserNode, and AudioWorklet to build a voice-reactive avatar is a separate implementation choice. Connecting an analysis node would not itself provide accurate phoneme-level lip sync, and it should not be described as a current Virmii feature. Set up the broadcast microphone in OBS separately.

This minimal example matches Virmii's camera request constraints. It opens a stream; it does not implement face tracking or the OBS connection:

async function openTrackingCamera() {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: false,
    video: {
      facingMode: "user",
      width: { ideal: 1280 },
      height: { ideal: 720 },
      frameRate: { ideal: 30, max: 30 }
    }
  });

  return {
    stream,
    stop() {
      stream.getTracks().forEach((track) => track.stop());
    }
  };
}

A real application also needs a secure context such as HTTPS, permission-denial handling, camera-availability checks, and cleanup when the session ends. Requested video dimensions are not a guarantee of the device's actual output.

Architectural Comparison: Native Desktop App vs. Browser Playback Engine

The tradeoffs concern distribution, permissions, device access, and maintenance, not a simple claim that one approach is always faster.

Native software may suit specialized hardware or detailed local control. For Virmii, we put more weight on a shared entry point and setup flow. The browser supports that choice without removing every constraint.

The Future of Virmii and Open Web Avatars

Virmii grew out of wanting an avatar workflow that more people could reach through the web. Work on a small macOS tool became a question about reviewing models and connecting them to broadcasts across different environments.

Open formats such as VRM and glTF help creators move models between tools. Downloading a file does not mean every service feature travels with it, however. Virmii's AI expression images are one example of why export limitations need to be explicit.

Model validation, expression setup, tracking adjustment, and interoperability remain useful directions for improvement. New APIs such as WebGPU are options to evaluate, not performance promises. We need to test their value and explain what remains available on unsupported devices.

Improving animation, state synchronization, modular implementations, and technical documentation is a direction for the project, not a list of features already shipped. The aim is a workspace that reduces setup work while making its current capabilities and limits understandable.

Q: Why was native macOS software not enough for mac-avatar-layer?

A: It remains useful as a macOS-specific tool. Extending its Metal renderer and native window handling to other operating systems would take additional work. Virmii is a separate browser-centered workspace chosen for broader access and a workflow from model preparation to OBS output.

Q: Can a web browser handle real-time 3D VTuber avatars smoothly?

A: Yes, browser-based real-time rendering and camera tracking are possible. Virmii uses them today, but performance depends on model complexity, hardware, the browser, and other running applications. It does not guarantee 60 FPS or zero latency.

Q: What makes Virmii different from traditional native VTuber software?

A: Instead of a dedicated Virmii desktop client, creators use a web workspace for generated-model review, VRM management, tracking, and a private stage for OBS. The stage connects through a Browser Source URL. Camera permission, a supported browser, sufficient local resources, and OBS setup are still necessary.

Q: Is Virmii suitable for independent creators with budget hardware?

A: Test the model and broadcast setup on the hardware you have. Avoiding a dedicated installer is useful, but browser execution still uses CPU and GPU resources. Run the game, avatar, and OBS together and adjust output resolution or frame rate if needed.