HTML-in-Canvas API: Build Next-Generation UIs with WebGL/WebGPU

Introduction

The HTML-in-Canvas API enables developers to draw standard DOM content directly into a canvas or WebGL/WebGPU texture, allowing for advanced 2D and 3D graphics while retaining full interactivity and browser feature integration.

Configuration Checklist

Element	Version / Link
Language / Runtime	JavaScript
Main library	HTML-in-Canvas API (browser built-in)
Required APIs	Canvas 2D Context, WebGL, WebGPU, ResizeObserver
Keys / credentials needed	Origin Trial token (for production deployment in Chrome)

Step-by-Step Guide

Step 1 — Set layoutsubtree property on canvas

This step is crucial because it makes the browser aware of the content within the canvas. This awareness enables hit testing, accessibility features, and other browser integrations for the rendered HTML.

<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
    <!-- Your HTML content will go here -->
</canvas>

Step 2 — Add your HTML inside canvas

Place the HTML elements you wish to render within the <canvas> tags. This defines the content that the API will draw onto the canvas or texture. This can be done directly in your HTML or programmatically via JavaScript.

<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
    <div id="form_element">
        name: <input>
    </div>
</canvas>

Step 3 — Render the UI element to canvas / texture

This step explicitly draws the DOM content onto the canvas or texture. The method used depends on the rendering context (2D, WebGL, or WebGPU).

Context2D rendering

const ctx = document.getElementById('canvas').getContext('2d');

// The onpaint event triggers whenever the element is getting redrawn
canvas.onpaint = () => {
    ctx.reset(); // Clear the canvas for redrawing
    // Draw the form element at position 0,0
    let transform = ctx.drawElementImage(form_element, 0, 0);
    // Update the transform of the element (see Step 4)
    form_element.style.transform = transform.toString();
};

WebGL Rendering

<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
    <div> <h1> I'm gonna be rendered in a WebGL texture! </h1> </div>
</canvas>

// webgl setup code [Editor's note: full WebGL setup code is not provided in the transcript]

canvas.onpaint = () => {
    // texture setup code [Editor's note: full texture setup code is not provided in the transcript]
    // Check if gl.texElementImage2D is available before calling
    if (gl.texElementImage2D) {
        // Pass the DOM element as the image source for the texture
        gl.texElementImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, uiElement);
    }
};

WebGPU Rendering

<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
    <div> <h1> I'm gonna be rendered in a WebGPU texture! </h1> </div>
</canvas>

// WebGPU setup code [Editor's note: full WebGPU setup code is not provided in the transcript]

canvas.onpaint = () => {
    // Use copyElementImageToTexture to render the DOM element into a WebGPU texture
    root.device.queue.copyElementImageToTexture(valueElement, 512, 128, { texture: targetTexture });
};

Step 4 — Update the transform of the element

It is vital to update the CSS transform of the DOM element whenever it is repainted. This informs the browser about the element's current position and orientation on the screen, ensuring that interactivity and browser features remain functional.

Updating WebGL & WebGPU Transform

// This code snippet assumes `uiElement` is the DOM element being rendered
// and `canvas` is the canvas element with the layoutsubtree attribute.

// Check if getElementTransform is available
if (canvas.getElementTransform) {
    // The following lines represent the logic to convert 3D coordinates to CSS coordinate space.
    // [Editor's note: `toCSSViewport`, `mvpDOM`, `toGLModel` are undefined in the transcript and represent external 3D matrix operations. 
    // You would typically use a 3D math library (e.g., gl-matrix) to compute these transformations.]
    // 1. Convert the MVP matrix (gl-matrix Float32Array) to a DOMMatrix
    // 2. Element CSS pixels -> WebGL Model Space
    // 3. WebGL Clip Space -> Canvas CSS pixels (Viewport Transform)
    // 4. Combine: Viewport * MVP * Model
    const finalTransform = toCSSViewport.multiply(mvpDOM).multiply(toGLModel);

    // Get the CSS transform matrix for the UI element based on its 3D position
    const transform = canvas.getElementTransform(uiElement, finalTransform);
    if (transform) {
        // Apply the calculated CSS transform to the DOM element's style
        uiElement.style.transform = transform.toString();
    }
}

Step 5 — Setting the canvas size properly

To prevent blurriness, especially on high-DPI screens, it's important to size the canvas grid to match the device's pixel scale factor. A ResizeObserver can be used to dynamically adjust the canvas dimensions.

// Create a ResizeObserver to monitor changes in the canvas's content box size
const observer = new ResizeObserver(([entry]) => {
    // Set the canvas width to match the inline size of the device pixel content box
    canvas.width = entry.devicePixelContentBoxSize[0].inlineSize;
    // Set the canvas height to match the block size of the device pixel content box
    canvas.height = entry.devicePixelContentBoxSize[0].blockSize;
});

// Observe the canvas element, specifying 'device-pixel-content-box' to get physical pixel dimensions
observer.observe(canvas, { box: 'device-pixel-content-box' });

Comparison Tables

Feature / Aspect	The DOM (Document Object Model)	The Canvas (HTML5 Element)	HTML-in-Canvas API
Content Representation	Semantically understood content (HTML elements)	Low-level grid of pixels	HTML elements rendered as pixels on canvas/texture
UI & Text Layout	Great out-of-the-box solutions, leverages semantic content	Requires complex logic or frameworks for layout	Leverages DOM's layout capabilities
Interactivity	Inherently interactive (buttons, forms, text selection)	Requires custom logic for interactivity	Fully interactive, inherits DOM interactivity
Accessibility	Excellent browser integration (screen readers, tab navigation)	Lacks inherent accessibility features	Fully accessible, exposes content to accessibility systems
Browser Features	Integrated (translate, find-in-page, dark mode, zoom, autofill, extensions)	No direct integration with browser features	Full integration with browser features
Graphics Capabilities	Standard 2D rendering, limited advanced graphics	High-performance 2D and 3D graphics (WebGL, WebGPU)	Combines high-performance graphics with rich HTML UI
Development Complexity	Easier for standard UI, complex for advanced graphics	Complex for UI, easier for advanced graphics	Bridges complexity, simplifies advanced UI in graphics
Bundle Size	Smaller for basic UI, larger with complex frameworks	Can be smaller if custom logic, larger with frameworks	Potentially reduces custom UI logic, leverages browser

⚠️ Common Mistakes & Pitfalls

1. Forgetting layoutsubtree: Without the layoutsubtree attribute on the <canvas> element, the browser will not process the HTML content within the canvas for hit testing, accessibility, or other browser features. Always include it.
2. Not explicitly rendering: The HTML content inside the <canvas> tag is not automatically drawn. You must make an explicit API call (e.g., ctx.drawElementImage, gl.texElementImage2D, root.device.queue.copyElementImageToTexture) to render it.
3. Failing to update CSS transform: After rendering the HTML content into the canvas/texture, you must update the CSS transform property of the original DOM element. This ensures the browser knows the element's visual position for correct interaction and feature integration.
4. Not removing elements when no longer painting: If you stop painting an HTML element into the canvas, it's critical to remove it from the canvas element's children. Otherwise, it will continue to be presented to browser and accessibility systems as part of the page, even if it's no longer visually rendered.
5. Incorrect canvas sizing: Not setting the canvas width and height to match the device's pixel scale factor can lead to blurry rendering. Use a ResizeObserver with device-pixel-content-box to ensure proper scaling.

Glossary

DOM (Document Object Model): A programming interface for web documents, representing the page structure as a tree of objects that can be manipulated by scripts.
Canvas: An HTML5 element that provides a bitmap canvas for rendering 2D graphics, and can also be used for WebGL and WebGPU for 3D graphics.
layoutsubtree: An attribute on the <canvas> element that signals to the browser that its children should be laid out and exposed to browser features, even if they are rendered into the canvas.

Key Takeaways

• The HTML-in-Canvas API allows developers to render live, interactive HTML content directly into a canvas or WebGL/WebGPU texture.
• It preserves crucial browser features like text selection, accessibility, translation, dark mode, browser zoom, and autofill for the rendered HTML.
• The API is currently available in Chrome as an Origin Trial, allowing for experimental production deployment.
• Integration involves setting the layoutsubtree attribute on the canvas, using specific rendering methods (drawElementImage, texElementImage2D, copyElementImageToTexture), and updating the CSS transform of the HTML element.
• Popular 3D frameworks like Three.js and PlayCanvas have already landed experimental support, simplifying its adoption.
• Properly sizing the canvas using ResizeObserver and device-pixel-content-box is essential to avoid blurriness.
• The API enables the creation of visually stunning and highly interactive web applications, such as 3D books with selectable text, dynamic billboards, and immersive overlays.
• It allows AI tools and browser extensions to interact with the rendered HTML content as if it were standard DOM.

Resources

• HTML-in-Canvas How-To
• Three.js HTML-in-Canvas Support
• Three.js HTML-in-Canvas Example
• PlayCanvas HTML-in-Canvas Support
• PlayCanvas HTML-in-Canvas Example
• PlayCanvas Shoe Demo
• HTML-in-Canvas Explainer
• Modern Web Guidance
• API Samples
• Origin Trial Signup
• Awesome HTML-in-Canvas GitHub Repo
• All Links Central