{"schemaVersion":1,"id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser","title":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser","description":"Low-level video encoding and decoding without [FFmpeg.wasm](https://ffmpegwasm.netlify.app/), VideoEncoder, VideoDecoder, AV1 encoding, and building a real-time video effects pipeline entirely in the browser.","language":"en","datePublished":"2026-03-15","dateModified":null,"contentFormat":"text/markdown","content":"Web APIs\n\n# WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser\n\n2026-03-1510 min read\n\n[Lucio Durán](<https://lucioduran.com/>)\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/webcodecs-api-video-processing-browser>)\n\n## WebCodecs Overview\n\nWebCodecs sits below the high-level media APIs (` <video> `, MediaRecorder, MediaStream) and above the raw byte manipulation level. It gives you:\n\n- **VideoDecoder**, Decode compressed video frames (H.264, VP8, VP9, AV1) into raw ` VideoFrame ` objects\n- **VideoEncoder**, Encode raw ` VideoFrame ` objects into compressed chunks\n- **AudioDecoder/AudioEncoder**, Same thing for audio (AAC, Opus, FLAC)\n- **VideoFrame**, A raw video frame that can come from a canvas, camera, or decoder\n- **EncodedVideoChunk**, A compressed video frame ready for muxing\n\nThe key insight: these are **hardware-accelerated**. When you create a VideoEncoder for H.264, the browser delegates to the GPU's dedicated encoding ASIC (NVENC on NVIDIA, VCE on AMD, Quick Sync on Intel). You get the same encoding speed as a native application.\n\n## Basic Encoding Pipeline\n\nLet's start with the fundamentals, encoding a canvas animation to an H.264 video:\n\n```javascript\n// Step 1: Check codec support\nconst support = await VideoEncoder.isConfigSupported({\n codec: 'avc1.42001f', // H.264 Baseline Level 3.1\n width: 1920,\n height: 1080,\n bitrate: 5_000_000, // 5 Mbps\n framerate: 30,\n});\n\nif (!support.supported) {\n console.error('H.264 encoding not supported');\n // Fall back to a supported codec\n}\n\n// Step 2: Collect encoded chunks\nconst chunks = [];\nconst encoder = new VideoEncoder({\n output: (chunk, metadata) => {\n // chunk is an EncodedVideoChunk\n const buffer = new ArrayBuffer(chunk.byteLength);\n chunk.copyTo(buffer);\n\n chunks.push({\n type: chunk.type, // 'key' or 'delta'\n timestamp: chunk.timestamp,\n duration: chunk.duration,\n data: buffer,\n // metadata.decoderConfig available on keyframes\n decoderConfig: metadata?.decoderConfig || null,\n });\n },\n error: (e) => console.error('Encoder error:', e),\n});\n\n// Step 3: Configure the encoder\nencoder.configure({\n codec: 'avc1.42001f',\n width: 1920,\n height: 1080,\n bitrate: 5_000_000,\n framerate: 30,\n latencyMode: 'quality', // Buffer frames for better compression\n avc: {\n format: 'annexb', // Raw H.264 NAL units (vs 'avc' for MP4-style)\n },\n});\n\n// Step 4: Encode frames from a canvas\nconst canvas = document.getElementById('my-canvas');\nconst ctx = canvas.getContext('2d');\n\nfor (let i = 0; i < 300; i++) { // 10 seconds at 30fps\n // Draw frame i on canvas\n renderFrame(ctx, i);\n\n // Create VideoFrame from canvas\n const frame = new VideoFrame(canvas, {\n timestamp: i * (1_000_000 / 30), // Microseconds!\n duration: 1_000_000 / 30,\n });\n\n // Encode the frame\n const keyFrame = i % 90 === 0; // Keyframe every 3 seconds\n encoder.encode(frame, { keyFrame });\n\n // CRITICAL: close the frame to release GPU memory\n frame.close();\n}\n\n// Step 5: Flush remaining frames\nawait encoder.flush();\nencoder.close();\n\n// chunks[] now contains all encoded H.264 data\n// You need a muxer (like mp4-muxer) to wrap it in a container\n```\n\nThat ` frame.close() ` call is essential. VideoFrames hold GPU texture references. If you forget to close them, you'll leak GPU memory and eventually the browser kills your tab. Without proper frame lifecycle management, video processing applications commonly crash after ~200 frames.\n\n## Decoding: Reading Video Frame-by-Frame\n\nThe decoder side is equally powerful. Here's how to decode an MP4 and get individual frames:\n\n```javascript\n// You need to demux the MP4 first, WebCodecs doesn't handle containers\n// Using mp4box.js for demuxing\nimport MP4Box from 'mp4box';\n\nasync function decodeVideo(videoBuffer) {\n const frames = [];\n\n const decoder = new VideoDecoder({\n output: (frame) => {\n // frame is a VideoFrame, raw pixels accessible via\n // canvas drawing, ImageBitmap, or copyTo()\n frames.push(frame);\n // Don't close here if you want to keep the frames!\n },\n error: (e) => console.error('Decoder error:', e),\n });\n\n // Demux the MP4 to get codec config and encoded chunks\n const mp4 = MP4Box.createFile();\n\n return new Promise((resolve) => {\n mp4.onReady = (info) => {\n const videoTrack = info.tracks.find(t => t.type === 'video');\n\n // Configure decoder with codec info from the container\n decoder.configure({\n codec: videoTrack.codec, // e.g., 'avc1.64001f'\n codedWidth: videoTrack.video.width,\n codedHeight: videoTrack.video.height,\n description: getAVCDecoderConfig(videoTrack),\n });\n\n mp4.setExtractionOptions(videoTrack.id);\n mp4.start();\n };\n\n mp4.onSamples = (trackId, user, samples) => {\n for (const sample of samples) {\n const chunk = new EncodedVideoChunk({\n type: sample.is_sync ? 'key' : 'delta',\n timestamp: sample.cts * 1_000_000 / sample.timescale,\n duration: sample.duration * 1_000_000 / sample.timescale,\n data: sample.data,\n });\n decoder.decode(chunk);\n }\n\n decoder.flush().then(() => {\n decoder.close();\n resolve(frames);\n });\n };\n\n // Feed the MP4 data\n videoBuffer.fileStart = 0;\n mp4.appendBuffer(videoBuffer);\n mp4.flush();\n });\n}\n```\n\n## Real-Time Video Effects Pipeline\n\nHere's where WebCodecs gets really interesting, building a real-time effects pipeline that takes camera input, applies GPU-accelerated effects via WebGL, and encodes the result:\n\n```javascript\nclass VideoEffectsPipeline {\n constructor(outputCanvas) {\n this.outputCanvas = outputCanvas;\n this.gl = outputCanvas.getContext('webgl2');\n this.encoder = null;\n this.running = false;\n\n this.setupShaders();\n }\n\n setupShaders() {\n const gl = this.gl;\n\n // Fragment shader for a sepia + vignette effect\n const fragmentShader = `#version 300 es\n precision highp float;\n in vec2 v_texCoord;\n out vec4 outColor;\n uniform sampler2D u_texture;\n uniform float u_time;\n\n void main() {\n vec4 color = texture(u_texture, v_texCoord);\n\n // Sepia tone\n float gray = dot(color.rgb, vec3(0.299, 0.587, 0.114));\n vec3 sepia = vec3(gray) * vec3(1.2, 1.0, 0.8);\n\n // Vignette\n vec2 center = v_texCoord - 0.5;\n float dist = length(center);\n float vignette = smoothstep(0.7, 0.4, dist);\n\n // Animated chromatic aberration\n float offset = sin(u_time * 2.0) * 0.002;\n float r = texture(u_texture, v_texCoord + vec2(offset, 0.0)).r;\n float b = texture(u_texture, v_texCoord - vec2(offset, 0.0)).b;\n\n outColor = vec4(\n mix(r, sepia.r, 0.5) * vignette,\n mix(color.g, sepia.g, 0.5) * vignette,\n mix(b, sepia.b, 0.5) * vignette,\n 1.0\n );\n }\n `;\n\n this.program = this.compileShaderProgram(fragmentShader);\n this.timeUniform = gl.getUniformLocation(this.program, 'u_time');\n }\n\n async start(mediaStream) {\n const videoTrack = mediaStream.getVideoTracks()[0];\n const { width, height } = videoTrack.getSettings();\n\n // Create a MediaStreamTrackProcessor to get VideoFrames\n const processor = new MediaStreamTrackProcessor({ track: videoTrack });\n const reader = processor.readable.getReader();\n\n // Setup encoder for output\n this.encoder = new VideoEncoder({\n output: (chunk, meta) => this.handleEncodedChunk(chunk, meta),\n error: (e) => console.error('Encode error:', e),\n });\n\n this.encoder.configure({\n codec: 'avc1.42001f',\n width,\n height,\n bitrate: 3_000_000,\n framerate: 30,\n latencyMode: 'realtime', // Low latency for live processing\n });\n\n this.running = true;\n let frameCount = 0;\n const startTime = performance.now();\n\n while (this.running) {\n const { value: frame, done } = await reader.read();\n if (done) break;\n\n // Apply WebGL effect\n const processedFrame = this.applyEffect(frame, frameCount);\n frame.close(); // Release original frame\n\n // Encode the processed frame\n const keyFrame = frameCount % 90 === 0;\n this.encoder.encode(processedFrame, { keyFrame });\n processedFrame.close();\n\n frameCount++;\n }\n\n await this.encoder.flush();\n this.encoder.close();\n }\n\n applyEffect(inputFrame, frameIndex) {\n const gl = this.gl;\n const { displayWidth: w, displayHeight: h } = inputFrame;\n\n this.outputCanvas.width = w;\n this.outputCanvas.height = h;\n gl.viewport(0, 0, w, h);\n\n gl.useProgram(this.program);\n gl.uniform1f(this.timeUniform, frameIndex / 30.0);\n\n // Upload VideoFrame as texture, zero-copy on supported browsers\n const texture = gl.createTexture();\n gl.bindTexture(gl.TEXTURE_2D, texture);\n gl.texImage2D(\n gl.TEXTURE_2D, 0, gl.RGBA,\n gl.RGBA, gl.UNSIGNED_BYTE,\n inputFrame // VideoFrame directly as texture source!\n );\n gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);\n\n // Draw fullscreen quad\n gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);\n\n // Create new VideoFrame from the processed canvas\n const outputFrame = new VideoFrame(this.outputCanvas, {\n timestamp: inputFrame.timestamp,\n duration: inputFrame.duration,\n });\n\n gl.deleteTexture(texture);\n return outputFrame;\n }\n\n stop() {\n this.running = false;\n }\n}\n\n// Usage\nconst canvas = document.getElementById('output');\nconst pipeline = new VideoEffectsPipeline(canvas);\n\nconst stream = await navigator.mediaDevices.getUserMedia({\n video: { width: 1920, height: 1080, frameRate: 30 }\n});\n\nawait pipeline.start(stream);\n```\n\nThe magic line is ` gl.texImage2D(..., inputFrame) `. On Chrome, when the VideoFrame's underlying buffer is a GPU texture (which it is when it comes from a camera or decoder), this is a zero-copy operation, the GPU texture is shared directly with WebGL. No CPU-side pixel copying.\n\n## AV1 Encoding: The Next Generation\n\nAV1 support in WebCodecs is maturing rapidly. Here's how to use it with proper feature detection:\n\n```javascript\nasync function getOptimalCodec(width, height) {\n // Try AV1 first (best compression)\n const av1Config = {\n codec: 'av01.0.08M.08', // AV1 Main Profile, Level 4.0, 8-bit\n width,\n height,\n bitrate: 3_000_000,\n framerate: 30,\n };\n\n const av1Support = await VideoEncoder.isConfigSupported(av1Config);\n if (av1Support.supported) {\n // Check if it's hardware-accelerated\n // (no direct API, but we can infer from encode speed)\n return { ...av1Config, label: 'AV1 (HW)' };\n }\n\n // Fall back to VP9\n const vp9Config = {\n codec: 'vp09.00.31.08', // VP9 Profile 0, Level 3.1, 8-bit\n width,\n height,\n bitrate: 4_000_000, // VP9 needs more bitrate for same quality\n framerate: 30,\n };\n\n const vp9Support = await VideoEncoder.isConfigSupported(vp9Config);\n if (vp9Support.supported) {\n return { ...vp9Config, label: 'VP9' };\n }\n\n // H.264 as last resort, universally supported\n return {\n codec: 'avc1.42001f',\n width,\n height,\n bitrate: 5_000_000,\n framerate: 30,\n avc: { format: 'annexb' },\n label: 'H.264',\n };\n}\n```\n\n## Performance Comparison\n\nEncoding 1080p30 video for 60 seconds on a 2024 MacBook Pro (M3 Pro):\n\n| Method | Encode Time | Memory Peak | CPU Usage |\n| --- | --- | --- | --- |\n| FFmpeg.wasm (H.264) | 4 min 12s | 890MB | 100% (1 core) |\n| WebCodecs (H.264 HW) | 6.2s | 125MB | 15% |\n| WebCodecs (VP9 HW) | 8.1s | 130MB | 18% |\n| WebCodecs (AV1 HW) | 11.4s | 145MB | 22% |\n| WebCodecs (AV1 SW fallback) | 3 min 38s | 310MB | 95% (1 core) |\n\nThe hardware acceleration numbers are staggering. 6.2 seconds vs 4 minutes for H.264 encoding of the same content. And note the memory, 125MB vs 890MB. The FFmpeg.wasm approach loads the entire binary plus maintains its own memory heap for frame buffers.\n\nThe AV1 software fallback row is instructive, without hardware support, AV1 encoding is brutal. Always check ` isConfigSupported() ` and fall back gracefully.\n\n## The Muxing Problem\n\nWebCodecs gives you raw encoded chunks, but not a container format. You need a muxer to wrap them into MP4, WebM, or whatever container your use case requires:\n\n```javascript\nimport { Muxer, ArrayBufferTarget } from 'mp4-muxer';\n\nfunction muxToMp4(encodedChunks, config) {\n const target = new ArrayBufferTarget();\n\n const muxer = new Muxer({\n target,\n video: {\n codec: 'avc',\n width: config.width,\n height: config.height,\n },\n fastStart: 'in-memory', // moov atom at the beginning for streaming\n });\n\n for (const chunk of encodedChunks) {\n muxer.addVideoChunk(\n new EncodedVideoChunk({\n type: chunk.type,\n timestamp: chunk.timestamp,\n duration: chunk.duration,\n data: chunk.data,\n }),\n chunk.decoderConfig ? {\n decoderConfig: chunk.decoderConfig\n } : undefined\n );\n }\n\n muxer.finalize();\n\n // Download the MP4\n const blob = new Blob([target.buffer], { type: 'video/mp4' });\n const url = URL.createObjectURL(blob);\n const a = document.createElement('a');\n a.href = url;\n a.download = 'output.mp4';\n a.click();\n}\n```\n\nLibraries like ` mp4-muxer ` and ` webm-muxer ` by Vani Wasek handle this well. They're pure JavaScript, run in Workers, and produce spec-compliant containers. Writing a custom muxer is generally not recommended unless there are very specific requirements, as the ISO 14496-12 specification is complex.\n\n## Browser Support and Gotchas\n\nWebCodecs is available in Chrome 94+, Edge 94+, and Opera 80+. Safari added support in 16.4 but with significant limitations, no hardware AV1, inconsistent VideoFrame-to-texture behavior. Firefox has it behind a flag as of early 2026.\n\nThe biggest gotcha: **VideoFrame timestamps are in microseconds, not milliseconds.** This bug appears frequently across codebases. You multiply by 1000 instead of 1,000,000 and your encoded video plays back at 1000x speed. The encoder doesn't complain, it just produces a video where 60 seconds of content plays in 60 milliseconds.\n\n```javascript\n// WRONG, timestamps in milliseconds\nconst frame = new VideoFrame(canvas, {\n timestamp: Date.now(), // ← milliseconds, will be interpreted as microseconds\n});\n\n// CORRECT, timestamps in microseconds\nconst frame = new VideoFrame(canvas, {\n timestamp: performance.now() * 1000, // ← convert ms to μs\n});\n```\n\nWebCodecs represents a significant advancement for the web platform. For the first time, we can do real video processing in the browser without loading a multi-megabyte WASM binary, without pegging the CPU, and with quality that matches native applications. The video editing, streaming, and communication tools being built on this API are going to be wild.\n\n## Frequently asked questions\n\n### What is the WebCodecs API?\n\nWebCodecs is a browser API that provides low-level access to hardware video and audio encoders and decoders. Unlike MediaRecorder or video elements, WebCodecs gives you frame-by-frame control over encoding parameters, codec selection, and output format, without loading a WASM-based FFmpeg.\n\n### Can I encode AV1 video in the browser with WebCodecs?\n\nYes, if the user's hardware supports it. WebCodecs can use hardware AV1 encoders on machines with Intel Gen12+, AMD RDNA3+, or NVIDIA Ada Lovelace+ GPUs. You should always check support with VideoEncoder.isConfigSupported() and fall back to VP9 or H.264.\n\n### How does WebCodecs compare to FFmpeg.wasm?\n\nWebCodecs uses the browser's native codec implementations (often hardware-accelerated), while FFmpeg.wasm runs a full FFmpeg binary compiled to WebAssembly. WebCodecs is faster, uses less memory, and doesn't require downloading a 25MB+ WASM binary. But FFmpeg.wasm supports more formats and offers more processing flexibility.\n\n## Tips\n\n1. **Tip:** Always check ` VideoEncoder.isConfigSupported() ` before creating your encoder, hardware AV1 encoding support varies wildly. Chrome on a machine with Intel Arc GPU supports it, the same Chrome on an older NVIDIA card doesn't. Fail gracefully to VP9 or H.264.\n2. **Tip:** Move your VideoEncoder to a dedicated Web Worker. Encoding even a single 1080p frame blocks the main thread for 2-8ms depending on the codec. At 30fps that's 60-240ms per second of encoding jank that kills your UI responsiveness.\n3. **Tip:** Set ` latencyMode: 'realtime' ` on your VideoEncoder config for live streaming use cases. The default ` 'quality' ` mode buffers frames for better compression, adding 100-300ms of latency that's unacceptable for video calls or live broadcasting.\n\n- webcodecs\n- video-processing\n- av1\n- browser-api\n- video-encoder\n- hardware-acceleration\n- webworkers\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#title"},{"level":2,"text":"WebCodecs Overview","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#webcodecs-overview"},{"level":2,"text":"Basic Encoding Pipeline","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#basic-encoding-pipeline"},{"level":2,"text":"Decoding: Reading Video Frame-by-Frame","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#decoding-reading-video-frame-by-frame"},{"level":2,"text":"Real-Time Video Effects Pipeline","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#real-time-video-effects-pipeline"},{"level":2,"text":"AV1 Encoding: The Next Generation","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#av1-encoding-the-next-generation"},{"level":2,"text":"Performance Comparison","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#performance-comparison"},{"level":2,"text":"The Muxing Problem","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#the-muxing-problem"},{"level":2,"text":"Browser Support and Gotchas","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#browser-support-and-gotchas"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#questions"},{"level":3,"text":"What is the WebCodecs API?","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#question-1"},{"level":3,"text":"Can I encode AV1 video in the browser with WebCodecs?","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#question-2"},{"level":3,"text":"How does WebCodecs compare to FFmpeg.wasm?","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#article-tips"}],"sections":[{"heading":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser","level":1,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser","content":"Web APIs"},{"heading":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser","level":1,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#title","content":"2026-03-1510 min read\n\n[Lucio Durán](<https://lucioduran.com/>)\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/webcodecs-api-video-processing-browser>)"},{"heading":"WebCodecs Overview","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#webcodecs-overview","content":"WebCodecs sits below the high-level media APIs (` <video> `, MediaRecorder, MediaStream) and above the raw byte manipulation level. It gives you:\n\n- **VideoDecoder**, Decode compressed video frames (H.264, VP8, VP9, AV1) into raw ` VideoFrame ` objects\n- **VideoEncoder**, Encode raw ` VideoFrame ` objects into compressed chunks\n- **AudioDecoder/AudioEncoder**, Same thing for audio (AAC, Opus, FLAC)\n- **VideoFrame**, A raw video frame that can come from a canvas, camera, or decoder\n- **EncodedVideoChunk**, A compressed video frame ready for muxing\n\nThe key insight: these are **hardware-accelerated**. When you create a VideoEncoder for H.264, the browser delegates to the GPU's dedicated encoding ASIC (NVENC on NVIDIA, VCE on AMD, Quick Sync on Intel). You get the same encoding speed as a native application."},{"heading":"Basic Encoding Pipeline","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#basic-encoding-pipeline","content":"Let's start with the fundamentals, encoding a canvas animation to an H.264 video:\n\n```javascript\n// Step 1: Check codec support\nconst support = await VideoEncoder.isConfigSupported({\n codec: 'avc1.42001f', // H.264 Baseline Level 3.1\n width: 1920,\n height: 1080,\n bitrate: 5_000_000, // 5 Mbps\n framerate: 30,\n});\n\nif (!support.supported) {\n console.error('H.264 encoding not supported');\n // Fall back to a supported codec\n}\n\n// Step 2: Collect encoded chunks\nconst chunks = [];\nconst encoder = new VideoEncoder({\n output: (chunk, metadata) => {\n // chunk is an EncodedVideoChunk\n const buffer = new ArrayBuffer(chunk.byteLength);\n chunk.copyTo(buffer);\n\n chunks.push({\n type: chunk.type, // 'key' or 'delta'\n timestamp: chunk.timestamp,\n duration: chunk.duration,\n data: buffer,\n // metadata.decoderConfig available on keyframes\n decoderConfig: metadata?.decoderConfig || null,\n });\n },\n error: (e) => console.error('Encoder error:', e),\n});\n\n// Step 3: Configure the encoder\nencoder.configure({\n codec: 'avc1.42001f',\n width: 1920,\n height: 1080,\n bitrate: 5_000_000,\n framerate: 30,\n latencyMode: 'quality', // Buffer frames for better compression\n avc: {\n format: 'annexb', // Raw H.264 NAL units (vs 'avc' for MP4-style)\n },\n});\n\n// Step 4: Encode frames from a canvas\nconst canvas = document.getElementById('my-canvas');\nconst ctx = canvas.getContext('2d');\n\nfor (let i = 0; i < 300; i++) { // 10 seconds at 30fps\n // Draw frame i on canvas\n renderFrame(ctx, i);\n\n // Create VideoFrame from canvas\n const frame = new VideoFrame(canvas, {\n timestamp: i * (1_000_000 / 30), // Microseconds!\n duration: 1_000_000 / 30,\n });\n\n // Encode the frame\n const keyFrame = i % 90 === 0; // Keyframe every 3 seconds\n encoder.encode(frame, { keyFrame });\n\n // CRITICAL: close the frame to release GPU memory\n frame.close();\n}\n\n// Step 5: Flush remaining frames\nawait encoder.flush();\nencoder.close();\n\n// chunks[] now contains all encoded H.264 data\n// You need a muxer (like mp4-muxer) to wrap it in a container\n```\n\nThat ` frame.close() ` call is essential. VideoFrames hold GPU texture references. If you forget to close them, you'll leak GPU memory and eventually the browser kills your tab. Without proper frame lifecycle management, video processing applications commonly crash after ~200 frames."},{"heading":"Decoding: Reading Video Frame-by-Frame","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#decoding-reading-video-frame-by-frame","content":"The decoder side is equally powerful. Here's how to decode an MP4 and get individual frames:\n\n```javascript\n// You need to demux the MP4 first, WebCodecs doesn't handle containers\n// Using mp4box.js for demuxing\nimport MP4Box from 'mp4box';\n\nasync function decodeVideo(videoBuffer) {\n const frames = [];\n\n const decoder = new VideoDecoder({\n output: (frame) => {\n // frame is a VideoFrame, raw pixels accessible via\n // canvas drawing, ImageBitmap, or copyTo()\n frames.push(frame);\n // Don't close here if you want to keep the frames!\n },\n error: (e) => console.error('Decoder error:', e),\n });\n\n // Demux the MP4 to get codec config and encoded chunks\n const mp4 = MP4Box.createFile();\n\n return new Promise((resolve) => {\n mp4.onReady = (info) => {\n const videoTrack = info.tracks.find(t => t.type === 'video');\n\n // Configure decoder with codec info from the container\n decoder.configure({\n codec: videoTrack.codec, // e.g., 'avc1.64001f'\n codedWidth: videoTrack.video.width,\n codedHeight: videoTrack.video.height,\n description: getAVCDecoderConfig(videoTrack),\n });\n\n mp4.setExtractionOptions(videoTrack.id);\n mp4.start();\n };\n\n mp4.onSamples = (trackId, user, samples) => {\n for (const sample of samples) {\n const chunk = new EncodedVideoChunk({\n type: sample.is_sync ? 'key' : 'delta',\n timestamp: sample.cts * 1_000_000 / sample.timescale,\n duration: sample.duration * 1_000_000 / sample.timescale,\n data: sample.data,\n });\n decoder.decode(chunk);\n }\n\n decoder.flush().then(() => {\n decoder.close();\n resolve(frames);\n });\n };\n\n // Feed the MP4 data\n videoBuffer.fileStart = 0;\n mp4.appendBuffer(videoBuffer);\n mp4.flush();\n });\n}\n```"},{"heading":"Real-Time Video Effects Pipeline","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#real-time-video-effects-pipeline","content":"Here's where WebCodecs gets really interesting, building a real-time effects pipeline that takes camera input, applies GPU-accelerated effects via WebGL, and encodes the result:\n\n```javascript\nclass VideoEffectsPipeline {\n constructor(outputCanvas) {\n this.outputCanvas = outputCanvas;\n this.gl = outputCanvas.getContext('webgl2');\n this.encoder = null;\n this.running = false;\n\n this.setupShaders();\n }\n\n setupShaders() {\n const gl = this.gl;\n\n // Fragment shader for a sepia + vignette effect\n const fragmentShader = `#version 300 es\n precision highp float;\n in vec2 v_texCoord;\n out vec4 outColor;\n uniform sampler2D u_texture;\n uniform float u_time;\n\n void main() {\n vec4 color = texture(u_texture, v_texCoord);\n\n // Sepia tone\n float gray = dot(color.rgb, vec3(0.299, 0.587, 0.114));\n vec3 sepia = vec3(gray) * vec3(1.2, 1.0, 0.8);\n\n // Vignette\n vec2 center = v_texCoord - 0.5;\n float dist = length(center);\n float vignette = smoothstep(0.7, 0.4, dist);\n\n // Animated chromatic aberration\n float offset = sin(u_time * 2.0) * 0.002;\n float r = texture(u_texture, v_texCoord + vec2(offset, 0.0)).r;\n float b = texture(u_texture, v_texCoord - vec2(offset, 0.0)).b;\n\n outColor = vec4(\n mix(r, sepia.r, 0.5) * vignette,\n mix(color.g, sepia.g, 0.5) * vignette,\n mix(b, sepia.b, 0.5) * vignette,\n 1.0\n );\n }\n `;\n\n this.program = this.compileShaderProgram(fragmentShader);\n this.timeUniform = gl.getUniformLocation(this.program, 'u_time');\n }\n\n async start(mediaStream) {\n const videoTrack = mediaStream.getVideoTracks()[0];\n const { width, height } = videoTrack.getSettings();\n\n // Create a MediaStreamTrackProcessor to get VideoFrames\n const processor = new MediaStreamTrackProcessor({ track: videoTrack });\n const reader = processor.readable.getReader();\n\n // Setup encoder for output\n this.encoder = new VideoEncoder({\n output: (chunk, meta) => this.handleEncodedChunk(chunk, meta),\n error: (e) => console.error('Encode error:', e),\n });\n\n this.encoder.configure({\n codec: 'avc1.42001f',\n width,\n height,\n bitrate: 3_000_000,\n framerate: 30,\n latencyMode: 'realtime', // Low latency for live processing\n });\n\n this.running = true;\n let frameCount = 0;\n const startTime = performance.now();\n\n while (this.running) {\n const { value: frame, done } = await reader.read();\n if (done) break;\n\n // Apply WebGL effect\n const processedFrame = this.applyEffect(frame, frameCount);\n frame.close(); // Release original frame\n\n // Encode the processed frame\n const keyFrame = frameCount % 90 === 0;\n this.encoder.encode(processedFrame, { keyFrame });\n processedFrame.close();\n\n frameCount++;\n }\n\n await this.encoder.flush();\n this.encoder.close();\n }\n\n applyEffect(inputFrame, frameIndex) {\n const gl = this.gl;\n const { displayWidth: w, displayHeight: h } = inputFrame;\n\n this.outputCanvas.width = w;\n this.outputCanvas.height = h;\n gl.viewport(0, 0, w, h);\n\n gl.useProgram(this.program);\n gl.uniform1f(this.timeUniform, frameIndex / 30.0);\n\n // Upload VideoFrame as texture, zero-copy on supported browsers\n const texture = gl.createTexture();\n gl.bindTexture(gl.TEXTURE_2D, texture);\n gl.texImage2D(\n gl.TEXTURE_2D, 0, gl.RGBA,\n gl.RGBA, gl.UNSIGNED_BYTE,\n inputFrame // VideoFrame directly as texture source!\n );\n gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);\n\n // Draw fullscreen quad\n gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);\n\n // Create new VideoFrame from the processed canvas\n const outputFrame = new VideoFrame(this.outputCanvas, {\n timestamp: inputFrame.timestamp,\n duration: inputFrame.duration,\n });\n\n gl.deleteTexture(texture);\n return outputFrame;\n }\n\n stop() {\n this.running = false;\n }\n}\n\n// Usage\nconst canvas = document.getElementById('output');\nconst pipeline = new VideoEffectsPipeline(canvas);\n\nconst stream = await navigator.mediaDevices.getUserMedia({\n video: { width: 1920, height: 1080, frameRate: 30 }\n});\n\nawait pipeline.start(stream);\n```\n\nThe magic line is ` gl.texImage2D(..., inputFrame) `. On Chrome, when the VideoFrame's underlying buffer is a GPU texture (which it is when it comes from a camera or decoder), this is a zero-copy operation, the GPU texture is shared directly with WebGL. No CPU-side pixel copying."},{"heading":"AV1 Encoding: The Next Generation","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#av1-encoding-the-next-generation","content":"AV1 support in WebCodecs is maturing rapidly. Here's how to use it with proper feature detection:\n\n```javascript\nasync function getOptimalCodec(width, height) {\n // Try AV1 first (best compression)\n const av1Config = {\n codec: 'av01.0.08M.08', // AV1 Main Profile, Level 4.0, 8-bit\n width,\n height,\n bitrate: 3_000_000,\n framerate: 30,\n };\n\n const av1Support = await VideoEncoder.isConfigSupported(av1Config);\n if (av1Support.supported) {\n // Check if it's hardware-accelerated\n // (no direct API, but we can infer from encode speed)\n return { ...av1Config, label: 'AV1 (HW)' };\n }\n\n // Fall back to VP9\n const vp9Config = {\n codec: 'vp09.00.31.08', // VP9 Profile 0, Level 3.1, 8-bit\n width,\n height,\n bitrate: 4_000_000, // VP9 needs more bitrate for same quality\n framerate: 30,\n };\n\n const vp9Support = await VideoEncoder.isConfigSupported(vp9Config);\n if (vp9Support.supported) {\n return { ...vp9Config, label: 'VP9' };\n }\n\n // H.264 as last resort, universally supported\n return {\n codec: 'avc1.42001f',\n width,\n height,\n bitrate: 5_000_000,\n framerate: 30,\n avc: { format: 'annexb' },\n label: 'H.264',\n };\n}\n```"},{"heading":"Performance Comparison","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#performance-comparison","content":"Encoding 1080p30 video for 60 seconds on a 2024 MacBook Pro (M3 Pro):\n\n| Method | Encode Time | Memory Peak | CPU Usage |\n| --- | --- | --- | --- |\n| FFmpeg.wasm (H.264) | 4 min 12s | 890MB | 100% (1 core) |\n| WebCodecs (H.264 HW) | 6.2s | 125MB | 15% |\n| WebCodecs (VP9 HW) | 8.1s | 130MB | 18% |\n| WebCodecs (AV1 HW) | 11.4s | 145MB | 22% |\n| WebCodecs (AV1 SW fallback) | 3 min 38s | 310MB | 95% (1 core) |\n\nThe hardware acceleration numbers are staggering. 6.2 seconds vs 4 minutes for H.264 encoding of the same content. And note the memory, 125MB vs 890MB. The FFmpeg.wasm approach loads the entire binary plus maintains its own memory heap for frame buffers.\n\nThe AV1 software fallback row is instructive, without hardware support, AV1 encoding is brutal. Always check ` isConfigSupported() ` and fall back gracefully."},{"heading":"The Muxing Problem","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#the-muxing-problem","content":"WebCodecs gives you raw encoded chunks, but not a container format. You need a muxer to wrap them into MP4, WebM, or whatever container your use case requires:\n\n```javascript\nimport { Muxer, ArrayBufferTarget } from 'mp4-muxer';\n\nfunction muxToMp4(encodedChunks, config) {\n const target = new ArrayBufferTarget();\n\n const muxer = new Muxer({\n target,\n video: {\n codec: 'avc',\n width: config.width,\n height: config.height,\n },\n fastStart: 'in-memory', // moov atom at the beginning for streaming\n });\n\n for (const chunk of encodedChunks) {\n muxer.addVideoChunk(\n new EncodedVideoChunk({\n type: chunk.type,\n timestamp: chunk.timestamp,\n duration: chunk.duration,\n data: chunk.data,\n }),\n chunk.decoderConfig ? {\n decoderConfig: chunk.decoderConfig\n } : undefined\n );\n }\n\n muxer.finalize();\n\n // Download the MP4\n const blob = new Blob([target.buffer], { type: 'video/mp4' });\n const url = URL.createObjectURL(blob);\n const a = document.createElement('a');\n a.href = url;\n a.download = 'output.mp4';\n a.click();\n}\n```\n\nLibraries like ` mp4-muxer ` and ` webm-muxer ` by Vani Wasek handle this well. They're pure JavaScript, run in Workers, and produce spec-compliant containers. Writing a custom muxer is generally not recommended unless there are very specific requirements, as the ISO 14496-12 specification is complex."},{"heading":"Browser Support and Gotchas","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#browser-support-and-gotchas","content":"WebCodecs is available in Chrome 94+, Edge 94+, and Opera 80+. Safari added support in 16.4 but with significant limitations, no hardware AV1, inconsistent VideoFrame-to-texture behavior. Firefox has it behind a flag as of early 2026.\n\nThe biggest gotcha: **VideoFrame timestamps are in microseconds, not milliseconds.** This bug appears frequently across codebases. You multiply by 1000 instead of 1,000,000 and your encoded video plays back at 1000x speed. The encoder doesn't complain, it just produces a video where 60 seconds of content plays in 60 milliseconds.\n\n```javascript\n// WRONG, timestamps in milliseconds\nconst frame = new VideoFrame(canvas, {\n timestamp: Date.now(), // ← milliseconds, will be interpreted as microseconds\n});\n\n// CORRECT, timestamps in microseconds\nconst frame = new VideoFrame(canvas, {\n timestamp: performance.now() * 1000, // ← convert ms to μs\n});\n```\n\nWebCodecs represents a significant advancement for the web platform. For the first time, we can do real video processing in the browser without loading a multi-megabyte WASM binary, without pegging the CPU, and with quality that matches native applications. The video editing, streaming, and communication tools being built on this API are going to be wild."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#questions","content":""},{"heading":"What is the WebCodecs API?","level":3,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#question-1","content":"WebCodecs is a browser API that provides low-level access to hardware video and audio encoders and decoders. Unlike MediaRecorder or video elements, WebCodecs gives you frame-by-frame control over encoding parameters, codec selection, and output format, without loading a WASM-based FFmpeg."},{"heading":"Can I encode AV1 video in the browser with WebCodecs?","level":3,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#question-2","content":"Yes, if the user's hardware supports it. WebCodecs can use hardware AV1 encoders on machines with Intel Gen12+, AMD RDNA3+, or NVIDIA Ada Lovelace+ GPUs. You should always check support with VideoEncoder.isConfigSupported() and fall back to VP9 or H.264."},{"heading":"How does WebCodecs compare to FFmpeg.wasm?","level":3,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#question-3","content":"WebCodecs uses the browser's native codec implementations (often hardware-accelerated), while FFmpeg.wasm runs a full FFmpeg binary compiled to WebAssembly. WebCodecs is faster, uses less memory, and doesn't require downloading a 25MB+ WASM binary. But FFmpeg.wasm supports more formats and offers more processing flexibility."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#article-tips","content":"1. **Tip:** Always check ` VideoEncoder.isConfigSupported() ` before creating your encoder, hardware AV1 encoding support varies wildly. Chrome on a machine with Intel Arc GPU supports it, the same Chrome on an older NVIDIA card doesn't. Fail gracefully to VP9 or H.264.\n2. **Tip:** Move your VideoEncoder to a dedicated Web Worker. Encoding even a single 1080p frame blocks the main thread for 2-8ms depending on the codec. At 30fps that's 60-240ms per second of encoding jank that kills your UI responsiveness.\n3. **Tip:** Set ` latencyMode: 'realtime' ` on your VideoEncoder config for live streaming use cases. The default ` 'quality' ` mode buffers frames for better compression, adding 100-300ms of latency that's unacceptable for video calls or live broadcasting.\n\n- webcodecs\n- video-processing\n- av1\n- browser-api\n- video-encoder\n- hardware-acceleration\n- webworkers\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"Español","url":"https://lucioduran.com/es/blog/webcodecs-api-video-processing-browser"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},{"@type":"WebSite","@id":"https://lucioduran.com/#website","url":"https://lucioduran.com/","name":"Lucio Durán","inLanguage":["en","es"],"publisher":{"@id":"https://lucioduran.com/#person"}},{"@type":"WebPage","@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#page","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser","name":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser | Lucio Durán","description":"Low-level video encoding and decoding without [FFmpeg.wasm](https://ffmpegwasm.netlify.app/), VideoEncoder, VideoDecoder, AV1 encoding, and building a real-time video effects pipeline entirely in the browser.","inLanguage":"en","author":{"@id":"https://lucioduran.com/#person"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"},"isAccessibleForFree":true,"datePublished":"2026-03-15","mainEntity":{"@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#article"},"breadcrumb":{"@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#breadcrumb"}},{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#article","mainEntityOfPage":{"@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#page"},"headline":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser","description":"Low-level video encoding and decoding without [FFmpeg.wasm](https://ffmpegwasm.netlify.app/), VideoEncoder, VideoDecoder, AV1 encoding, and building a real-time video effects pipeline entirely in the browser.","datePublished":"2026-03-15","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@id":"https://lucioduran.com/#person"},"url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Web APIs","keywords":"webcodecs, video-processing, av1, browser-api, video-encoder, hardware-acceleration, webworkers","wordCount":1793,"timeRequired":"PT10M","workTranslation":[{"@type":"BlogPosting","@id":"https://lucioduran.com/es/blog/webcodecs-api-video-processing-browser#article","inLanguage":"es","url":"https://lucioduran.com/es/blog/webcodecs-api-video-processing-browser"}],"isAccessibleForFree":true},{"@type":"Blog","@id":"https://lucioduran.com/blog#blog","url":"https://lucioduran.com/blog","name":"Blog","inLanguage":"en","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"WebCodecs API: Hardware-Accelerated Video Encoding and Decoding in the Browser","item":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser"}],"@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#breadcrumb"},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is the WebCodecs API?","acceptedAnswer":{"@type":"Answer","text":"WebCodecs is a browser API that provides low-level access to hardware video and audio encoders and decoders. Unlike MediaRecorder or video elements, WebCodecs gives you frame-by-frame control over encoding parameters, codec selection, and output format, without loading a WASM-based FFmpeg."}},{"@type":"Question","name":"Can I encode AV1 video in the browser with WebCodecs?","acceptedAnswer":{"@type":"Answer","text":"Yes, if the user's hardware supports it. WebCodecs can use hardware AV1 encoders on machines with Intel Gen12+, AMD RDNA3+, or NVIDIA Ada Lovelace+ GPUs. You should always check support with VideoEncoder.isConfigSupported() and fall back to VP9 or H.264."}},{"@type":"Question","name":"How does WebCodecs compare to FFmpeg.wasm?","acceptedAnswer":{"@type":"Answer","text":"WebCodecs uses the browser's native codec implementations (often hardware-accelerated), while FFmpeg.wasm runs a full FFmpeg binary compiled to WebAssembly. WebCodecs is faster, uses less memory, and doesn't require downloading a 25MB+ WASM binary. But FFmpeg.wasm supports more formats and offers more processing flexibility."}}],"@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#questions","url":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#questions","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser#page"},"author":{"@id":"https://lucioduran.com/#person"}}]}],"contentHash":"3bcbfab9f5c8b30cb1ebe0c57d8f9ccadcfa5271ca55196fc67b845259f712f9","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/webcodecs-api-video-processing-browser","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":73797,"contentBytes":15972}}