feat(windows): canvas-media, WebGPU video frames and MediaElementAudioSourceNode - #157
Merged
Merged
Conversation
…bGLShader - CanvasRenderingContext2D.fillText(text, x, y, maxWidth?): maxWidth is optional, as on the web (strokeText already was). - WebGLRenderingContext.deleteShader(shader: WebGLShader | null) was declared as taking a WebGLRenderbuffer; null is a no-op, as in WebGL (it threw on `shader.native`).
canvas-media's Video, Audio and VideoFrame on NativeScript Windows, over a WinRT MediaPlayer. - crates/canvas-media-napi (canvasmedia.node): NSCMediaPlayerBridge subscribes the player's events natively (Media Foundation raises them on its own threads, where the runtime cannot run JS delegates) and delivers them through threadsafe functions. Video players run in frame-server mode: each frame is copied (CopyFrameToVideoSurface) into a ring of BGRA textures on the module's D3D11 device, presented into the view's XAML SurfaceImageSource and read back (RGBA) on demand. Frames are also shareable with WebGPU (gpuFrame(): NT handles plus ready / release fences) and the audio can be tapped (createAudioTap(): an IBasicAudioEffect the module serves as an in-process WinRT class, registered in the app manifest by plugin.targets). - Exit safety: a MediaPlayer released during process exit (after ExitProcess has killed Media Foundation's threads, e.g. from a thread-local destructor) spins forever in the graphics driver and leaves an unkillable process. An env cleanup hook closes every live player while its threads run, nothing releasable lives in TLS, and no frame copy starts after close(). - player-windows.ts: HTMLMediaElement state and events over the player (readyState, play() promises with AbortError / NotSupportedError, timeupdate while playing, seeking / seeked, volumechange, <Source> fallback in order, ms-appx: / file / http sources, canPlayType). - Video: frames shown by an Image; 2D drawImage, WebGL texImage2D / texImage3D / texSubImage3D, WebGPU copyExternalImageToTexture (ImageAsset) and VideoFrame read the current frame once per decoded frame. `controls`: WinUI's transport controls over the frames (the stock MediaPlayerElement template with its presenter kept for layout but invisible: in frame-server mode it only paints black), so canvases keep getting frames. - Audio: WinUI's compact transport controls. - Packaging: platforms/windows plugin.props / plugin.targets, build-napi.sh canvas-media-napi, `make windows-media`, the PR workflow's artifact placement. - CI: canvas-svg, audio-context and canvas-media share one napi-windows matrix job (one entry per module), with the earlier canvas-svg fixes applied to all: no cache on manual runs, the default branch checked out on them, a source key from git tree hashes (Cargo.lock is untracked), and a timeout on the Node-API tests. - demo: a `media` spec suite on Windows (events, drawImage, texImage2D, VideoFrame, copyExternalImageToTexture, importExternalTexture, seeking, errors, audio, createMediaElementSource).
wgpu's DX12 backend supports external textures, so importExternalTexture now works on Windows and copyExternalImageToTexture(video) stays on the GPU, over the frames canvas-media shares. - canvas-c gpu_shared_frame: a CanvasD3DSharedFrame (`nativeTexture`) names a shareable BGRA texture and two shared fences by NT handle. The texture is opened on wgpu's D3D12 device (OpenSharedHandle, texture_from_raw; cached by id), only ever sampled so it stays in COMMON between submits, and the fork's add_wait_fence / add_signal_fence bracket the submit that reads it: wait for the producer's copy, signal when the read is done so the producer may reuse it. A frame from another adapter (hybrid GPUs) is refused, and the callers take the CPU path. - copyExternalImageToTexture: the existing blit, factored into blit_texture (any source and destination texture). - importExternalTexture: the frame is drawn at once into a plane of our own, so the external texture's later uses never read the producer's texture. - canvas-napi: GPUDevice.__getAdapterLuid(); canvas-c canvas_native_webgpu_device_get_adapter_luid. - packages/canvas: GPUDevice.__frameDevice (the MTLDevice on Apple, the adapter LUID on Windows) replaces __metalDevice where videos are asked for GPU frames.
createMediaElementSource(element) on Windows, over the tap canvas-media puts on the element's MediaPlayer (as MTAudioProcessingTap does on iOS and an ExoPlayer AudioProcessor on Android). - canvas-media installs the tap (an audio effect in the player's pipeline) before any source is set, passing the audio through; createMediaElementSource routes it: the element goes silent and its decoded audio, at the element's volume (silent when muted), plays through the graph. It works for <audio> and <video>, and for canvas-polyfill's elements. disposeMediaElementSource gives the element its output back. - audiocontext.node: createMediaElementSourceFromTap(address) reads the tap through the AudioTapSource canvas-media's module exposes (retained for as long as the node lives), as a MediaStreamTrack source: pulled on the render thread, silence when the element is paused. Realtime contexts only, as on the web.
…d audiocontext modules Release builds (release-napi) with the shared video frames, the audio tap and the exit-safe player teardown, so npm_release.yml ships them until the napi-windows / canvas-windows CI jobs provide them.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Drops comments that restate the next lines or a member's name, keeping the whys: threading, the exit-time release hazard, D3D12 resource states, cross-module struct layouts, the frame/tap contracts and platform quirks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
canvas-media on NativeScript Windows, plus the two integrations it enables: video frames that reach WebGPU without the CPU, and
MediaElementAudioSourceNode.canvas-media on Media Foundation
crates/canvas-media-napi(canvasmedia.node) over a WinRTMediaPlayer.MediaPlayerraises its events on Media Foundation threads, where the Windows runtime can't run JS delegates. So the module subscribes to them natively and forwards them to JS through threadsafe functions.SurfaceImageSource, and read back only when a canvas asks.drawImage, WebGLtexImage2D/texImage3D/texSubImage3D,VideoFrame, and WebGPU.controls: WinUI's transport controls over the frames, using the stock template with its presenter kept for layout but invisible. Canvases keep getting frames while controls are shown.player-windows.ts:readyState,play()promises (withAbortError/NotSupportedError),timeupdate,seeking/seeked,volumechange,<Source>fallback, andcanPlayType.MediaPlayerreleased during process exit (afterExitProcesshas killed Media Foundation's threads) spins forever in the graphics driver and leaves an unkillable process. The module now closes live players from an env cleanup hook, and keeps nothing releasable in thread-local storage.WebGPU: video frames without the CPU
wgpu's DX12 backend supports external textures.
CanvasD3DSharedFrame: a shareable texture plus ready/release fences, by NT handle. canvas-c opens it on wgpu's D3D12 device and brackets the reading submit with the fork'sadd_wait_fence/add_signal_fence.copyExternalImageToTexture(video)stays on the GPU.importExternalTexture(video)now works on Windows.GPUDevice.__frameDevice(theMTLDeviceon Apple, the adapter LUID on Windows) replaces__metalDevicewhere videos are asked for GPU frames.MediaElementAudioSourceNode
MTAudioProcessingTap) and Android (ExoPlayerAudioProcessor):canvasmedia.nodeis also the in-process WinRT server of anIBasicAudioEffectclass, which canvas-media'splugin.targetsregisters in the app manifest;createMediaElementSourceroutes it: the element goes silent, and its audio, at the element's volume, plays through the graph.audiocontext.nodereads the tap as a media-stream source.Also
napi-windowsmatrix job, one entry per module. The earlier canvas-svg fixes (no cache on manual runs, default-branch checkout) now apply to all of them. The cache is keyed on git tree hashes.fillText'smaxWidthis optional, anddeleteShadertakes aWebGLShader(ornull).canvasnative.node,canvasmedia.nodeandaudiocontext.node.Testing
mediaspec suite: 11/11. It covers events,drawImage,texImage2D,VideoFrame, zero-copycopyExternalImageToTexture,importExternalTexture, seeking, errors, audio andcreateMediaElementSource.webgpu.videofailures come from @nativescript/core's~/resolution on Windows:knownFolders.currentApp()is the package root, notapp/.