compositing.md

Compositing

How are composited visuals represented?

• Scene is an opaque, immutable representation of composited UI. Each frame, the rendering pipeline produces a scene that is uploaded to the engine for rasterization (via Window.render). A scene can also be rasterized directly into an Image (via Scene.toImage). Generally, a Scene encapsulates a stack of rendering operations (e.g., clips, transforms, effects) and graphics (e.g., pictures, textures, platform views).
• SceneBuilder is lowest level mechanism (within the framework) for assembling a scene. SceneBuilder manages a stack of rendering operations (via SceneBuilder.pushClipRect, SceneBuilder.pushOpacity, SceneBuilder.pop, etc) and associated graphics (via SceneBuilder.addPicture, SceneBuilder.addTexture, SceneBuilder.addRetained, etc). As operations are applied, SceneBuilder produces EngineLayer instances that it both tracks and returns. These may be cached by the framework to optimize subsequent frames (e.g., updating an old layer rather than recreating it, or reusing a previously rasterized layer). Once complete, the final Scene is obtained via SceneBuilder.build.
  • Drawing operations are accumulated in a Picture (via PictureRecorder) which is added to the scene via SceneBuilder.addPicture.
  • External textures (represented by a texture ID established using the platform texture repository) are added to the scene via SceneBuilder.addTexture. This supports a “freeze” parameter so that textures may be paused when animation is paused.
  • Previously rasterized EngineLayers can be reused as an efficiency operation via SceneBuilder.addRetained. This is known as retained rendering.
• Picture is an opaque, immutable representation of a sequence of drawing operations. These operations can be added directly to a Scene (via SceneBuilder.addPicture) or combined into another picture (via Canvas.drawPicture). Pictures can also be rasterized (via Picture.toImage) and provide an estimate of their memory footprint.
• PictureRecorder is an opaque mechanism by which drawing operations are sequenced and transformed into an immutable Picture. Picture recorders are generally managed via a Canvas. Once recording is completed (via PictureRecorder.endRecording), a final Picture is returned and the PictureRecorder (and associated Canvas) are invalidated.

How are layers represented?

• Layer represents one slice of a composited scene. Layers are arranged into a hierarchy with each node potentially affecting the nodes below it (i.e., by applying rendering operations or drawing graphics). Layers are mutable and freely moved around the tree, but must only be used once. Each layer is composited into a Scene using SceneBuilder (via Layer.addToScene); adding the root layer is equivalent to compositing the entire tree (e.g., layers add their children). Note that the entire layer tree must be recomposited when mutated; there is no concept of dirty states.
  • SceneBuilder provides an EngineLayer instance when a layer is added. This handle can be stored in Layer.engineLayer and later used to optimize compositing. For instance, rather than repeating an operation, Layer.addToScene might pass the engine layer instead (via Layer._addToSceneWithRetainedRendering), potentially reusing the rasterized texture from the previous frame. Additionally, an engine layer (i.e., “oldLayer”) can be specified when performing certain rendering operations to allow these to be implemented as inline mutations.
  • Layers provide support for retained rendering (via Layer.markNeedsAddToScene).This flag tracks whether the previous engine layer can be reused during compositing (i.e., whether the layer is unchanged from the last frame). If so, retained rendering allows the entire subtree to be replaced with a previously rasterized bitmap (via SceneBuilder.addRetained). Otherwise, the layer must be recomposited from scratch. Once a layer has been added, this flag is cleared (unless the layer specifies that it must always be composited).
    • All layers must be added to the scene initially. Once added, the cached engine layer may be used for retained rendering. Other layers disable retained rendering altogether (via Layer.alwaysNeedsAddToScene). Generally, when a layer is mutated (i.e., by modifying a property or adding a child), it must be re-added to the scene.
    • If a layer must be added to the scene, all ancestor layers must be added, too. Retained rendering allows a subtree to be replaced with a cached bitmap. If a descendent has changed, this bitmap becomes invalid and therefore the ancestor layer must also be added to the scene (via Layer.updateSubtreeNeedsAddToScene).
      • Note that this property isn’t enforced during painting; the layer tree is only made consistent when the scene is composited (via ContainerLayer.buildScene).
    • A layer’s parent must be re-added when it receives a new engine layer [?]
    • Generally, only container layers use retained rendering?
  • Metadata can be embedded within a layer for later retrieval (via Layer.find and Layer.findAll).
• ContainerLayer manages a list of child layers, inserting them into the composited scene in order. The root layer is a subclass of ContainerLayer (TransformLayer); thus, ContainerLayer is responsible for compositing the full scene (via ContainerLayer.buildScene). This involves making retained rendering state consistent (via ContainerLayer.updateSubtreeNeedsAddToScene), adding all the children to the scene (via ContainerLayer.addChildrenToScene), and marking the container as no longer needing to be added (via Layer._needsAddToScene). ContainerLayer implements a number of child operations (e.g., ContainerLayer.append, ContainerLayer.removeAllChildren), and multiplexes most of its operations across these children (e.g., ContainerLayer.find; note that later children are “above” earlier children).
  • ContainerLayer will use retained rendering for its children provided that they aren’t subject to an offset (i.e., are positioned at the layer’s origin). This is the entrypoint for most retained rendering in the layer tree.
  • ContainerLayer.applyTransform transforms the provided matrix to reflect how a given child will be transformed during compositing. This method assumes all children are positioned at the origin; thus, any layer offsets must be removed and expressed as a transformation (via SceneBuilder.pushTransform or SceneBuilder.pushOffset). Otherwise, the resulting matrix will be inaccurate.
• OffsetLayer is a ContainerLayer subclass that supports efficient repainting (i.e., repaint boundaries). Render objects defining repaint boundaries correspond to OffsetLayers in the layer tree. If the render object doesn’t need to be repainted or is simply being translated, its existing offset layer may be reused. This allows the entire subtree to avoid repainting.
  • OffsetLayer applies any offset as a top-level transform so that descendant nodes are composited at the origin and therefore eligible for retained rendering.
  • OffsetLayer.toImage rasterizes the subtree rooted at this layer (via Scene.toImage). The resulting image, consisting of raw pixel data, is rendered into a bounding rectangle using the specified pixel ratio (ignoring the device’s screen density).
• TransformLayer is a ContainerLayer subclass that applies an arbitrary transform; it also happens to (typically) be the root layer corresponding to the RenderView. Any layer offset (via Layer.addToScene) or explicit offset (via OffsetLayer.offset) is removed and applied as a transformation to ensure that TransformLayer.applyTransform behaves consistently.
• PhysicalModelLayer is the engine by which physical lighting effects (e.g., shadows) are integrated into the composited scene. This class incorporates an elevation, clip region, background color, and shadow color to cast a shadow behind its children (via SceneBuilder.pushPhysicalShape).
• AnnotatedRegionLayer incorporates metadata into the layer tree within a given bounding rectangle. An offset (defaulting to the origin) determines the rectangle’s top left corner and a size (defaulting to the entire layer) represents its dimensions. Hit testing is performed by AnnotatedRegionLayer.find and AnnotatedRegionLayer.findAll. Results appearing deeper in the tree or later in the child list take precedence.
• FollowerLayer and LeaderLayer allow efficient transform linking (e.g., so that one layer appears to scroll with another layer). These layers communicate via LayerLink, passing the leader’s transform (including layerOffset) to the follower. The follower uses this transform to appear where the follower is rendered. CompositedTransformFollower and CompositedTransformTarget widgets create and manage these layers.
• There are a variety of leaf and interior classes making up the layer tree.
  • Leaf nodes generally extend Layer directly. PictureLayer describes a single Picture to be added to the scene. TextureLayer is analogous, describing a texture (by ID) and a bounding rectangle; PlatformViewLayer is nearly identical, applying a view instead of a texture.
  • Interior nodes generally extend ContainerLayer or one of its subclasses. OffsetLayer is key to implementing efficient repainting. ClipPathLayer, ClipRectLayer, ColorFilterLayer, OpacityLayer, etc., apply the corresponding effect by pushing a scene builder state, adding all children (via ContainerLayer.addChildrenToScene, potentially utilizing retained rendering), then popping the state.
• EngineLayer and its various specializations (e.g., OpacityEngineLayer, OffsetEngineLayer) represent opaque handles to backend layers. SceneBuilder produces the appropriate handle for each operation it supports. These may then be used to enable retained rendering and inline updating.

What are the compositing building blocks?

• Canvas provides an interface for performing drawing operations within the context of a single PictureRecorder. That is, all drawing performed by a Canvas is constrained to a single Picture (and PictureLayer). Rendering and drawing operations spanning multiple layers are supported by PaintingContext, which manages Canvas lifecycle based on the compositing requirements of the render tree.
• PaintingContext.repaintCompositedChild is a static method that paints a render object into its own layer. Once nodes are marked dirty for painting, they will be processed during the painting phase of the rendering pipeline (via PipelineOwner.flushPaint). This performs a depth-first traversal of all dirty render objects. Note that only repaint boundaries are ever actually marked dirty; all other nodes traverse their ancestor chain to find the nearest enclosing repaint boundary, which is then marked dirty. The dirty node is processed (via PaintingContext._repaintCompositedChild), retrieving or creating an OffsetLayer to serve as the subtree’s container layer. Next, a new PaintingContext is created (associated with this layer) to facilitate the actual painting (via PaintingContext._paintWithContext).
  • Layers can be attached and detached from the rendering pipeline. If a render object with a detached layer is found to be dirty (by PipelineOwner.flushPaint), the ancestor chain is traversed to find the nearest repaint boundary with an attached (or as yet uncreated) layer. This node is marked dirty to ensure that, when the layer is reattached, it will be repainted as expected (via RenderObject._skippedPaintingOnLayer).
• PaintingContext provides a layer of indirection between the paint method and the underlying canvas to allow render objects to adapt to changing compositing requirements. That is, a render object may introduce a new layer in some cases and re-use an existing layer in others. When this changes, any ancestor render objects will need to adapt to the possibility of a descendent introducing a new layer (via RenderObject.needsCompositing); in particular, this requires certain operations to be implemented by introducing their own layers (e.g., clipping). PaintingContext manages this process and provides a number of methods that encapsulate this decision (e.g., PaintingContext.pushClipPath). PaintingContext also ensures that children introducing repaint boundaries are composited into new layers (via PaintingContext.paintChild). Last, PaintingContext manages the PictureRecorder provided to the Canvas, appending picture layers (i.e., PictureLayer) whenever a recording is completed.
  • PaintingContext is initialized with a ContainerLayer subclass (i.e., the container layer) to serve as the root of the layer tree produced via that context; painting never takes place within this layer, but is instead captured by new PictureLayer instances which are appended as painting proceeds.
  • PaintingContext may manage multiple canvases due to compositing. Each Canvas is associated with a PictureRecorder and a PictureLayer (i.e., the current layer) for the duration of its lifespan. Until a new layer must be added (e.g., to implement a compositing effect or because a repaint boundary is encountered), the same canvas (and PictureRecorder) is used for all render objects.
    • Recording begins the first time the canvas is accessed (via PaintingContext.canvas). This allocates a new picture layer, picture recorder, and canvas instance; the picture layer is appended to the container layer once created.
    • Recording ends when a new layer is introduced directly or indirectly; at this point, the picture is stored in the current layer (via PictureRecorder.endRecording) and the canvas and current layer are cleared. The next time a child is painted, a new canvas, picture recorder, and picture layer will be created.
    • Composited children are painted using a new PaintingContext initialized with a corresponding OffsetLayer. If a composited child doesn’t actually need to be repainted (e.g., because it is only being translated), the OffsetLayer is reused with a new offset.
  • PaintingContext provides a compositing naive API. If compositing is necessary, new layers are automatically pushed to achieve the desired effect (via PaintingContext.pushLayer). Otherwise, the effect is implemented directly using the Canvas (e.g., via PictureRecorder).
    • Pushing a layer ends the current recording, appending the new layer to the container layer. A PaintingContext is created for the new layer (if present, the layer’s existing children are removed since they’re likely outdated). If provided, a painting function is applied using the new context; any painting is contained within the new layer. Subsequent operations with the original PaintingContext will result in a new canvas, layer, and picture recorder being created.
• ClipContext is PaintingContext’s base class. Its primary utility is in providing support for clipping without introducing a new layer (e.g., methods suitable for render objects that do not need compositing).

What do the “needs compositing” bits do?

• Compositing describes the process by which layers are combined by the engine during rasterization. Within the context of the framework, compositing typically refers to the allocation of render objects to layers with respect to painting. Needing compositing does not imply that a render object will be allocated its own layer; instead, it indicates that certain effects must be implemented by introducing a new layer rather than modifying the current one (e.g., applying opacity or clipping). For instance, if a parent first establishes a clip and then paints its child, a decision must be made as to how that clip is implemented: (1) as part of the current layer (i.e., Canvas.clipPath), or (2) by pushing a ContainerLayer (i.e., ClipPathLayer). The “needs compositing” bit is how this information is managed.
  • Conceptually, this is because this node might eventually paint a descendent that pushes a new layer (e.g., because it is a repaint boundary). Thus, any painting that might have non-local consequences must be implemented in a way that would work across layers (i.e., via compositing).
  • Note that this is different than from marking a render object as being a repaint boundary. Doing this causes a new layer to be introduced when painting the child (via PaintingContext.paintChild). This invalidates the needs compositing bits for all the ancestors of the repaint boundary. This might cause some ancestors to introduce new layers when painting, but only if they utilize any non-local operations

How are “needs compositing” bits updated?

• RenderObject.markNeedsCompositingBitsUpdate marks a render object as requiring a compositing bit update (via PipelineOwner. _nodesNeedingCompositingBitsUpdate). If a node is marked dirty, all of its ancestors are marked dirty, too. As an optimization, this walk may be cut off if the current node or the current node’s parent is a repaint boundary (or the parent is already marked dirty).
  • If a node’s compositing bits need updating, it’s possible that it will now introduce a new layer (i.e., it will need compositing). If so, all ancestors will need compositing, too, since they may paint a descendent that introduces a new layer. As described, certain non-local effects will need to be implemented via compositing.
  • The walk may be cut off at repaint boundaries since all ancestors must already have been marked as needing compositing.
  • Adding or removing children (via RenderObject.adoptChild and RenderObject.dropChild) might alter the compositing requirements of the render tree. Similarly, changing the RenderObject.alwaysNeedsCompositing bits would require that the bits be updated.
• During the rendering pipeline, PipelineOwner.flushCompositingBits updates all dirty compositing bits (via RenderObject._updateCompositingBits). A given node needs compositing if (1) it’s a repaint boundary (RenderObject.isRepaintBoundary), (2) it’s marked as always needing compositing (RenderObject.alwaysNeedsCompositing), or (3) any of its descendants need compositing.
  • This process walks all descendants that have been marked dirty, updating their bits according to the above policy. Note that this will typically only walk the path identified when the compositing bits were marked dirty.
  • If a node’s compositing bit is changed, it will be marked dirty for painting (as painting may now require additional compositing).
• If a render object always introduces a layer, it should toggle alwaysNeedsCompositing. If this changes (other than when altering children, which calls this automatically), markNeedsCompositingBitsUpdate should be called.

What are repaint boundaries?

• Certain render objects introduce repaint boundaries within the render tree (via RenderObject.isRepaintBoundary). These render objects are always painted into a new layer, allowing them to paint separately from their parent. This effectively decouples the subtree rooted at the repaint boundary from previously painted nodes.
  • This is useful when part of the UI remains mostly static and part of the UI updates frequently. A repaint boundary helps to avoid repainting the static portion of the UI.
  • Render objects that are repaint boundaries are associated with an OffsetLayer, a special type of layer that can update its position without re-rendering.
• Render objects that form repaint boundaries are handled differently during painting (via PaintingContext.paintChild).
  • If the child doesn’t need painting, the entire subtree is skipped. The corresponding layer is updated to reflect the new offset (via PaintingContext._compositeChild) and appended to the current parent layer.
  • If the child needs painting, the current offset layer is cleared or created (via PaintingContext._repaintCompositedChild) then used for painting (via RenderObject._paintWithContext).
  • Otherwise, painting proceeds through RenderObject._paintWithContext.
• Since repaint boundaries always push new layers, all ancestors must be marked as needing compositing. If any such node utilizes a non-local painting effect, that node will need to introduce a new layer, too.

How is painting managed given that children can push new layers?

• PaintingContext.paintChild manages this process by (1) ending any ongoing recording, (2) creating a new layer for the child, (3) painting the child in the new layer using a new painting context (unless it is a clean repaint boundary, which will instead be used as is), (4) appending that layer to the current layer tree, and (5) creating a new canvas and layer for any future painting with the original painting context.
• Methods in PaintingContext will consulting the compositing bits to determine whether to implement a behavior by inserting a layer or altering the canvas (e.g., clipping).

How can layers make painting more efficient?

• All render objects may be associated with a ContainerLayer (via RenderObject.layer). If defined, this is the last layer that was used to paint the render object (if multiple layers are used, this is the root-most layer).
  • Repaint boundaries are automatically assigned an OffsetLayer (via PaintingContext._repaintCompositedChild). All other layers must specifically set this field when painting.
  • Render objects that do not directly push layers must set this to null.
• ContainerLayer tracks any associated EngineLayer (via ContainerLayer.engineLayer). Certain operations can use this information to allow a layer to be updated rather than rebuilt.
  • Methods accept an “oldLayer” parameter to enable this optimization.
  • Stored engine layers can be reused directly (via SceneBuilder.addRetained); this is retained rendering.

How can the widget tree be manually rasterized?

• A layer can be rasterized to an image via OffsetLayer.toImage. A convenient way to obtain an image of the entire UI is to use this method on the RenderView’s own layer. To obtain a subset of the UI, an appropriate repaint boundary can be inserted and its layer used similarly.

How are clips, transforms, and other effects managed?

• There are two code paths. Layers that do not require compositing handle transforms and clips within their own rendering context (thus, the effects are limited to that single layer), via Canvas. Those that are composited, however, introduce these effects as special ContainerLayers, via PaintingContext.
  • A variety of useful ContainerLayers are included: AnnotatedRegionLayer, which is useful for storing metadata within sections of a layer, BackdropFilterLayer, which allows the background to be blurred or filtered, OpacityLayer, which allows opacity to be varied, and OffsetLayer, which is the key mechanism by which the repaint boundary optimization works.
• Canvas.saveLayer / Canvas.restore allow drawing commands to be grouped such that effects like blending, color filtering, and anti-aliasing can be applied once for the group instead of being incrementally stacked (which could lead to invalid renderings).

How are external textures composited?

• Textures are entirely managed by the engine and referenced by a simple texture ID. Layers depicting textures are associated with this ID and, when composited, integrated with the backend texture. As such, these are painted out-of-band by the engine.
• Texture layers are integrated with the render tree via TextureBox. This is a render box that tracks a texture ID and paints a corresponding TextureLayer. During layout, this box expands to fill the incoming constraints.