CompositionBitmapInterpolationMode Nearest does not work correctly

[Gavin-Williams]

Describe the bug

Nearest neighbor = no interpolation or point sampling, it doesn't mean whatever WinUI is doing.

You can see that the 16x9 texture has irregularly sized pixels. This is completely unusable. The point sampled texture should match the original texture exactly, that is the whole point of point sampling. How would we ever produce a scaled image using WinUI's version of nearest neighbor given its current behavior? I'm not sure even what they've done to muck it up. Have they stretched the texel centers to the edge of the texture?

Steps to reproduce the bug

Create a WinUI project with the following xaml

<Grid Background="CornflowerBlue">
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="*"/>
            <ColumnDefinition Width="*"/>
        </Grid.ColumnDefinitions>
        <Image Name="ImageCtrl"  Grid.Column="0" HorizontalAlignment="Stretch" VerticalAlignment="Stretch"/>
        <Canvas Name="CanvasCtrl" Background="DarkSlateBlue" Grid.Column="1" HorizontalAlignment="Stretch" VerticalAlignment="Stretch"/>
    </Grid>

And code...

            Bitmap = new WriteableBitmap(Width, Height);
            Pixels = new ColorARGB8[Width * Height];
            GenerateColorData();
            ModifyBitmap(Bitmap, Pixels);
            ImageCtrl.Source = Bitmap;
            GenerateCompositionVisual();

  private void GenerateColorData()
  {
      //BitmapBytes ??= new byte[Width * Height * 4];

      for (int y = 0; y < Height; y++)
      {
          for (int x = 0; x < Width; x++)
          {
              //Pixels[y * Width + x] = new ColorARGB8((uint)0xff00ff00); // AARRGGBB
              Pixels[y * Width + x] = ColorARGB8.Random();
          }
      }
  }

  public unsafe static void ModifyBitmap(WriteableBitmap bitmap, ColorARGB8[] pixels)
  {
      if (bitmap.PixelWidth * bitmap.PixelHeight != pixels.Length)
          throw new Exception("ModifyBitmap | bitmap & pixels array dimensions do not match");

      // get bitmap pixel buffer and update it
      using Stream stream = bitmap.PixelBuffer.AsStream();
      Span<byte> byteSpan = MemoryMarshal.AsBytes(pixels.AsSpan());
      stream.Write(byteSpan);

      //bitmap.Invalidate();
  }
  private void GenerateCompositionVisual()
  {
      // get visual layer's compositor
      var compositor = ElementCompositionPreview.GetElementVisual(CanvasCtrl).Compositor;

      // create a surface brush, this is where we can use NearestNeighbor interoplation
      var brush = compositor.CreateSurfaceBrush();
      brush.BitmapInterpolationMode = CompositionBitmapInterpolationMode.MagNearestMinNearestMipNearest;

      // create a visual
      var imageVisual = compositor.CreateSpriteVisual();
      imageVisual.Brush = brush;

      // load the image
      //LoadedImageSurface imgSurface = LoadedImageSurface.StartLoadFromUri(new Uri(@"c:\somepath\q4QAb.png"));
      IRandomAccessStream stream = GetRandomAccessStreamFromWriteableBitmap(Bitmap);

      LoadedImageSurface imgSurface = LoadedImageSurface.StartLoadFromStream(stream);
      brush.Surface = imgSurface;

      // set the visual size when the image has loaded
      imgSurface.LoadCompleted += (s, e) =>
      {
          // choose any size here
          imageVisual.Size = new System.Numerics.Vector2((float)ImageCtrl.ActualWidth, (float)ImageCtrl.ActualHeight);
          float x = (float)(CanvasCtrl.ActualWidth - imageVisual.Size.X) / 2f;
          float y = (float)(CanvasCtrl.ActualHeight - imageVisual.Size.Y) / 2f;

          imageVisual.TransformMatrix = Matrix4x4.CreateTranslation(x, y, 0);
          imgSurface.Dispose();
      };

      // add the visual as a child to canvas
      ElementCompositionPreview.SetElementChildVisual(CanvasCtrl, imageVisual);
  }

  public IRandomAccessStream GetRandomAccessStreamFromWriteableBitmap(WriteableBitmap bitmap)
  {
      var stream = new InMemoryRandomAccessStream();
      var encoder = BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream).GetResults();
      encoder.SetPixelData(BitmapPixelFormat.Bgra8,
                           BitmapAlphaMode.Straight,
                           (uint)bitmap.PixelWidth,
                           (uint)bitmap.PixelHeight,
                           96, // DPI
                           96, // DPI
                           bitmap.PixelBuffer.ToArray());

      encoder.FlushAsync().GetAwaiter().GetResult();
      return stream;
  }

And a color type for your convenience...

/// <summary>
/// This is a color type that is compatible with Windows graphics frameworks.
/// </summary>
[StructLayout(LayoutKind.Explicit)]
public struct ColorARGB8 : IEquatable<ColorARGB8>
{
    [FieldOffset(0)]
    public int ARGB;
    [FieldOffset(0)]
    public byte A;
    [FieldOffset(1)]
    public byte R;
    [FieldOffset(2)]
    public byte G;
    [FieldOffset(3)]
    public byte B;

    public ColorARGB8(uint value)
    {
        this = default;
        unchecked
        {
            ARGB = (int)value;
        }
    }

    public ColorARGB8(int value)
    {
        // ARGB in
        this = default;
        ARGB = value;
    }

    /// <summary>
    /// This method is not thread safe.
    /// </summary>
    /// <returns></returns>
    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public static ColorARGB8 Random()
    {
        int rand = rnd.Next(int.MinValue, int.MaxValue);
        unchecked
        {
            rand |= (int)0xff000000;
        }
        return new ColorARGB8(rand);
    }
}

Expected behavior

The second texture should be an exact scaled representation of the original texture. Each point should map to it's respective point on the original.

The image on the left here, is actually the best WinUI can do to represent a texture AFAICT. I can't find any way to configure point sampling in WinUI - I have asked the question here ... #10312. The image on the right is the SpriteVisual with an incorrect copy of the original 16x9 texture.

Screenshots

Here's a 4x4 texture...

Here's a diagram showing that at least some people do understand that nearest neighbor actually means point...

NuGet package version

None

Windows version

Windows 11 (24H2): Build 26100

Additional context

No response

[castorix]

How would we ever produce a scaled image using WinUI's version of nearest neighbor given this behavior?

A way is with Direct2D (or Win2D)
I had posted a sample with Direct2D in this thread : #1138

[Gavin-Williams]

@castorix Thanks for that. But they added nearest neighbor and mucked it up. Does anybody claim that their implementation of point sampling is correct?

I don't want to have to turn to DirectX for everything. I only just started looking at WinUI again yesterday, it's been 24 hours and I've already hit a wall. To show a texture as it actually is, without any color interpolation is a pretty basic requirement.

I would love to see the source code for this. But I don't know how to find it. I searched and nothing looked like the relevant code.

[castorix]

I did some tests and I seem to get same result as Direct2D by scaling image with BitmapTransform.InterpolationMode, then called with BitmapDecoder.GetSoftwareBitmapAsync

[Gavin-Williams]

I'll take a look.

Though none of this double handling and rescaling should be necessary. WinUI should be able to simply render textures with point sampling, so that we can correctly zoom in to small textures.