Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add SixelImage #436

Open
wants to merge 4 commits into
base: v2-exp
Choose a base branch
from
Open

Conversation

CannibalVox
Copy link

@CannibalVox CannibalVox commented Nov 14, 2024

This PR adds support for sixels to lipgloss. Sixels are a protocol for writing images to the terminal by writing a large blob of ANSI-escaped data. They function by encoding columns of 6 pixels into a single character (in much the same way base64 encodes data 6 bits at a time). Sixel images are paletted, with a palette established at the beginning of the image blob and pixels identifying palette entires by index while writing the pixel data.

Sixels are written one 6-pixel-tall band at a time, one color at a time. For each band, a single color's pixels are written, then a carriage return is written to bring the "cursor" back to the beginning of a band where a new color is selected and pixels written. This continues until the entire band has been drawn, at which time a line break is written to begin the next band.

Supporting sixels requires overcoming a few challenges:

  1. Sixels traditionally supported only 256 colors. Modern terminals that have added sixel support may not have this restriction, but there's no real way of knowing, and anyway, too many colors massively inflate the size of a sixel image. As a result, a 256 color quantization process needs to be undergone on the image to produce the palette. sixel_palette.go has a Median Cut implementation for sixels.

  2. Sixels do not support alpha, so how should it be handled? When writing the sixel image, we use the param p2=1 which leaves pixels that are not written as part of the sixel's pixel data the color that is already in the terminal window. For fully-transparent pixels we simply do not write the pixels as part of the pixel data. But what about semi-transparent pixels? In Style.RenderSixelImage, if the style has a background color, I use the alpha channel in the palette to mix the background color with the pixel color to produce a semi-transparent effect. Be aware that sixel's color space is only 0-100 rather than the traditional 0-255, so it is sometimes possible for the human eye to distinguish between the background color in the terminal and the background color as drawn with sixels. Generally it looks good as long as the two colors aren't side by side.
    image
    If there is no background color in the style, the pixels are drawn fully-opaque at the color they appear in the image.

  3. Because colors must be drawn one at a time, we generate the pixel data in two phases: first, we traverse the image and write filled bits to a BitSet. Then, we use the BitSet to generate the pixel data string and store it for later use. Style.RenderSixelImage is responsible for generating the palette, image size, and pixel data into a coherent ANSI blob.

I chose to not tackle these challenges (though I can if they need to be solved before merge):

  • There are not multiple render options. For instance, it's not unusual for sixel images to support dithering (increases the size of the ANSI blob due to defeating RLE, but improves how it looks). I believe it's also possible to take advantage of overdraw to increase the use of RLE and make pixel data smaller, but at the cost of longer creation times.
  • sixelColor is right now just an arbitrary struct that does not implement color.Color. If this were fixed, it would be possible for sixelPalette to be turned into a general-purpose Median Cut utility.
  • SixelImages do not know or attempt to know how large they are in terminal characters. Additionally, on Windows, because Windows Terminal has chosen to size sixels such that characters are 10px by 20px, images are distorted based on what your current font's aspect ratio is. Hypothetically, the sixel code in lipgloss could figure out how big everything in the terminal is and make sixels compatible with other Width/Height measuring methods in Style, and/or provide size correction on windows. My current thought is that size correction could be done to the image itself before passing it into lipgloss, though.

Copy link
Member

@aymanbagabas aymanbagabas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing! Though, I would move the ANSI escape sequence part to x/ansi in a separate PR 🙂

@aymanbagabas
Copy link
Member

@CannibalVox Why didn't you use go-sixel here?

@CannibalVox
Copy link
Author

@CannibalVox Why didn't you use go-sixel here?

The only real reason is that I didn't know about it. After a little testing, the quantizer is prettier than mine and dithering support is really cool. The only downside to the library is that it takes about 250-300ms longer to encode an image than the code in this PR, but that's probably not important.

@aymanbagabas
Copy link
Member

aymanbagabas commented Jan 13, 2025

After a little testing, the quantizer is prettier than mine and dithering support is really cool.

We could add dithering like go-sixel using FloydSteinberg.

The only downside to the library is that it takes about 250-300ms longer to encode an image than the code in this PR, but that's probably not important.

I think that is really important.

You did an incredible job and it would be superb to have an alternative pure-go library to encode/decode Sixel graphics other than go-sixel. Let's move this to x/ansi.

We could have a x/ansi/sixel package that define implementation details such as sixel.NewEncoder/Decoder and implements the Sixel format (without escape sequences \x1bP ... \x1b\\). This should encode/decode the Sixel data to and from image.Image types similar to how image/png and image/jpeg APIs look like.

// ansi/sixel/sixel.go

type Encoder struct {
  Dither bool
  Width int
  Height int
}

func (e *Encoder) Encode(w io.Writer, m image.Image) error

Then in x/ansi, we can have a func SixelGraphics(p1 int, p2 int, data []byte) string. This can simply return \x1bP P1 ; P2 q <data> \x1b\\, where data is encoded using sixel.Encoder from x/ansi/sixel.

I've recently started looking into terminal graphics protocols, and we have implemented kitty graphics in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants