Package portable version of Unibeautify CLI

[stevenzeck]

From @lassik on June 26, 2018 11:56

Greetings from Emacs-land,

We would like to have the main functionality of Unibeautify in Emacs. Before discovering Unibeautify I started writing the format-all package. Right now it's a very bare-bones Unibeautify workalike: it calls pre-defined formatters with their default options. There's also a similar and older package for Vim called Neoformat with more features than we have.

Our Emacs package could simply use Unibeautify as its sole backend. However, currently that wouldn't work for us since installing NPM modules as extra dependencies is too difficult for users. Currently our users only have to install one Emacs package, plus whatever external formatters they use. There is no required dependency on anything outside Emacs.

If I understand correctly, the core of Unibeautify is essentially a huge data structure that maps programming languages and formatting options to compatible external formatters and their command line flags. Would you be interested in explicitly encoding this data structure in a JSON file? The file could be shared by Unibeautify and different editor plugins - each plugin project would simply have a script to convert the common JSON file into whatever representation is needed in its implementation language. E.g. in Emacs Lisp we could convert it into nested lists to store in a variable in Emacs.

Of course, in a perfect world Unibeautify would be the only universal formatting package anyone needed, but currently relying on Node and NPM makes deployment prohibitively difficult in many situations. (NPM is part of the standard workflow of JavaScript developers but many others are not comfortable using it and frequently encounter errors they do not understand.) Hence I think it still makes sense to have editor-specific packages, and extract as many common parts as we can into a shared specification file.

To recap, the common JSON file would specify:

• All the external formatters that exist.
• Common names for those external formatters.
• Common names for programming languages.
• Common names for formatting options (e.g. arrow_parens, multiline_ternary).
• A common specification of the valid range of values for all those formatting options (for machine validation).
• A mapping to convert the common option names to the command line flags expected by each external formatter.
• Maybe some human-readable help text for the formatting options.

Then Unibeautify and various editor plugins could mainly consist of some plumbing to construct command lines from this data structure.

For my part, I would be happy to contribute to the maintenance of such a JSON file. The master copy could be hosted in its own Git repo or in the Unibeautify repo. I don't really mind which approach is taken if people are willing to do this. What do you think?

Copied from original issue: Unibeautify/unibeautify#111

[stevenzeck]

From @Glavin001 on June 26, 2018 14:27

tl;dr: I think with the help of https://github.com/zeit/pkg we can get what we want 😃.

---

We would like to have the main functionality of Unibeautify in Emacs.
...
Our Emacs package could simply use Unibeautify as its sole backend.

This would be great!

However, currently that wouldn't work for us since installing NPM modules as extra dependencies is too difficult for users. Currently our users only have to install one Emacs package, plus whatever external formatters they use. There is no required dependency on anything outside Emacs.

Unfortunately, users will likely always need to install Unibeautify as an NPM package, since it is written in Node.js.

However, we could try to package using something like https://github.com/zeit/pkg
Which reports:

No need to install Node.js and npm to run the packaged application

Although I have not used this bundling tool before so no promises it would work. The standard approach is always using Node.js and installing via npm.

If I understand correctly, the core of Unibeautify is essentially a huge data structure that maps programming languages and formatting options to compatible external formatters and their command line flags.

This is only part of it. Unibeautify has a registry for common options in JSON. However, the beautifiers themselves are implemented with Node.js/JavaScript and are functions registered with Unibeautify (core). Then Unibeautify CLI uses Unibeautify (core) to perform the beautification process. There are a lot of moving parts.

To recap, the common JSON file would specify:

This proposed JSON file is only so valuable. Unibeautify's implementation itself is to process and perform the beautification.

For my part, I would be happy to contribute to the maintenance of such a JSON file. The master copy could be hosted in its own Git repo or in the Unibeautify repo. I don't really mind which approach is taken if people are willing to do this. What do you think?

Unfortunately, this JSON approach is not actually practical in reality.

However, I think with the help of https://github.com/zeit/pkg we can get what we want 😃.

[stevenzeck]

From @Glavin001 on June 26, 2018 14:29

@szeck87 this issue should be moved to https://github.com/unibeautify/unibeautify-cli . I do not (yet) know how, if you do not mind moving it for us 😃. Thank you.

Also, we should rename repo https://github.com/unibeautify/unibeautify-cli to https://github.com/unibeautify/cli 😝

@lassik I would like to move forward testing https://github.com/zeit/pkg on Unibeautify CLI. I do not have the time right now, however, if you are interested in contributing this would be great! Please let me know if you have any questions.

[stevenzeck]

From @Glavin001 on June 26, 2018 14:30

Once we are done, we can update https://unibeautify.com/docs/editor-emacs.html and share with the world! 🎉

[Glavin001]

@lassik Awesome work on https://github.com/lassik/emacs-format-all-the-code ! I would love to work together to bring Unibeautify to Emacs! 😃 🎉

[lassik]

If I understand correctly, the core of Unibeautify is essentially a huge data structure that maps programming languages and formatting options to compatible external formatters and their command line flags.

This is only part of it. Unibeautify has a registry for common options in JSON. However, the beautifiers themselves are implemented with Node.js/JavaScript and are functions registered with Unibeautify (core). Then Unibeautify CLI uses Unibeautify (core) to perform the beautification process. There are a lot of moving parts.

OK, I see that some of the beautifiers are implemented by calling existing third-party Node modules. Do you plan to eventually move as many beautifiers as possible to use Node implementations instead of external programs?

@lassik Awesome work on https://github.com/lassik/emacs-format-all-the-code ! I would love to work together to bring Unibeautify to Emacs! 😃 🎉

Thanks for the kind words :) I didn't put that much effort into it because I think a project like Unibeautify is what's really needed. Help is definitely welcome bringing it to Emacs :)

I believe the main thing that would boost Unibeautify adoption among Emacs users is for the Unibeautify CLI to be as easy as possible to install from the OS package manager. (Mainly Homebrew for Mac, Apt/Yum for Linux distros, the Ports systems for the BSDs. I don't know what Windows users use nowadays - NuGet perhaps?) The Emacs user base is probably somewhat old-school, and most of us usually go to the OS package manager to install things, avoiding the direct use of language-specific package managers such as NPM where possible. I believe Node is now so popular that most or all of the major OS package managers have built-in shortcuts to integrate Node programs. So that could be an easier / more conventional route than using zeit/pkg (though that could also be a fine choice, especially for Windows where package management is less standardized).

Having a single super-package in the OS package manager to "install Unibeautify, the CLI, and all beautifier front-ends" would be great. (The external formatter programs like gofmt and clang-format would still have to be installed separately, for the beautifiers that use them.) So that for example, to format C code, one could simply do:

brew install unibeautify clang-format  # Mac
yum  install unibeautify clang-format  # Linux
...

The other thing is to translate between Emacs settings and Unibeautify beautifier options. I think this will be reasonably easy and I can definitely help with it. I could start a Wiki page to collect all the settings. Anyway, I think making Unibeautify easy to install from OS package managers would be the lion's share of the work (and would also help a ton with Vim users and probably others :)

[Glavin001]

Do you plan to eventually move as many beautifiers as possible to use Node implementations instead of external programs?

Ideally the beautifiers are Node.js. Unfortunately, many many are not. See the Preinstalled column of https://github.com/Glavin001/atom-beautify#beautifiers

It is not the intention of Unibeautify to reimplement any beautifier in Node.js if it is not already. Instead we wrap the executable CLI.

I believe the main thing that would boost Unibeautify adoption among Emacs users is for the Unibeautify CLI to be as easy as possible to install from the OS package manager.
...
The Emacs user base is probably somewhat old-school, and most of us usually go to the OS package manager to install things, avoiding the direct use of language-specific package managers such as NPM where possible.

Makes sense, I agree.

I believe Node is now so popular that most or all of the major OS package managers have built-in shortcuts to integrate Node programs.

I am not familiar with this.

Node programs need Node.js to be installed to execute them. Even the pkg tool requires downloading prebuilt Node.js binaries support the required targets: https://github.com/zeit/pkg#targets

Then it would become something like:

brew cask install unibeautify-cli

Anyway, I think making Unibeautify easy to install from OS package managers would be the lion's share of the work (and would also help a ton with Vim users and probably others :)

Definitely. I personally have a Mac and could contribute to Homebrew. However, the other package managers I am less familiar with.

---

🎉 I'm excited to work together on this! 😃

[lassik]

See the Preinstalled column of https://github.com/Glavin001/atom-beautify#beautifiers

That table is very impressive. In fact, it was one of the original inspirations for the Emacs package :)

I also mainly use a Mac, but also familiar with Linux/BSD. In particular, I could assist on BSDs, since it's often hard to find maintainers for them.

I looked into Node support a bit deeper and unfortunately support on BSDs doesn't seem to be entirely unproblematic. Node seems to have only a couple maintainers running FreeBSD and had some trouble with FreeBSD's version of the clang compiler. OpenBSD and NetBSD are less popular so they are probably in a worse situation. Nevertheless there is a node package installable on all of the major BSDs, but it may be an older version of Node.

It just occurred to me that a lightweight, portable JavaScript interpreter might be one possibility. One called Duktape seems to have a lot of traction and is readily packaged for Linux and all the BSDs (not for Homebrew, for some weird reason, but it can't be hard to do) which means it ought to work well there. Unibeautify is written in TypeScript, and some people have been running the JavaScript output of the tsc compiler on Duktape without encountering any problems (they even managed to run tsc itself on Duktape by replacing calls to some Node I/O APIs with Duktape equivalents). Links at http://wiki.duktape.org/CompatibilityTypeScript.html I don't know how many Node-specific APIs Unibeautify uses, but if those are not too hard to replace, Duktape or equivalent could be the most portable, least-hassle-for-users choice for deployment.

I could try to hack up a Travis/Docker job to download Unibeautify's Node/NPM modules and compile a Duktape-based native executable from them but it may be a few months before I can find the time. I'm also most likely not the best person to do it, but I can try if nobody else beats me to it 😄

[Glavin001]

Duplicate of #2 (closing #2 instead)

Also see https://github.com/nexe/nexe