Implement Arduino CLI (instead of IDE) backend

[ianfixes]

Feature Request

There are many benefits to using the Arduino CLI instead of the IDE to run commands. But the IDE was the only way when this project was written.

Update ArduinoCmd (arduino_cmd.rb) to use the CLI instead of IDE.

This is expected to fix #137 as well as the issue of "hidden built-in" libraries like Ethernet.

This also will satisfy cross-platform, see #180 (comment)

[matthijskooijman]

I think this is a duplicate of #147.

[matthijskooijman]

Oh, scratch that, that issue about compiling the unit tests with arduino-cli, IIUC, this issue is about compiling the examples/sketches with arduino-cil. Totally different thing (and this one is probably a lot less invasive to implement).

[ianfixes]

probably a lot less invasive to implement

That's my hope. ArduinoCmd defines the interface for all the commands that need to be run. ArduinoInstallation instantiates that class with the location of the binary.

So if that can go smoothly, all that's needed is to write down what flags drive each command. Here's the interface I had made for the "linux builder" CLI which IIRC was a beta version of that CLI (which wasn't as mature as I needed it to be at the time I initially wrote the library):

require "arduino_ci/host"
require 'arduino_ci/arduino_cmd'

module ArduinoCI

  # Implementation of Arduino linux CLI commands
  class ArduinoCmdLinuxBuilder < ArduinoCmd

    flag :get_pref,        "--get-pref"          # apparently doesn't exist
    flag :set_pref,        "--pref"              # apparently doesn't exist
    flag :save_prefs,      "--save-prefs"        # apparently doesn't exist
    flag :use_board,       "-fqbn"
    flag :install_boards,  "--install-boards"    # apparently doesn't exist
    flag :install_library, "--install-library"   # apparently doesn't exist
    flag :verify,          "-compile"

  end

end

If you know the flags to accomplish all these operations in the CLI, this might be a quick win. There might also be some workaround; use_board is probably something that just gets set when running the actual compilation.

[matthijskooijman]

I haven't done too much with arduino-cli, but I can try to comment.

Wrt prefs, arduino-cli does not use the same preferences.txt mechanism as the IDE. It does have a arduino-cli.yaml for configuration, but that only has a few options, which I'm not sure are relevant here. arduino-cli config init can create a new config file, but at first glance there does not seem to be a programmatic way to set config options. What do you use the pref options for?

For installing boards (rather, cores that supply boards):

$ arduino-cli core install --help
Installs one or more cores and corresponding tool dependencies.

Usage:
  arduino-cli core install PACKAGER:ARCH[@VERSION] ... [flags]

Examples:
  # download the latest version of Arduino SAMD core.
  arduino-cli core install arduino:samd

  # download a specific version (in this case 1.6.9).
  arduino-cli core install arduino:[email protected]

Flags:
  -h, --help                help for install
      --run-post-install    Force run of post-install scripts (if the CLI is not running interactively).
      --skip-post-install   Force skip of post-install scripts (if the CLI is running interactively).

Global Flags:
      --additional-urls strings   Comma-separated list of additional URLs for the Boards Manager.
      --config-file string        The custom config file (if not specified the default will be used).
      --format string             The output format, can be {text|json}. (default "text")
      --log-file string           Path to the file where logs will be written.
      --log-format string         The output format for the logs, can be {text|json}.
      --log-level string          Messages with this level and above will be logged. Valid levels are: trace, debug, info, warn, error, fatal, panic
  -v, --verbose                   Print the logs on the standard output.

Note that you can pass --additional-urls to configure third-party board urls, without needing to configure them in a config file at all.

For libraries, there is:

$ arduino-cli lib install --help
Installs one or more specified libraries into the system.

Usage:
  arduino-cli lib install LIBRARY[@VERSION_NUMBER](S) [flags]

Examples:
  arduino-cli lib install AudioZero       # for the latest version.
  arduino-cli lib install [email protected] # for the specific version.

Flags:
      --git-url    Enter git url for libraries hosted on repositories
  -h, --help       help for install
      --no-deps    Do not install dependencies.
      --zip-path   Enter a path to zip file

Global Flags:
      --additional-urls strings   Comma-separated list of additional URLs for the Boards Manager.
      --config-file string        The custom config file (if not specified the default will be used).
      --format string             The output format, can be {text|json}. (default "text")
      --log-file string           Path to the file where logs will be written.
      --log-format string         The output format for the logs, can be {text|json}.
      --log-level string          Messages with this level and above will be logged. Valid levels are: trace, debug, info, warn, error, fatal, panic
  -v, --verbose                   Print the logs on the standard output.

This even allows installing libraries outside of the library manager by supplying a direct git url, but I'm not sure what the constraints are for that.

For compiling, there is:

$ arduino-cli compile --help
Compiles Arduino sketches.

Usage:
  arduino-cli compile [flags]

Examples:
  arduino-cli compile -b arduino:avr:uno /home/user/Arduino/MySketch
  arduino-cli compile -b arduino:avr:uno --build-property "build.extra_flags=\"-DMY_DEFINE=\"hello world\"\"" /home/user/Arduino/MySketch
  arduino-cli compile -b arduino:avr:uno --build-property "build.extra_flags=-DPIN=2 \"-DMY_DEFINE=\"hello world\"\"" /home/user/Arduino/MySketch
  arduino-cli compile -b arduino:avr:uno --build-property build.extra_flags=-DPIN=2 --build-property "compiler.cpp.extra_flags=\"-DSSID=\"hello world\"\"" /home/user/Arduino/MySketch


Flags:
      --build-cache-path string      Builds of 'core.a' are saved into this path to be cached and reused.
      --build-path string            Path where to save compiled files. If omitted, a directory will be created in the default temporary path of your OS.
      --build-property stringArray   Override a build property with a custom value. Can be used multiple times for multiple properties.
      --clean                        Optional, cleanup the build folder and do not use any cached build.
  -e, --export-binaries              If set built binaries will be exported to the sketch folder.
  -b, --fqbn string                  Fully Qualified Board Name, e.g.: arduino:avr:uno
  -h, --help                         help for compile
      --libraries strings            List of custom libraries paths separated by commas. Or can be used multiple times for multiple libraries paths.
      --optimize-for-debug           Optional, optimize compile output for debugging, rather than for release.
      --output-dir string            Save build artifacts in this directory.
  -p, --port string                  Upload port, e.g.: COM10 or /dev/ttyACM0
      --preprocess                   Print preprocessed code to stdout instead of compiling.
  -P, --programmer string            Optional, use the specified programmer to upload.
      --quiet                        Optional, suppresses almost every output.
      --show-properties              Show all build properties used instead of compiling.
  -u, --upload                       Upload the binary after the compilation.
  -t, --verify                       Verify uploaded binary after the upload.
      --vid-pid string               When specified, VID/PID specific build properties are used, if board supports them.
      --warnings string              Optional, can be "none", "default", "more" and "all". Defaults to "none". Used to tell gcc which warning level to use (-W flag). (default "none")

Global Flags:
      --additional-urls strings   Comma-separated list of additional URLs for the Boards Manager.
      --config-file string        The custom config file (if not specified the default will be used).
      --format string             The output format, can be {text|json}. (default "text")
      --log-file string           Path to the file where logs will be written.
      --log-format string         The output format for the logs, can be {text|json}.
      --log-level string          Messages with this level and above will be logged. Valid levels are: trace, debug, info, warn, error, fatal, panic
  -v, --verbose                   Print the logs on the standard output.

With --fqbn that, I think, accepts the same thing as the IDE -fqbn option. You might also need --export-binaries if you want to inspect the resulting binaries after compilation.

[per1234]

there does not seem to be a programmatic way to set config options

You can set them via environment variables:
https://arduino.github.io/arduino-cli/latest/configuration/#environment-variables
Or you may prefer to just write directly to the configuration file. If YAML is not convenient, other formats are supported, including JSON:
https://arduino.github.io/arduino-cli/latest/configuration/#supported-formats

You might also need --export-binaries if you want to inspect the resulting binaries after compilation.

Yeah, that is not currently necessary, but it will be soon due to arduino/arduino-cli#1051. However, this option is not available in the current release version of Arduino CLI because it was just added a few weeks ago (arduino/arduino-cli#1042)

The backwards compatible way to do it is by setting the ARDUINO_SKETCH_ALWAYS_EXPORT_BINARIES environment variable to true. Older versions of Arduino CLI will just ignore it and have the current behavior, while new versions will be configured to retain the current behavior.

[ianfixes]

@per1234 this may be a stupid question, but is the IDE a prerequisite for installing the CLI? I don't see anything in the README to indicate that it would be... but I don't want to run into arduino/Arduino#9351

I'm just hacking my way through it at the moment but if I'm definitely headed for a blocker then I'll pick something else to work on in the meantime.

[per1234]

No. The CLI is completely standalone.

One thing that might be surprising to people accustomed to the ways of the Arduino IDE is that Arduino IDE comes with a bundled copy of the Arduino AVR Boards platform, while Arduino CLI doesn't come with anything bundled. So after installing the Arduino IDE, you can immediately start compiling sketches for the Uno et al, which is a good thing for beginners, but wasteful if you are only using other architectures and have no need for the whole AVR toolchain.

Some 3rd party AVR boards platforms are configured/documented with the assumption that the arduino:avr Boards platform and the arduino:avr-gcc and arduino:avrdude tools will always already be present on the user's system, so they don't explicitely define or document their dependency on these resources. But it's easy enough for the user to install the dependency via arduino-cli core install arduino:avr once they understand the dependency exists.

[ianfixes]

I made good progress with this, happening in #218 now. If arduino-cli is something you personally contribute to, I can't thank you (all) enough for providing it -- I was able to unceremoniously delete so much workaround code.

If I'm understanding correctly, the arduino-cli core install _____ will automatically bring in compiler dependencies?

[per1234]

If arduino-cli is something you personally contribute to

By some mysterious twist of fate, I found myself working with the people who make that magic happen. I can't claim to make much of a significant contribution, but it's certainly a privilege to have even a small role in such an important project.

I'm glad if you find it to be useful for your project!

If I'm understanding correctly, the arduino-cli core install _____ will automatically bring in compiler dependencies?

The mechanism is there to do so (https://arduino.github.io/arduino-cli/latest/package_index_json-specification/#__code_3:~:text=toolsDependencies,-%3A%20the), but it all depends on whether the boards platform author has defined the tools dependencies in their package index. As I mentioned in my previous comment, some 3rd party platform authors (typically AVR) don't bother to define the tools dependencies because an Arduino IDE user will already have them installed.

Platforms can reference resources from other platforms (core libraries, core variants, tool recipes) (https://arduino.github.io/arduino-cli/latest/platform-specification/#referencing-another-core-variant-or-tool), and in this case there is a dependency on that other platform being installed. There isn't currently a mechanism for programmatically defining a dependency on another platform (though there is an open feature request for this), so in this case the platform author must rely on the user to install the referenced platform.

For example, https://github.com/damellis/attiny references arduino:avr's core library and tool recipes, and the latest version also doesn't define toolsDependencies, so you must run the commands to install both platforms:

arduino-cli core install attiny:avr
arduino-cli core install arduino:avr

With any of the official Arduino boards platforms, the complete installation of all dependencies is accomplished with a single arduino-cli core install command.

[ianfixes]

The more progress I make on this, the more I'm curious whether arduino-cli would adopt unit testing support (i.e. merge from this library) at some future point. Assuming #169 is received favorably, arduino_ci is simply delegating everything except the unit test / mock library to the CLI tool.

[matthijskooijman]

@ianfixes A related report is arduino/arduino-cli#936 where others have similar thoughts (not sure why I only remembered this now).

[ianfixes]

That is excellent, thanks for letting me know!