example of switching p2a to dune

[Isweet]

I'd like to propose switching to the dune build system for OCaml. This performs the port for the p2a project.

This should not be merged! It is merely to show the diff, and spark a discussion.

Some advantages of this include:

1. Simpler build configuration
2. dune integrates with the utop toplevel, so to test the functions in a library (say basics.ml):

8c8590432779:p2a iansweet$ dune utop src
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                      │ Welcome to utop version 2.2.0 (using OCaml version 4.06.1)! │
                                                                                                                                                      └─────────────────────────────────────────────────────────────┘

Type #utop_help for help about using utop.

─( 16:57:30 )─< command 0 >──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # open Basics;;
─( 16:57:30 )─< command 1 >──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─
utop # Basics.eq [ 1 ; 2 ; 3 ] [ 2 ; 3; 1 ];;
Exception: Failure "unimplemented".

3. Running tests is similarly easy:

8c8590432779:p2a iansweet$ dune runtest .
EEEEEEEEEEEEEEE
==============================================================================
Error: public:14:product.

...

Ran: 15 tests in: 0.11 seconds.
FAILED: Cases: 15 Tried: 15 Errors: 15 Failures: 0 Skip:  0 Todo: 0 Timeouts: 0.

4. It has rapidly become the default build system for OCaml packages on OPAM.

The only (potential) obstacle is the submit server. I'd have to investigate a bit to figure out how hard it would be to modify the submit server to accept dune projects. Essentially, the dune binary would need to be available on the submit server. It is available through OPAM.

Let me know what you think.

[camoy]

This seems like a really good idea. Easy integration with utop is nice (no more forgetting .ocamlinit files!) For P4B we link in pre-compiled .cmo/.cmi files. Do you know off-hand how to do that with dune?

[Isweet]

Off the top of my head, no. But I'm sure there is a way, and I'd be happy to look into it.

[camoy]

Only if it's not too much trouble. I'd greatly appreciate it.

[Isweet]

This is in the p4 directory? Just sets.cmi / sets.cmo -- is that right?

[Isweet]

See: https://discuss.ocaml.org/t/dune-link-against-cmo-cmi-without-source/2488

Short answer: there isn't a way to do this.
Long answer: is there a good reason you are doing this? If not, it'd be pretty simple to just (a) distribute the source or (b) use an existing Set library (e.g. Jane Street's).

[camoy]

Yes, for P4 it's not as necessary (it provides a canonical implementation of a prior project they implemented in P2 just so they don't have to learn a new library). It's really more necessary for P3B where we give a correct lexer and parser. During the summer we did away with all this and just suggested they copy in prior code, mainly to avoid bytecode version incompatibilities since many people had different OCaml versions. However for some students, bugs in their prior projects only manifested in later projects. This was a very difficult problem to debug (both for them and in office hours).

I don't really think there's a nice complete solution to all this. Nonetheless, I will definitely talk to Dr. Mamat about switching to dune. I'm pretty sure he will be OK with it.

[Isweet]

I assume that the reason for not giving them the source for the canonical implementation is that you don't want it to be distributed (i.e. cheating, etc.)?

Dune has support for generating lexers / parsers from .mll and .mly files, but if these are also .cmo/cmi files then you will have the same issue.

Another (slightly crazy, but also nice) option is to create an OPAM package for the canonical implementations and then host them on a local UMD server. Then you can do:

(library
  (name regexpi)
  (libraries sets))

in the dune file.

[Isweet]

Actually, now that I think about it, dune uses findlib for the packages in its (libraries ...) clause. So, if you had a way to make the binary version of the modules into findlib packages and register them then you could also do what I suggested above.

[Isweet]

I think this is the best solution, because it would allow you to only distribute binaries but then also depend on them properly as a library. Although it does require someone to fashion the sets.cm* files into a findlib library.

[camoy]

That definitely sounds like the best option. I'll look into doing that. Thank you so much Ian!

[Isweet]

Okay, I was able to create an opam package for the sets module. Now I am going to see if I did it correctly, and if dune can access this package... please stand by...

[Isweet]

Currently I'm imagining things like:

src/
  regexp.ml
  regexp.mli
  nfa.ml
  nfa.mli
  dune
dep/
  sets/
    sets.cmo
    sets.cmi
    Makefile
    sets.opam
bin/
  viz.ml
  dune
test/
  testUtils.ml
  public.ml
  dune

and you'd just need to do:

opam pin add sets ./sets

to register the sets package from opam. Then, the dune file in src/ just has:

(libraries sets)

and everything should work out.

[Isweet]

Then anytime you have things you want to distribute as binary, you stick them in dep and write a quick little .opam file and do a opam pin add and you should be good.

Doing this for p4b now to verify.

[Isweet]

And you can even add an OCaml version constraint for 4.04.x so that when students try to use the wrong version of OCaml, opam will yell at them.

[Isweet]

I've added a first crack at my idea. It's here (second commit on the branch):

1f54b08

The basic idea is to turn any dependencies whose source you don't want to distribute into opam and findlib packages, and then opam pin add them so that dune can find them easily.

The stuff in deps/sets is messy but all the other stuff should work once I've got that figured out.

[camoy]

As noted in your Makefile I'm not sure *.cmxa files can be generated without the original *.ml source (at least I can't find any reference online). And without that it seems you can't run the tests :(

[Isweet]

Yeah, in order to build the *.cmxa you need the *.cmx.

Seems strange that you can't run tests, since they should be able to be run via the compiled version just as well as the bytecode.

I'll see if I can find something on this tomorrow.

[camoy]

"test" --> "executable" will get rid of the cmxa warning.