Hygiene problems and slow compilation with extra directories

[copy]

Consider a trivial app:

open Solvuu_build.Std

let app = Project.app
    ~file:"app.ml"
    "test_app"

let () = Project.basic1 ~project_name:"test" ~version:"dev" [app]

print_endline "test"

Everything builds fine. A problem arises if we have unrelated directories in the same folder: Compilation is slow (unrelated directories seem to be traversed) and compilation fails with hygiene problems. The latter can be fixed by adding -no-hygiene to the ocamlbuild command line. I'm not sure if that is an appropriate fix though.

For testing, you can symlink any big folder to the project folder, for example. ln -s ~/.opam/ .

[copy]

As a workaround, one can create an _tags file to ignore unrelated directories:

<node_modules>: -traverse
<.git>: -traverse

[agarwal]

Adding -no-hygiene could be fine. Would you mind testing in your environment and see if it firstly resolves this issue, and second that some other unexpected thing doesn't happen. As a quick way, you could directly modify ~/.opam/<switch>/lib/solvuu-build/solvuu.mk.

[copy]

Adding -no-hygiene fixes the problem of hygiene errors and doesn't cause any further problems in my project. However, compilation is still noticeably slower when a large directory is present.

[agarwal]

Hmm... so is -no-hygiene the culprit? Is there some way to tell ocamlbuild -no-traverse, other than creating an _tags file?

[copy]

Hmm... so is -no-hygiene the culprit?

I think ocamlbuild traverses all directories independent of this setting. Apparently this behaviour is set automatically when a myocamlbuild.ml file is present: https://github.com/ocaml/ocamlbuild/blob/master/manual/manual.adoc#Sec_Directories
I couldn't find any parameter to override this behaviour, adding -tag '-traverse' to the command line didn't work either. Maybe this is something that can be changed using the Ocamlbuild_plugin api, but I'm not familiar with that.

[agarwal]

We should be able to handle this by adding Ocamlbuild_plugin.tag_any ["-traverse"] to our plugin definitions, which effectively does "true: -traverse". However, it doesn't work. I tried adding it to Before_hygiene and Before_options, but in both cases the large directory is still traversed AFAICT from the amount of time it takes to do the build.

Regardless, "true: -traverse" is too broad. We get errors like "Ocamlbuild knows of no rules that apply to a target named src/a.ml". It no longer knows about the source files even though they are referenced in rules. I'm not sure we'll be able to resolve this. We'd have to add rules to copy src/a.ml to _build/src/a.ml but expressing such a rule to ocamlbuild is impossible.

Next best option is to add a no_traverse : string list argument to Project.basic1. Thus, users could specify which directories to avoid traversing. Then we could do List.iter ~no_traverse ~f:(fun x -> Ocamlbuild_plugin.tag_file x ["-traverse"]), but as above this seems to have no effect.

[pveber]

I'm affraid there is no workaround here without modifying ocamlbuild: look at the proceed function in ocamlbuild Main module. Whatever you're doing with tags in Before_hygiene and Before_options hooks, it is over-written at L108. Modifying _tags works, but that's because the file is processed at a later step if it exists (L136).