Missing dependency between .cmx and .o files

[pveber]

The problem here is analogous to #21, that is a (probable) missing dep between cmx and o files. I have just pushed a native-compilation-missing-dep branch to show the problem (see the commit message for instructions on how to reproduce).

[agarwal]

Something is strange. Here's some additional observation:

On the second invocation of make native, we get this:

$ make clean && make native
$ make native
ocamlbuild -use-ocamlfind -classic-display -verbose 10 -plugin-tag "package(solvuu-build)" ./my_project_lib.cmxa ./my_project_lib2.cmxa ./my_project_lib.cmxs ./my_project_lib2.cmxs
include directories are: [ "." ]
Doing sanity checks
include directories are: [ "." ]
==> my_project_lib.cmxa
====> my_project_lib.cmx
======> lib/a.ml
lib/a.ml exists in source dir -> import it
======> lib/b.ml
lib/b.ml exists and up to date
start rule lib/a.ml,lib/b.ml -> my_project_lib.cmx,my_project_lib.o ()
======> lib/a.cmx
========> lib/a.ml
lib/a.ml already built
start rule lib/a.ml -> lib/a.cmi,lib/a.cmx,lib/a.o ()
dyndeps: {.  .}
mid rule lib/a.ml -> lib/a.cmi,lib/a.cmx,lib/a.o (): cache miss: a dependency has changed (lib/a.ml)
ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml
resource_changed: lib/a.cmx
resource_changed: lib/a.o
end rule lib/a.ml -> lib/a.cmi,lib/a.cmx,lib/a.o ()
new dyndep for "lib/a.ml,lib/b.ml -> my_project_lib.cmx,my_project_lib.o ()"(
  [ my_project_lib.cmx; my_project_lib.o ]): "lib/a.cmx"
======> lib/b.cmx
========> lib/b.ml
lib/b.ml already built
========> lib/b.cmi
==========> lib/b.mli
lib/b.mli exists and up to date
start rule lib/b.mli -> lib/b.cmi ()
dyndeps: {.  .}
mid rule lib/b.mli -> lib/b.cmi (): cache hit
[cache hit] b.mli
end rule lib/b.mli -> lib/b.cmi ()
start rule lib/b.ml,lib/b.cmi -> lib/b.cmx,lib/b.o ()
========> lib/a.cmi
lib/a.cmi already built
new dyndep for "lib/b.ml,lib/b.cmi -> lib/b.cmx,lib/b.o ()"([ lib/b.cmx;
                                                              lib/b.o ]): "lib/a.cmi"
dyndeps: {. lib/a.cmi .}
mid rule lib/b.ml,lib/b.cmi -> lib/b.cmx,lib/b.o (): cache hit
[cache hit] b.ml
end rule lib/b.ml,lib/b.cmi -> lib/b.cmx,lib/b.o ()
new dyndep for "lib/a.ml,lib/b.ml -> my_project_lib.cmx,my_project_lib.o ()"(
  [ my_project_lib.cmx; my_project_lib.o ]): "lib/b.cmx"
dyndeps: {. lib/a.cmx, lib/b.cmx .}
mid rule lib/a.ml,lib/b.ml -> my_project_lib.cmx,my_project_lib.o (): 
cache miss: a dependency has changed (lib/a.ml)
ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
+ ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
File "./my_project_lib.cmx", line 1:
Error: Files lib/b.cmx and lib/a.cmx
       make inconsistent assumptions over implementation A
Command exited with code 2.
make: *** [native] Error 10

AFAICT, the only things being done are:

• ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml
• ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx

where the 2nd step fails.

However, running these commands manually succeeds:

$ make clean && make native
(* change a.ml *)
$ cd _build
$ ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml
$ ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
(* compilation succeeds *)

So the build is doing something extra that messes things up. I don't know what.

[pveber]

This is really twisted... But it gets worse. Look at this slight modification of the commands you describe:

$ make clean && make native
(* change a.ml *)
$ make native
(* compilation fails *)
$ cd _build
$ ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml
$ ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
(* compilation fails *)

It does seem that the second call to make does something wrong.

[pveber]

I made a recursive diff on the whole directory between the state just after editing a.ml and after the second invocation, and the only relevant difference is a.o, which is expected. BTW, it is pretty instructive to look at the _build/_log file:

### Starting build.
# Target: ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml, tags: {  }
ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml
# Target: ocamlfind ocamlc -c -I lib/. -o lib/b.cmi -package '' lib/b.mli, tags: {  }
ocamlfind ocamlc -c -I lib/. -o lib/b.cmi -package '' lib/b.mli # cached
# Target: ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/b.cmx -package '' lib/b.ml, tags: {  }
ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/b.cmx -package '' lib/b.ml # cached
# Target: ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx, tags: {  }
ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
+ ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
File "./my_project_lib.cmx", line 1:
Error: Files lib/b.cmx and lib/a.cmx
       make inconsistent assumptions over implementation A
Command exited with code 2.
# Compilation unsuccessful.

where the only problem IMO is that one should not use the cached version of b.cmx. In fact it is really your observation that I don't understand: I don't get how the compilation of the pack can work if you don't recompile b.cmx

[pveber]

Ok, got it. Your example is wrong, here's what you should be doing:

$ make clean && make native
(* change a.ml *)
$ cp lib/a.ml _build/lib/a.ml
$ cd _build
$ ocamlfind ocamlopt -c -for-pack My_project_lib -I lib/. -o lib/a.cmx -package '' lib/a.ml
$ ocamlfind ocamlopt -o ./my_project_lib.cmx -pack lib/a.cmx lib/b.cmx
(* compilation fails *)

So the "extra thing that messes things up" that ocamlbuild does is just to copy the file you modified in the _build directory 😛

[agarwal]

I wasn't compiling the changed file! Ugh... sorry for wasting your time. Okay investigating further.

[agarwal]

You can make compilation succeed by re-compiling b.ml, but is that the right thing to do? The implementation b.ml should depend only on the interface a.cmi, which doesn't change. Any idea what the correct thing to do is even manually.

[pveber]

The implementation b.ml should depend only on the interface a.cmi

I don't think this is true for native compilation of packed libraries, and the error message is a hint of that IMO. Maybe we can have a look at the original compilation rules in ocamlbuild? Or read the manual ;0)...

[pveber]

Couldn't find a clue in the manual but this quote gives a hint:

The rationale is that with inlining enabled, ocamlopt when building
module B has embedded in it implementations coming from module A. If
that is changed module B needs to be rebuilt as well.