Managing package dependencies with Bazel

Questions
1

Hi

I have a mono-repo with Bazel. There are two components A and B, both a DAML project, where A depends on B.

I can daml damlc build components B, no problem. The file B.dar is then somewhere in bazel-out/.

But how do I make component B available to the build of component A?

The documentation suggests to list B.dar in the dependencies section of the daml.yaml file of component A. But this means doing something like ../../bazel-out/k8-fastbuild/bin/components/B/B.dar ugh.

I understand I could also just list B as an absolute name under A’s dependencies. But then I need to have B.dar be included in the package db that the build for component A is using. Am I getting this right?

If so, how do I create a package db that includes B.dar? And how would the Bazel build rule look like that accomplishes this?

Or is there another way I am not seeing right now?

Thanks and regard,
Alex

2 Likes
2

Since your B.dar ends up in bazel-out, I assume the daml build command (no need for damlc) is run inside a Bazel rule. Probably a Genrule?
And I assume the Genrule has the B.dar as its output as well.

If so, the “right” way of using that dependency in A would be to also build A using a Genrule, which takes the rule of B as a dependency.

Say you have

genrule(
    name = "B",
    srcs = glob(sourcepathB/**),
    outs = ["B.dar"],
    tools = ["//.../daml"]
    cmd = """
      cd sourcepathB
      $(execpath //.../daml) build -o $@
    """
)

Then you can use that dependency in the build rule for A.

genrule(
    name = "A",
    srcs = glob(sourcepathA/**) + [
    "//pathToB:B"
    ],
    outs = ["A.dar"],
    tools = ["//.../daml"]
    cmd = """
      cd sourcepath
      mkdir -p lib
      cp $(location //pathToB:B) lib
      $(execpath //.../daml) build -o $@
    """
)

Now you’d have to have to reference lib/B.dar in your daml.yaml file.

This doesn’t integrate with the Daml VsCode plugin. You’d have to move the B.dar file to the lib directory after changing B.

If you want an even more solid Bazel solution you can write Daml rules in Skylark. You can find examples of this in the Daml repo.

I think the only way to get bazel rules and a working Daml Studio is probably to do what you suggested (dependency path in bazel_out) or at best prettifying that via a symlink.

1 Like
3

Thank you very much, @bernhard , for your help.

How does daml.yaml need to list the dependency on B.dar for damlc to pick it up from the component A’s lib folder? Or is this a runtime flag for damlc?

For me damlc is looking for B.dar in /usr/local/.daml/sdk/1.9.0/daml-libs/ and obviously can’t find it there.

I will check out the Skylark implementation. I was just hoping I could get away with genrules.

Regards,
Alex

4

The stanza in the daml.yaml of A should be something like

dependencies:
  - daml-prim
  - daml-stdlib
  - daml-script
  - ./libs/B.dar

ie if you give it a relative or absolute path, it’ll look there, not in the daml-libs folder of the SDK.

1 Like
5

Thank you, @bernhard, this works.

For the records, this is my BUILD file for A:

genrule(
    name = "A",
    srcs = glob(["src/**"]) + ["daml.yaml", "//path/to/B:B"],
    outs = ["A.dar"],
    cmd = """
        mkdir -p path/to/A/lib
        cp $(location //path/to/B:B) path/to/A/lib
        daml build --project-root path/to/A -o $@
)

and my daml.yaml looks just like what you have quoted.

dependencies:
  - daml-prim
  - daml-stdlib
  - daml-script
  - ./libs/B.dar

Thanks again.
Alex

1 Like
6

This is a common problem when using Bazel with off-the shelf, language-specific build systems (I’d count daml build as one in this case). If you hand over control fully to Bazel, things like IDE integrations break which is obviously annoying (this is the approach taken by rules_daml @bernhard pointed you to above). If you don’t, then you have to keep things in sync. In your example that means manually updating libs/B.dar when it changes (or try to symlink it from the bazel dir but that gets increasingly hard once you take different operating systems, optimization levels, … into account all of which influence the directory bazel uses).

There are a few options to get out of this conundrum that different languages have chosen:

  1. Instead of pointing your IDE directly at your underlying tool (damlc in this case) you point it at some wrapper script which queries Bazel to build and locate depedencies and then invokes the underlying tool with some extra flags to make it find dependencies. This is the approach taken by Haskell tooling for example and Scala also does something along those lines.
    For Daml, this would mean that you have a daml.yaml with an empty list of data-dependencies/dependencies. You then need your Bazel build setup to produce package databases, e.g., by wrapping daml damlc init or maybe even go lower-level and construct them manually (unfortunately no docs on this atm). Finally, your wrapper script would find the location of those package databases and then invoke daml damlc ide --package-db=… --package-db= … --package=… --package=… and your IDE should find dependencies. This is probably the approach I’d recommend but it is a fair amount of effort.
  2. Another option taken by rules_nodejs is to have Bazel step out of its build root and mess around in the actual source tree, e.g., modify node_modules. In this case, this could mean modifying daml.yaml to point to the location of the dependencies. That requires in some sense less effort since you don’t need the wrapper script but it also gives up on Bazel managing your stuff cleanly and you get back to the good old days where rm -rf node_modules solves your problem (or whatever the equivalent of that in Daml is :wink: )
  3. Lastly, you could also imagine some kind of native support in the compiler to make the location of source files and dependencies configurable. I’m not really aware of any language that has proper support for this atm that works well with Bazel but it has at least been discussed for Go in the context of Expose generated Go files to editors and IDEs · Issue #512 · bazelbuild/rules_go · GitHub.

So unfortunately, while I think making 1 work is definitely feasible, the current situation is not great. I’ve opened Improve UX when compiling Daml code in Bazel or other build systems · Issue #8767 · digital-asset/daml · GitHub to track this on our side and we’ll see what we can do to make the UX here a bit better.

1 Like
7

Thank you very much, @cocreature, for this extensive response. This is really helpful, since I have been wondering about these things.

From my perspective the native tools should indeed be highly configurable with where they read input from and where they write output to. I guess that would be option 3.

I wonder into which category the Java and C++ toolchains fall. They seem to be rather well supported by Bazel.

Regards,
Alex

8

Java and C++ are a bit special since they are natively supported by Bazel rather than being based on a separate rule set. You still run into the same issue though. Your IDE tooling might expect a Maven project or a CMake project and you have to teach it how to handle Bazel. Not quite sure how that works there.