Ledger API protobuf files on Maven

Questions
1

Hi,

We are trying to generate both Client and Server gRPC bindings in Scala using Akka gRPC (based on ScalaPB). How can we include the ledger API .proto files as a dependency in our sbt build setup?

We would like to depend on the proto files that match the SDK version of the rest of our application. We have ruled out hacking a script together that downloads the files from Github during the build process.

What I have done so far (click to expand):

Summary

I have found the files in two outdated artifacts, here and here, but unfortunately both stopped to be published in about March this year. They are also in a format difficult to use for build automation (.tar.gz without .pom)

I have gone on to check the published source artifacts for various daml jars. The published sources of artifacts like proto-google-common-protos contain the .proto files that ScalaPB and other tools can use to generate their own bindings. It would be nice if this was the case for protobuf-generated jars such as ledger-api-scalapb (currently empty)

We need the Server bindings to provide proxying capabilities for critical tests. We cannot also depend on the official Scala (client-side) bindings for the rest of our application due to unsolvable dependency conflicts surrounding protobuf codegen.

Is there a way, or can we make a way, to include the .proto files as a dependency in the JVM universe? This would also enable other users that have a desire / need to generate their own bindings with custom configuration.

1 Like
2

Hi @iggy,

We currently publish and maintain the following artifacts:

  1. The zip archive with the protobuf files on Github releases.
  2. The Java bindings which include the generated protobuf at https://mvnrepository.com/artifact/com.daml/bindings-java
  3. The generated Scala code using scalapb at https://mvnrepository.com/artifact/com.daml/ledger-api-scalapb. Here the bindings that build on top of the generated code are in a separate package. We do have bindings to integrate this with akka at https://mvnrepository.com/artifact/com.daml/bindings-akka.

I would recommend to stick to the bindings we provide but if you want to generate your own code, the protobufs on Github releases are your best option. A lot of build systems allow you to download zip archives already and if yours doesn’t a script to download it upfront should be easy to setup. We don’t have any plans to distribute them via Maven at this point.

3

For others looking for this: We did not want to permanently rely on generated (assumed) Github URLs in our build. So we merged a pull request in the official DAML repository on Github so that (according to convention and best practice) ledger-api-scalapb will now include the Protobuf files. This means tooling such as ScalaPB or Akka gRPC will now find them where it expects them, and can generate bindings from them. This should be available in DAML SDK version 1.7.0 and later.

As a fallback for earlier versions, if using Scala with sbt, you can use the following script to download the Protobuf files from Github:

lazy val downloadLedgerProtobuf = taskKey[Unit]("Download Ledger API protobuf files from Github")
downloadLedgerProtobuf := {
  if (VersionNumber(sdkVersion).matchesSemVer(SemanticSelector("<1.7.0"))) {
    IO.createDirectory(target.value / "download")
    val filename   = s"protobufs-$sdkVersion.zip"
    val targetFile = target.value / "download" / filename
    if (!targetFile.exists()) {
      val sourceUrl = s"https://github.com/digital-asset/daml/releases/download/v$sdkVersion/$filename"
      (url(sourceUrl) #> targetFile).!
    }
    val temp = IO.createTemporaryDirectory
    IO.unzip(targetFile, temp, !_.contains("/daml_lf")).toSeq
    IO.copyDirectory(temp / s"protos-$sdkVersion", PB.externalSourcePath.value)
  }
  ()
}

// then in your settings
(Compile / PB.generate) := ((Compile / PB.generate) dependsOn downloadLedgerProtobuf).value