Skip to content

AddonDepgraph

Jump to bottom Edit New page
John Casey edited this page Feb 13, 2014 · 21 revisions

Depgraph Add-On

The Depgraph Add-On is an integration point between AProx and the Cartographer project. It came out of the recognition that the immutable relationships expressed in Maven POM metadata is far more useful than simply as another file to be downloaded. These relationships can be stitched together into a graph that can be saved an traversed at will with a host of different task-specific filters. Cartographer is the spin-off project resulting from exploring this potential, and provides the ability to embed this graphing logic into any application. The result of this spin-off is a new, leaner Depgraph add-on, which simply provides AProx integration and a suite of thin REST wrappers around the Cartographer APIs.

Overview of REST Resources

As of the 0.11.2 release, Depgraph can:

  • Manage Atlas DB workspaces
    • Workspaces provide a durable configuration of profiles and source URIs allowed in depgraph results, plus saved selections for snapshots, version ranges, and similar
    • Basic CRUD operations, plus addition of activated profiles and allowed source URIs
    • Workspaces can be named or assigned a generated serial number
  • Resolve dependency graphs
  • Add or query metadata attached to GAVs
  • Query for particular relationships on either the project (direct) or graph (indirect OR direct) level
  • Calculate things like intersection, addition, subtraction, or simple diff between two graphs (or even filtered views of the same graph)
  • Generate specific output formats of the relationships contained within a depgraph
    • BOMs
    • Graphviz files
    • dependency:tree -like output
    • Raw JSON
    • JSON mapping of GAV to list-of-urls with options to configure the artifact attachments to include in the list
    • Dense mock-up of the Maven console output's Downloading: ... lines, which are consumed by some tools

Repository Generation

Repository generation commands are various ways of formatting a list of files for the artifacts in a particular (filtered?) depgraph. Some are particularly useful for handing off to external tools for the purpose of bootstrapping a repository specially for a project build, or something similar, while the zip command produces content that can be used directly by Maven. Currently, the supported commands include:

  • urlmap (URL path: /aprox/api/1.0/depgraph/repo/urlmap) - This is a JSON mapping of GAV to a list of URLs that correspond to the main artifact, POM, and any attachments that are enabled by configuration.
  • downlog (URL path: /aprox/api/1.0/depgraph/repo/downlog) - This is a simple list of downloadable URLs, prefixed by 'Downloading ' to simulate a much denser version of the console output from an actual Maven build. The URLs will include main artifact, POM, and any attachments that are enabled by configuration.
  • zip (URL path: /aprox/api/1.0/depgraph/repo/downlog) - This is a .zip archive containing all artifacts resolved given the posted configuration.

All repository-generation operations include a resolve step that will discover the dependency graph before generating the output. This saves the need to resolve then query in two steps. It doesn't incur a lot of extra penalty in URL complexity because these operations already require the use of a POST'ed JSON configuration to function properly.

The basic configuration looks like this:

{
  "graphComposition": {
    "graphs": [
      {
        "roots": [
          "org.commonjava.aprox.wars:aprox-savant:0.11.2"
        ],
        "preset": "sob-build"
      }
    ]
  }
  "extras": [
    {
      "classifier": "sources",
      "type": "jar"
    },
    {
      "classifier": "javadoc"
    }
  ],
  "workspaceId": "aprox-0.11.2",
  "source": "group:public",
  "resolve": true
}

As a more advanced example, consider the case where you want all of the build-time dependencies and toolchain MINUS the runtime dependency sub-graph (in case you already have the runtime deps somewhere else). Your configuration might look like this:

{
  "graphComposition": {
    "calculation": "SUBTRACT",
    "graphs": [
      {
        "roots": ["org.commonjava.maven.atlas:atlas-driver-neo4j-embedded:0.9.5"],
        "preset": "managed-sob"
      },
      {
        "roots": ["org.commonjava.maven.atlas:atlas-driver-neo4j-embedded:0.9.5"],
        "preset": "sob-build"
      }
    ]
  },
  "extras": [
    {
      "classifier": "*", 
      "type": "*"
    }
  ],
  "localUrls": true,
  "workspaceId": "atlas-0.9.5",
  "source": "repository:central",
  "resolve": true
}

As you can see, this configuration is designed to allow multiple roots and multiple extra, attached artifacts in the generated output. Multiple roots are useful if you're building more than one top-level project in your environment, and these top-level projects may have overlapping depgraphs. Multiple attached artifacts are useful when you want to certain extras beyond the simple runtime bits in your repository. For instance, if you want to generate a repository to support IDEs, which often rely heavily on source jars.

NOTE: I've included a type field in one of the extras sections. The type jar is the default; I only included it here to illustrate that as a configuration point.

The urlmap Command

The basic command looks like this:

curl -X POST -H 'Content-Type: application/json' --data @config.json http://localhost:8080/aprox/api/1.0/depgraph/repo/urlmap
{
  "org.commonjava.maven.atlas:atlas-parent:0.8.0": [
    "http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-parent/0.8.0/atlas-parent-0.8.0.pom"
  ],
  "org.neo4j:neo4j-udc:1.9.RC2": [
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2-sources.jar",    
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2-javadoc.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2.pom"
  ],
  "commons-codec:commons-codec:1.6": [
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6.pom", 
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6-sources.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6-javadoc.jar"
  ],
  "org.neo4j:neo4j-kernel:1.9.RC2": [
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2-javadoc.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2.pom", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2-sources.jar"
  ],
...
}

The downlog Command

The basic command looks like this:

curl -X POST -H 'Content-Type: application/json' --data @config.json http://localhost:8080/aprox/api/1.0/depgraph/repo/downlog
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j/1.9.RC2/neo4j-1.9.RC2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-core/3.6.2/lucene-core-3.6.2.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-identities/0.8.0/atlas-identities-0.8.0-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-parent/3.6.2/lucene-parent-3.6.2.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/commonjava/2/commonjava-2.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/com/googlecode/concurrentlinkedhashmap/concurrentlinkedhashmap-lru/1.3.1/concurrentlinkedhashmap-lru-1.3.1-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-graph-matching/1.9.RC2/neo4j-graph-matching-1.9.RC2-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-graph-algo/1.9.RC2/neo4j-graph-algo-1.9.RC2-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-drivers-parent/0.8.0/atlas-drivers-parent-0.8.0.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-core/3.6.2/lucene-core-3.6.2.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-graph-matching/1.9.RC2/neo4j-graph-matching-1.9.RC2.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-relationships-api/0.8.0/atlas-relationships-api-0.8.0-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/commons/commons-parent/17/commons-parent-17.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-identities/0.8.0/atlas-identities-0.8.0.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-parent/0.8.0/atlas-parent-0.8.0.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-core/3.6.2/lucene-core-3.6.2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/log4j/log4j/1.2.12/log4j-1.2.12.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-jmx/1.9.RC2/neo4j-jmx-1.9.RC2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2-sources.jar
...

Deprecated Workflow

In its original form, Depgraph REST calls were designed to be used in conjunction with one another in a fairly complex interaction carried over multiple REST calls. This approach is considered deprecated in favor of the POSTed JSON configuration approach used by the repository workflows (above), though most of the Depgraph REST calls have not yet been converted.

In general, the older workflow consisted of the following rough steps:

  1. Establish a new workspace:
    $ curl -X POST http://localhost:8080/aprox/api/1.0/depgraph/ws/new
    {"config":{"forceVersions":true},"selectedVersions":{},"wildcardSelectedVersions":{},"id":"409","open":true,"lastAccess":1376968924979}
  1. Configure the workspace with active profiles, etc.:
    $ curl -X PUT http://localhost:8080/aprox/api/1.0/depgraph/ws/409/profile/foo
    {
      "config": {
        "activePomLocations": ["pom:profile:foo", "pom:root"],
        "activeSources": ["unknown:unknown"],
        "forceVersions": true
      },
      "selectedVersions": {},
      "wildcardSelectedVersions": {},
      "id": "409",
      "open": true,
      "lastAccess": 1376969003663
    }
  1. Discover the dependency graph for a project:
    $ curl 'http://localhost:8080/aprox/api/1.0/depgraph/resolve/group:public/org.commonjava.aprox.wars/aprox-savant/0.8.1?wsid=409&preset=sob'
    {"resolvedTopLevelGAVs": ["org.commonjava.aprox.wars:aprox-savant:0.8.1"]}

Notice that I'm using a preset named sob. This is the default, but I've included it to illustrate the preset feature, which sets up a hard-coded graph filter for some common task. In this case, 'sob' stands for Shipping-Oriented Builds. In other words, it includes all the runtime artifacts you might want to rebuild yourself, plus directly-referenced test- and provided-scope dependencies, which you'll need in order to run your builds. Beyond that, it also includes all referenced plugins and their runtime dependency-subgraphs, which you'll require in a cleanroom environment in order to be able to execute the build.

The full listing of current presets include:

  • none - This uses an AnyFilter, which means no filter at all. It traverses ALL relationships, and is quite exhaustive, as if you wanted to rebuild your projects and everything in their build-time environment. Recursively. You probably don't mean to use this filter, except maybe as a curiosity.
  • sob - Explained above.
  • sob-build - This roughly equates to the Maven runtime dependency closure, returning everything you might want to rebuild for your project.

  1. Query the resolved dependency graph.

    The simplest way to do this might be to generate a modified dependency:tree output for the graph:

       $ curl 'http://localhost:8080/aprox/api/1.0/depgraph/render/tree/org.commonjava.aprox.wars/aprox-savant/0.8.1?wsid=409&preset=sob'

       org.commonjava.aprox.wars:aprox-savant:0.8.1
         org.commonjava.aprox.wars:aprox-wars:0.8.1:pom:PARENT
           org.commonjava.aprox:aprox-parent:0.8.1:pom:PARENT
         org.commonjava.aprox:aprox-api:0.8.1:jar:compile (0)
           org.commonjava.aprox:aprox-parent:0.8.1:pom:PARENT
           org.commonjava.util:configuration-api:0.4:jar:compile (0)
           com.google.code.gson:gson:2.2.2:jar:compile (1)
           org.commonjava.web:json-serialization:0.5:jar:compile (2)
           commons-lang:commons-lang:2.6:jar:compile (3)
             org.apache.commons:commons-parent:17:pom:PARENT
               org.apache:apache:7:pom:PARENT
                 org.apache:apache:7:pom:PARENT
           org.commonjava.maven.galley:galley-transport-httpclient:0.1.0:jar:compile (5)
           org.commonjava.maven.galley:galley-core:0.1.0:jar:compile (6)
         org.commonjava.aprox:aprox-core:0.8.1:jar:compile (1)
           org.commonjava.aprox:aprox-parent:0.8.1:pom:PARENT
           org.apache.maven.archetype:archetype-catalog:2.2:jar:compile (0)
           org.commonjava.util:configuration-dotconf:0.4:jar:compile (7)
           org.apache.maven:maven-repository-metadata:3.0.3:jar:compile (8)
           org.hamcrest:hamcrest-core:1.1:jar:test (10)
           org.jboss.weld.se:weld-se-core:1.1.8.Final:jar:test (11)
           org.commonjava.qarqas:qarqas-test-harness:0.3:jar:test (12)
       ...

You'll notice the presence of PARENT listings. This is why I call it modified dependency:tree. The original plugin doesn't include these, which makes it hard to know where they're used, or what list of parent POMs need to be included when constructing a repository. For dependencies listed in this output, you'll notice a number at the end; this is the index of the dependency in the relevant POM file. If there are gaps in the sequence for any given POM, it's because this tree listing is suppressing duplicates in the same way Maven's collision management logic works.

  1. (OPTIONAL) Delete the workspace when you're done with it:

    $ curl -X DELETE http://localhost:8080/aprox/api/1.0/depgraph/ws/409

    I say this part is optional because the real strength of workspaces it their durability. You can setup and fine-tune a workspace for use with many related operations...say, for working on many projects in a single new environment you're setting up.

Add a custom footer
Add a custom sidebar
Clone this wiki locally