Add support for parsing dependencies from *.cabal files

[andrew]

Will very likely need to be parsed by a haskell programming or shelling out to $ cabal

Example: https://github.com/cakesolutions/the-pragmatic-haskeller/blob/master/01-json/pragmatic.cabal

Name:                pragmatic
Version:             0.1
Synopsis:            Project Synopsis Here
Description:         Project Description Here
License:             AllRightsReserved
Author:              Author
Maintainer:          [email protected]
Stability:           Experimental
Category:            Web
Build-type:          Simple
Cabal-version:       >=1.2

Executable pragmatic
  hs-source-dirs: src
  main-is: Main.hs

  Build-depends:
    aeson                     >= 0.6.0.1 && < 0.7,
    base                      >= 4       && < 5,
    bytestring                >= 0.9.1   && < 0.11,
    text                      >= 0.11    && < 0.12

Documentation of the file format: https://www.haskell.org/cabal/users-guide/developing-packages.html

Some of the documentation in hpack might be useful for better understanding it: https://github.com/sol/hpack

[alunduil]

I'd love to help out with this but as I saw noted in #175 the documentation on how to add a package manager parser isn't available. Is there a quick start or at least a list of methods with signatures? I can probably figure most of it out by reading the others but would like to read something definitive if it's available.

[andrew]

@alunduil awesome, thanks for offering! I don't think I'll get chance to complete the documentation before I break for christmas but in the meantime check out the collection of existing parsers: https://github.com/librariesio/bibliothecary/tree/master/lib/bibliothecary/parsers

For cabal support, unless you're thinking of using regexes, I'd guess we'll need a separate service for parsing cabel files, perhaps a haskell webservice?

[alunduil]

@andrew, now you're speaking my language. I'd love to avoid regexes if possible so a Haskell web service sounds great. Does any other parser use that pattern so I could see how you'd like it setup or should I design something that provides a list of packages with ranges and call it good? Also, where would you like that code to live?

[alunduil]

@andrew just a shameless bump on this one.

[andrew]

@alunduil ah yeah, forgot about this whilst I was on holiday over xmas.

https://github.com/librariesio/yarn-parser is an example one of these services, it accepts a post request with the body of file contents and returns json data that looks something like this:

[
  {
    "name": "foobar",
    "requirement": ">0.1",
    "type": "runtime"
  }
]

Where:

• name is the package name on https://hackage.haskell.org/package/lens
• requirement is the version range of the dependency, usually a semver range or exact version number
• type is where the dependency is used, usually runtime, development or test but can be any specific strings for the given package manager functionality

n.b. Yarn-parser returns version instead of requirement due to legacy reasons

[alunduil]

@andrew awesome! I can whip something up for that. I'll start a new repository for cabal-parser and if it looks good you we can transfer ownership.

Can you provide an example of a requirement that's closed and not just a half-open one? What about wildcards, will those need to be translated?

Thanks for all of the guidance.

[andrew]

@alunduil I don't think you'll need to do any translation of the requirements, just pass them straight through as is, if it's blank then maybe default to *

Looking at the example in https://downloads.haskell.org/~ghc/7.0.2/docs/html/Cabal/authors.html#buildinfo

library
  build-depends:
    base >= 2,
    foo >= 1.2 && < 1.3,
    bar

would translate to

[
  {
    "name": "base",
    "requirement": ">= 2",
    "type": "build"
  },
  {
    "name": "foo",
    "requirement": ">= 1.2 && < 1.3",
    "type": "build"
  },
  {
    "name": "bar",
    "requirement": "*",
    "type": "build"
  },
]

[alunduil]

@andrew Awesome! I've almost got the basic structure of the web service put together. Hopefully by this weekend I can have it functioning and ready for review.

[andrew]

@alunduil sweet, looking forward to it, thanks for contributing!

[alunduil]

@andrew I've got a working version. I passed through all versions to requirements unmodified except the missing or any version. I changed any version to "*" as you suggested.

Take a look at https://github.com/alunduil/librariesio-cabal-parser and let me know what you think. I've run it against a couple of my projects and it appears to work as intended but I've got a couple of tests to add more checks later.

If this looks good, let me know if you want to bring it into librariesio and I'll start working on the bibliothecary side of the project as well.

[andrew]

Just attempting to set this up locally and learn me some haskell, looking good though 👍

[andrew]

I got it running locally and working with bibliothecary 🎉

I had to made a few changes to docker-compose.yml to get it exposing the internal port to localhost on my laptop:

version: '3'

services:
  app:
    build: ./
    ports:
      - 5000:5000

One thing that would be nice to add to the parsing service would be some logging to make it a little easier to keep track of the internal state of the application, just to STDOUT would be fine.

[alunduil]

@andrew what kind of logging are you looking for? Just apache style logs or something nicer like APM data?

[andrew]

@alunduil apache style logs are fine, mainly to see frequency and response time of the endpoint being hit

[alunduil]

@andrew Sounds good. I'll add that in today and let you know.

[alunduil]

@andrew I've added basic Apache logging but it doesn't include response time unless I add custom logic. Let me know if what's on master is acceptable, otherwise I'll see about adding response time as well.

[alunduil]

@andrew just a friendly bump in case this has been forgotten. Let me know if any more changes would be nice to have or what next steps would be.

[andrew]

@alunduil sorry for the delay, I'll be deploying your parsing service soon and then merging the bibliothecary pr, should be ready to go on Monday

[alunduil]

@andrew sweet! Let me know if I can do anything else before then. Do you have an interest in moving the cabal-parser repository into the libraries.io organization?

[alunduil]

@andrew just checking in to see how the deployment is going. Let me know if I can help with anything.

[andrew]

@alunduil sorry, not got round to it yet, FOSDEM is this weekend and has kind of destroyed all my productivity!

[alunduil]

@andrew no worries. I was afraid that might interfere with things. I'll ping you again mid to late next week if I don't hear anything. Enjoy FOSDEM!

[andrew]

Deployed 🚢 thanks for contributing!

[alunduil]

@andrew, awesome! Thanks for everything!

[andrew]

Just enqueued about 50,000 jobs to parse all the cabal dependencies every haskell repo on github. Example from https://libraries.io/github/egison/egison

[alunduil]

@andrew That's awesome! I'm excited to see my repositories and packages fill in as well. Any estimate on how long that run will take?

[andrew]

Looks like it's mostly finished now

[alunduil]

@andrew seems like my repositories are stuck in syncing. Want me to open another issue for that? Also, when I force a private repo to sync on libraries.io it doesn't pick up the cabal dependencies.

[andrew]

Yeah a second issue for that would be good