Add versioning to conda envs

[jerowe]

270

I would like to add user controlled versioning to conda envs. I've got the uploader part done, but I still need to do some checks for creating. Here is my initial effort.

Best,
Jillian

[jerowe]

You can now add a version tag to an environment.yml file, which will be correctly propogated to anaconda cloud.

In addition you can create a specific version of an environment.

conda env create jerowe/gencore_build_1.0/1.0    #version
conda env create jerowe/gencore_build_1.0          #gets the latest version

[jerowe]

Hi @msarahan. You took a look at my last PR. Could you take a look at this? If not something the conda env community is interested in I completely understand, especially with some of the cool new changes like jinja templates you guys want to implement. If you could give me a general yay/nay that would be excellent.

Best,
Jillian

[msarahan]

This seems like a debugging message. Can it be cut or turned into a log message instead?

[jerowe]

I removed it.

Is there a logging utlity?

[msarahan]

@jerowe I think this is a nice addition. I have made some minor comments. @kalefranz is in charge of conda-env, and will need to make the decision on this PR. He may want you to put this PR against conda/conda instead, as he is trying to merge conda and conda-env.

[jerowe]

@msarahan, thanks for the comments!

I thought I would go into a bit of why we want this.

I worked for a genomics core closely beside the HPC department, and conda along with conda env has become a major part of our infrastructure. We have a whole workflow from researchers requesting software stacks for analyses, configuring those analyses using conda env environment.yml files, and building those environments against a docker image to check for conflicting versions. Each of these conda envs is then built as a HPC module.

Conda env is wonderful, but what would really take it to the next level is being able to version the envs. For instance, more than one researcher may be using basically the same environment, but each needs a different version of software X for whatever reason. Or there are some major updates in the software stack. These envs would be gencore_cure_cancer/1.0 and gencore_cure_cancer/1.1.

In any case we have had great success with conda env. Thanks for the great project! ;-)

[msarahan]

Yes, I understand the use case, and I agree it makes a lot of sense. As an intermediate, you could always place your environment.yml scripts under version control, and check out tags. I understand the barrier to entry there is higher, and you also have less direct knowledge of the environment, though.

[jerowe]

@msarahan, what we're doing now is actually appending the version names to the environment names themselves. Environment names are gencore_env_1.0, gencore_env_1.1, but when the HPC module is built it is gencore_env/1.0. It is working, so its not as if its a huge hurtle. I just like the idea of versioned environments better. Environment files are already under version control, but I hadn't thought of tags. I was just hard coding everything. Things to think about!

[jerowe]

@msarahan , I have a question about code change.

According to anaconda_client the function 'get_binstar' has been deprecated in favor of 'get_server_api'. They both take the same arguments, so substituting them would be a simple change. Should I go ahead and make that change while I'm at it?

[jerowe]

Alright, I tried to do some refactoring, but then broke the mock tests. I added back in a binstar, and will wait for feedback before I go all out with refactoring.

ETA - I'm sorry. I just can't help myself.

[jerowe]

I'm not a real python developer, and I have few questions if you care to answer.

What is the point of this?

class BinstarSpec(object):
    _file_data = None
...
    @property
    def file_data(self):
        if self._file_data is None:
            self._file_data = [data for data in self.package['files'] if data['type'] == ENVIRONMENT_TYPE]
        return self._file_data

We have _file_data, and we have a property file data. Do I need to just go and read up on property decorators? Why not just return the [data for data STUFF] the first time around?

[jerowe]

@msarahan, any word on this feature? It would be very useful for us to have!

[jerowe]

@msarahan , how is it going?

Any word? We'd really like to have this feature. I would also really like to add summary and tags/labels fields, but versioning is the most important.

[msarahan]

Sorry, I am out on paternity leave. Pinging @kalefranz.

At the very least, you probably need to merge this into conda instead, as conda env had been rolled into conda proper.

[jerowe]

Congratulations! I have a 2 year old and a 5 year old. They are the best. Enjoy your time with the baby!

[jerowe]

@kalefranz, What do you think?

I am very interested in versioning, summaries, and tags, but I understand if this is not something the conda community is not interested in.