Directory management

[SamGG]

Hi,

Following the issue #76 I start here a discussion concerning directories management. Thomas told about the next major release, and I am wondering if the devel team has already started thinking about directories management? IMO, this is one of the difficult notions to transfer to experimenters.

When people are using CytoBank or OMIQ (the ones for which I viewed demonstrations), they are locked into a framework. This could be viewed as a constraint, but I think it is not. Once the framework is understood, the user does not care anymore about it because navigating through the various steps of the analysis has become "natural" and no brainer. So people can concentrate on learning the data analysis. In fact, people don't have to care about where are files (FCS, CSV, reports...) on disk because the framework do care of them for the user. The user should only care when the files are downloaded to the local computer.

I think OMIQ is offering a nice hierarchical organization of processing steps and result files. IMO, it is the same as the one of TMEV (aka Multi-Experiment Viewer orchestrated by J. Quackenbush). Its advantage is that one always knows where the input data is originating from. One alternative might be to use the approach of the Galaxy project. But it does not result in a clear organisation of directories on disk.

So my point is that such a hierarchy should be adopted. I hope some people will propose interesting discussions and alternatives.

Best,
Samuel

[ghar1821]

Hi Samuel,

I guess with raw R script, you do have to care about where the raw data files are stored at least for the step where the files are read in. For the remaining functions, especially the ones which does the actual analysis (clustering, dim red), in my opinion, none should perform any directory changing. This is because ideally, the data should already be stored in the variables. Functions which export any kind of files (CSV or PDF/PNG plots) should optionally allow user to specify where the files will be stored, if this path is not given, then it'll default to storing to the current working directory.

@tomashhurst ?

[SamGG]

Hi Givanna,

I agree with you concerning the variables, tables... As they are not saved on disk, the directory could remain simple. Nevertheless, I started with the simple discovery workflow, and the intermediate variables/tables are saved, although this is not mandatory. While I agree that every output file could end up in a single output directory, I find organizing the output files by step is easier for my brain and to transfer results to the biologist. It is also easier to explain during workshops.

I used the simple discovery workflow to put my thoughts as code lines. So here is the final linear structure. This is not amazing but I find it useful and will pursuit this idea. The hierarchical structure is more complex to code. I will go with it later on.

At the end (or at any step), it possible to get the structure of the directory, which could be put into an HTML file with links.

Let me know.