Skip to content
This repository has been archived by the owner on Feb 23, 2023. It is now read-only.

Data Management

Jump to bottom
Andrei Tsaregorodtsev edited this page Jan 26, 2015 · 10 revisions

COMDIRAC provide a complete set of commands for Data manipulation through the DIRAC FileCatalog (FC).

Data management commands are strongly inspired from iRODS ICommands.

DCommands Working Directory

After Session Initialization, your environment has a working directory option used to store the path inside the FileCatalog you can use to specify relative paths. These relative paths aren't to be confused with relative paths from your operating shell.

There are two commands associated with your FileCatalog working directory:

  • dpwd - same as shell pwd commands for FileCatalog context
  • dcd - change your FileCatalog working directory

For example (supposing you just initiated your DCommands session):

$ # print FC working directory
$ dpwd
/vo.france-grilles.fr/user/p/pgay
$ # moving "up"
$ dcd ..
$ dpwd
/vo.france-grilles.fr/user/p
$ # go back to "home_dir" as stored in DCommands session
$ dcd
$ dpwd
/vo.france-grilles.fr/user/p/pgay

FileCatalog Directories

There are some DCommands related to FileCatalog directory manipulation:

  • dls - list FC directory contents
  • dmkdir - create directory
  • drmdir - remove directory

Examples:

$ # create a test directory
$ dmkdir tutorial
Successfully created directory: /vo.france-grilles.fr/user/p/pgay/tutorial
$ dcd tutorial
$ # create a subdirectory
$ dmkdir d1
Successfully created directory: /vo.france-grilles.fr/user/p/pgay/tutorial/d1
$ # two subdirectories at once
$ dmkdir d2 d3
Successfully created directory: /vo.france-grilles.fr/user/p/pgay/tutorial/d2
Successfully created directory: /vo.france-grilles.fr/user/p/pgay/tutorial/d3
$ # inspect current directory
$ dls
/vo.france-grilles.fr/user/p/pgay/tutorial:
d1
d2
d3
$ # more detailed output
$ dls -l
/vo.france-grilles.fr/user/p/pgay/tutorial:
drwxrwxr-x 0 pgay frangrilles_user 0 2013-05-02 20:34:24 d1
drwxrwxr-x 0 pgay frangrilles_user 0 2013-05-02 20:34:31 d2
drwxrwxr-x 0 pgay frangrilles_user 0 2013-05-02 20:34:32 d3
$ dcd ..
$ dpwd
/vo.france-grilles.fr/user/p/pgay
$ # won't work
$ drmdir tutorial
lfn: /vo.france-grilles.fr/user/p/pgay/tutorial
ERROR: Failed to remove non-empty directory
$ # only empty directoris can be removed this way. Be patient
$ drmdir tutorial/d1
lfn: /vo.france-grilles.fr/user/p/pgay/tutorial/d1
Directory /vo.france-grilles.fr/user/p/pgay/tutorial/d1 removed from the catalog
$ drmdir tutorial/d2
lfn: /vo.france-grilles.fr/user/p/pgay/tutorial/d2
Directory /vo.france-grilles.fr/user/p/pgay/tutorial/d2 removed from the catalog
$ drmdir tutorial/d3
lfn: /vo.france-grilles.fr/user/p/pgay/tutorial/d3
Directory /vo.france-grilles.fr/user/p/pgay/tutorial/d3 removed from the catalog
$ drmdir tutorial
lfn: /vo.france-grilles.fr/user/p/pgay/tutorial
Directory /vo.france-grilles.fr/user/p/pgay/tutorial removed from the catalog

Downloading/Uploading Files From/To DIRAC

To send files from your local computer to File Catalog and Storage Elements, use the dput command. dget command allows you to get files back from the grid to your client.

  • dput - works like a kind of scp command (in the local->remote direction):
  • dput <local-source-path> [<lfn-destination-file-or-directory>]
  • or dput <local-source-path>... <lfn-destination-directory>
  • dget - works in the opposite direction (remote->local):
  • dget <lfn-source-path>
  • dget <lfn-source-path>... <local-destination-directory>

Note that if you are using multiple source arguments in dput or dget, you have to give a destination directory argument.

Examples:

$ # suppose you have a file named "file1" in your local working directory
$ dmkdir tutorial
$ dcd tutorial
$ # register file1 to DIRAC File Catalog using default Storage Element
$ dput file1
File /vo.france-grilles.fr/user/p/pgay/tutorial/file1 successfully uploaded to the DIRAC-USER SE
$ dls -l
/vo.france-grilles.fr/user/p/pgay/tutorial:
-rwxrwxr-x 0 pgay frangrilles_user 6 2013-05-03 20:26:17 file1
$ # get file1 back
$ dget file1 /tmp/
File /vo.france-grilles.fr/user/p/pgay/tutorial/file1 successfully downloaded
$ # compare initial and download file
$ cmp file1 /tmp/file1
$ # now, try with multiple files
$ dput file2 file3
File /vo.france-grilles.fr/user/p/pgay/tutorial/file3 successfully uploaded to the DIRAC-USER SE
$ # wrong! dput command has sent file2(local) to file3(FC)
$ # let's remove file3(FC)
$ drm file3
$ # good syntax: use lfn relative path to destination directory
$ dput file2 file3 ./
File /vo.france-grilles.fr/user/p/pgay/tutorial/file2 successfully uploaded to the DIRAC-USER SE
File /vo.france-grilles.fr/user/p/pgay/tutorial/file3 successfully uploaded to the DIRAC-USER SE
$ # error: try to overwrite file3(FC)
$ dput file3
putAndRegister: This file GUID already exists for another file. Please remove it and try again. True
Error: putAndRegister: This file GUID already exists for another file. Please remove it and try again. True
$ dls -l
/vo.france-grilles.fr/user/p/pgay/tutorial:
-rwxrwxr-x 0 pgay frangrilles_user  6 2013-05-03 20:26:17 file1
-rwxrwxr-x 0 pgay frangrilles_user  8 2013-05-03 20:42:39 file2
-rwxrwxr-x 0 pgay frangrilles_user 10 2013-05-03 20:42:41 file3
$ # now download all files at once
$ dget file1 file2 file3 /tmp
File /vo.france-grilles.fr/user/p/pgay/tutorial/file1 successfully downloaded
File /vo.france-grilles.fr/user/p/pgay/tutorial/file2 successfully downloaded
File /vo.france-grilles.fr/user/p/pgay/tutorial/file3 successfully downloaded
$ # succeeded: local /tmp/file1 was overwritten

As you have noticed, we also introduced a drm command that can delete files from the FileCatalog and all Storage Elements where it is put. You can remove several files in one command by giving the names in the command line:

$ drm file1 file2 file3

Or you can save a list file names in a file, for example, the output of dfind command, and pass this file name as an argument:

$ dfind /vo.france-grilles.fr/user/p/pgay/tutorial > tutorial.files
$ drm -F tutorial.files

One can remove a whole directory recursively with contained files and subdirectories:

$ drm -r /vo.france-grilles.fr/user/p/pgay/tutorial

Be careful as this can be a lengthy operation as removing files on the grid is quite complex because all the physical replicas on distributed storage elements must be deleted.

Destination Storage Element

In the last example, we used dput with the default Storage Element. This can be overidden with dput --destination-se=SOME-SE-disk on the commandline or also by setting destination_se session option:

$ dsetenv default_se=SOME-SE-disk

Or if you want to modify default_se for future sessions, set it in the profile configuration by editing ~/.dirac/dcommands.conf.

File replication

One of the powerfull features of Grid middlewares and more specifically DIRAC is to allow users to register files in a central Catalog and store several copies of these files in different server worldwide.

DCommands provide an easy way to make replicas of a file already registered in the Fle Catalog:

$ # register a new file and store it in a LCG SE
$ dput -D MSFG-disk file4
File /vo.france-grilles.fr/user/p/pgay/tutorial/file4 successfully uploaded to the MSFG-disk SE
# replicate this file to another SE
$ drepl -D M3PEC-disk file4
{'Failed': {},
 'Successful': {'/vo.france-grilles.fr/user/p/pgay/tutorial/file4': {'register': 0.37358498573303223,
                                                                     'replicate': 16.705672979354858}}}
File /vo.france-grilles.fr/user/p/pgay/tutorial/file4 successfully replicated to the M3PEC-disk SE
$ # list file replicas
$ dreplicas file4
lfn: /vo.france-grilles.fr/user/p/pgay/tutorial/file4
M3PEC-disk      srm://se0.m3pec.u-bordeaux1.fr:8446/srm/managerv2?SFN=/dpm/m3pec.u-bordeaux1.fr/home/vo.france-grilles.fr/user/p/pgay/tutorial/file4
MSFG-disk       srm://lptadpmsv.msfg.fr:8446/srm/managerv2?SFN=/dpm/msfg.fr/home/vo.france-grilles.fr/user/p/pgay/tutorial/file4
$ # or mix dls output with list of replicas for each file
$ dls -L
/vo.france-grilles.fr/user/p/pgay/tutorial:
-rwxrwxr-x 0 pgay frangrilles_user    6 2013-05-03 20:26:17 file1
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file1
-rwxrwxr-x 0 pgay frangrilles_user    8 2013-05-03 20:42:39 file2
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file2
-rwxrwxr-x 0 pgay frangrilles_user   10 2013-05-03 20:42:41 file3
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file3
-rwxrwxr-x 0 pgay frangrilles_user    8 2013-05-03 22:29:38 file4
   M3PEC-disk      srm://se0.m3pec.u-bordeaux1.fr:8446/srm/managerv2?SFN=/dpm/m3pec.u-bordeaux1.fr/home/vo.france-grilles.fr/user/p/pgay/tutorial/file4
   MSFG-disk       srm://lptadpmsv.msfg.fr:8446/srm/managerv2?SFN=/dpm/msfg.fr/home/vo.france-grilles.fr/user/p/pgay/tutorial/file4

File Removal

One can remove files from the File Catalog and all associated replicas:

$ dls -L
/vo.france-grilles.fr/user/p/pgay/tutorial:
drm-rwxrwxr-x 0 pgay frangrilles_user    6 2013-05-03 20:26:17 file1
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file1
-rwxrwxr-x 0 pgay frangrilles_user    8 2013-05-03 20:42:39 file2
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file2
-rwxrwxr-x 0 pgay frangrilles_user   10 2013-05-03 20:42:41 file3
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file3
-rwxrwxr-x 0 pgay frangrilles_user    8 2013-05-03 22:29:38 file4
   M3PEC-disk      srm://se0.m3pec.u-bordeaux1.fr:8446/srm/managerv2?SFN=/dpm/m3pec.u-bordeaux1.fr/home/vo.france-grilles.fr/user/p/pgay/tutorial/file4
   MSFG-disk       srm://lptadpmsv.msfg.fr:8446/srm/managerv2?SFN=/dpm/msfg.fr/home/vo.france-grilles.fr/user/p/pgay/tutorial/file4
$
$ # remove some file
$ drm file4
$ # check FC again
$ dls -L
/vo.france-grilles.fr/user/p/pgay/tutorial:
-rwxrwxr-x 0 pgay frangrilles_user    6 2013-05-03 20:26:17 file1
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file1
-rwxrwxr-x 0 pgay frangrilles_user    8 2013-05-03 20:42:39 file2
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file2
-rwxrwxr-x 0 pgay frangrilles_user   10 2013-05-03 20:42:41 file3
   DIRAC-USER      dips://ccdirac04.in2p3.fr:9148/DataManagement/StorageElement/vo.france-grilles.fr/user/p/pgay/tutorial/file3
$ # ok

You can remove several files in one command by giving the names in the command line:

$ drm file1 file2 file3

Or you can save a list of file names in a file, for example, the output of dfind command, and pass this file name as an argument to drm:

$ dfind /vo.france-grilles.fr/user/p/pgay/tutorial > tutorial.files
$ drm -F tutorial.files

One can remove a whole directory recursively with contained files and subdirectories:

$ drm -r /vo.france-grilles.fr/user/p/pgay/tutorial

Be careful as this can be a lengthy operation as removing files on the grid is quite complex because all the physical replicas on distributed storage elements must be deleted.

Managing ACLs

dls -l commands displays a UNIX-like set of permissions for each listed file or directory.

These permissions can be changed with the dchmod command:

$ dls -l
/vo.france-grilles.fr/user/p/pgay/tutorial:
-rwxrwxr-x 0 pgay frangrilles_user  6 2013-05-03 20:26:17 file1
-rwxrwxr-x 0 pgay frangrilles_user  8 2013-05-03 20:42:39 file2
-rwxrwxr-x 0 pgay frangrilles_user 10 2013-05-03 20:42:41 file3
$ # restrict access to some file
$ dchmod 700 file3
$ dls -l
/vo.france-grilles.fr/user/p/pgay/tutorial:
-rwxrwxr-x 0 pgay frangrilles_user  6 2013-05-03 20:26:17 file1
-rwxrwxr-x 0 pgay frangrilles_user  8 2013-05-03 20:42:39 file2
-rwx------ 0 pgay frangrilles_user 10 2013-05-14 14:20:08 file3

Usual equivalent of chown and chgrp commands are also available:

$ dchown vhamar file1
$ dchgrp dirac_user file2
Clone this wiki locally