Investigate following symbolic links and flagging file as symlink

[rsmith013]

There are two desired views of the archive.

1. The view we present to the world, using symlinks to create logical groupings
2. The view for management purposes where we only want to know where the actual files are

Proposed solution:

• follow links when building file lists
• when scanning. Turn the path into the normalised path (where the file actually is) and compare to
  the "feed-in-path". If they are the same, then we have an actual file eg. symlink = False. If they are
  different, we have a symlinked route to the file. symlink = True

[agstephens]

@rsmith013 will that work for the "extending" cases? We don't really want the real path in those cases, do we? I think we want to blindly follow symlinks when they resolve to a directory.

[rsmith013]

@agstephens Not sure. Need to investigate.

[gap736uk]

those cases would be easy to spot as they always link to a path starting "/datacentre/archvol" so you can put in a check for that and there's no need to mark them as symbolic.

however - is there a use case to check here for Sam's purpose, i.e. knowing where in the structure such storage extension links are used ? Or could that be pulled back from the cedaarchiveinfoDB app thingy?

[rsmith013]

@gap736uk @agstephens Do you have an example of a file which is an extending case?

I have examples for the following cases:

DIR LINK

link_path:

/badc/ncas-observations/data/man-radar-1290mhz/previous-versions/2002/08/31/aber-radar-1290mhz_macehead_20020831_hig-res-1h-1.na

archive_path:

/badc/namblex/data/aber-radar-1290mhz/20020831/aber-radar-1290mhz_macehead_20020831_hig-res-1h-1.na

storage_path:

/datacentre/archvol3/pan125/archive/namblex/data/aber-radar-1290mhz/20020831/aber-radar-1290mhz_macehead_20020831_hig-res-1h-1.na

FILE LINK

link_path:

/badc/cmip5/data/cmip5/output1/MOHC/HadCM3/historical/day/atmos/day/r10i1p1/latest/tas/tas_day_HadCM3_historical_r10i1p1_19841201-20051230.nc

archive_path:

/badc/cmip5/data/cmip5/output1/MOHC/HadCM3/historical/day/atmos/day/r10i1p1/files/tas_20110728/tas_day_HadCM3_historical_r10i1p1_19841201-20051230.nc

storage_path:

/datacentre/archvol3/pan104/archive/spot-390/day/atmos/day/r10i1p1/files/tas_20110728/tas_day_HadCM3_historical_r10i1p1_19841201-20051230.nc

EXTENSION LINK

link_path:

/badc/ecmwf-era-interim/data/gg/ap

archive_path:

/badc/ecmwf-era-interim/data/gg/ap

storage_path:

/datacentre/archvol3/pan123/archive/spot-447

[agstephens]

@rsmith013 , I'm not sure what you mean by a "file which is an extending case" - please elaborate :-)

[rsmith013]

@agstephens It's ok. I have an example. I just mean links that go out to another storage pot.
eg cmip5 is a spot which has lots of sub-spots. These would be extension links.