Skip to content

Commit

Permalink
Merge branch 'main' into 52-add-hooks-for-gui
Browse files Browse the repository at this point in the history
  • Loading branch information
@ntlhui
ntlhui authored Jan 29, 2025
2 parents 183cbd4 + dd9a076 commit 68d1e86
Show file tree
Hide file tree
Showing 11 changed files with 354 additions and 94 deletions.
90 changes: 90 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,87 @@
# CHANGELOG

## v0.2.0 (2025-01-11)

### Chore

* chore: Fixes naming ([`379e0a1`](https://github.com/UCSD-E4E/e4e-data-management/commit/379e0a157375a3bc25765cb3fbae50fc2bb2c85d))

### Feature

* feat: Adds zip ([`6cdc42b`](https://github.com/UCSD-E4E/e4e-data-management/commit/6cdc42b7e60853efa9a50aa2fc383bb043740279))

### Style

* style: Fixes styling ([`c3bead9`](https://github.com/UCSD-E4E/e4e-data-management/commit/c3bead9a2cec33096e4f593b37611da35aceb727))

* style: Fixes spaces and unused variables ([`7c920b1`](https://github.com/UCSD-E4E/e4e-data-management/commit/7c920b1fc836cb04455f5567fb6436fd3d777f86))

### Unknown

* Merge pull request #92 from UCSD-E4E/37-zip-command

37 zip command ([`24b24d5`](https://github.com/UCSD-E4E/e4e-data-management/commit/24b24d53d3985147f1d71278d80b0c683aeba620))

* Merge branch 'main' into 37-zip-command ([`1b573b5`](https://github.com/UCSD-E4E/e4e-data-management/commit/1b573b59113fc1f2d89d1a3c1f1c990fd4e9b879))

* Merge branch 'main' into 37-zip-command ([`f49afac`](https://github.com/UCSD-E4E/e4e-data-management/commit/f49afac204ce134df63b3def37d9e19a21e0fa9a))

* Merge branch '34-remove-dataset-after-pushing-to-server' into 37-zip-command ([`751363a`](https://github.com/UCSD-E4E/e4e-data-management/commit/751363a888875952b7903b265855a055f2e4a0c3))

* Merge branch 'main' into 37-zip-command ([`6bb26e7`](https://github.com/UCSD-E4E/e4e-data-management/commit/6bb26e7463fb440bb7a130236f0706258b529453))

* wip: Creates zip file ([`705346f`](https://github.com/UCSD-E4E/e4e-data-management/commit/705346f284b00954536212dc57f184aaf0915068))

* Merge pull request #47 from UCSD-E4E/42-implement-e4edm-validate

42 implement e4edm validate ([`e5b4340`](https://github.com/UCSD-E4E/e4e-data-management/commit/e5b43401462353d02a0928cd0c5111ba5ab609ff))

* Merge branch 'main' into 42-implement-e4edm-validate ([`b114488`](https://github.com/UCSD-E4E/e4e-data-management/commit/b1144886ed295299613a75bd4d6c0130533f7e88))

* Merge pull request #83 from UCSD-E4E/60-chore-datamangercli-vs-datamanagercli

chore: Fixes naming ([`95533b5`](https://github.com/UCSD-E4E/e4e-data-management/commit/95533b56a7068b682d63bb3e89d38daa0603d668))

* Merge branch 'main' into 60-chore-datamangercli-vs-datamanagercli ([`1500fca`](https://github.com/UCSD-E4E/e4e-data-management/commit/1500fca5d06f3eeeca290539b11ea0417464c3da))

* Merge branch 'main' into 60-chore-datamangercli-vs-datamanagercli ([`6aa2b72`](https://github.com/UCSD-E4E/e4e-data-management/commit/6aa2b72bdfb17d68635f7fa2f3f6ad522bc377db))

* Merge remote-tracking branch 'origin/main' into 42-implement-e4edm-validate ([`836df5a`](https://github.com/UCSD-E4E/e4e-data-management/commit/836df5adbe135b0f28ed4f5cabed4e25a972618c))

## v0.1.5 (2025-01-11)

### Ci

* ci: Tests for prune behavior ([`31173a1`](https://github.com/UCSD-E4E/e4e-data-management/commit/31173a1ad25897de965548767d7cbbf42fb85f36))

* ci: Fixes failing tests ([`3810a02`](https://github.com/UCSD-E4E/e4e-data-management/commit/3810a029971cf5e146765d57fed36eb9bed5d12a))

### Documentation

* docs: Documents fixture ([`a98cfea`](https://github.com/UCSD-E4E/e4e-data-management/commit/a98cfead653cc847af04ae90ee23371062e91347))

### Fix

* fix: Adds exception logging to main invocation ([`9f0aa7f`](https://github.com/UCSD-E4E/e4e-data-management/commit/9f0aa7f21ccc75bc7e031bc7443ac148ada70d98))

### Style

* style: Fixes long line ([`0d795f7`](https://github.com/UCSD-E4E/e4e-data-management/commit/0d795f73b470b4824cbe9cac06dd52eb879ba1a3))

### Unknown

* Merge pull request #57 from UCSD-E4E/44-add-exception-and-traceback-to-logs

fix: Adds exception logging to main invocation ([`cb9c732`](https://github.com/UCSD-E4E/e4e-data-management/commit/cb9c732cd6e8426513615b36aad287493e21ebbc))

* Merge branch 'main' into 44-add-exception-and-traceback-to-logs ([`de9e0d5`](https://github.com/UCSD-E4E/e4e-data-management/commit/de9e0d5bbacd41a9ef8fcd7cce0e8ba5cd3f942c))

* Merge pull request #91 from UCSD-E4E/34-remove-dataset-after-pushing-to-server

34 remove dataset after pushing to server ([`8a811be`](https://github.com/UCSD-E4E/e4e-data-management/commit/8a811bec27556f1dfba5044f3875cbecdc9b38e4))

* Merge branch 'main' into 42-implement-e4edm-validate ([`e09c335`](https://github.com/UCSD-E4E/e4e-data-management/commit/e09c3353ae6ce15f3161dcd644ba17d13bc7ff03))

## v0.1.4 (2024-11-04)

### Fix
Expand Down Expand Up @@ -108,6 +190,10 @@ Added self.save() call to dataset activation ([`cb5168d`](https://github.com/UCS

* added save ([`f9fddfb`](https://github.com/UCSD-E4E/e4e-data-management/commit/f9fddfb44911be81afab8ed8697f0d5d91cc7c15))

* Update version ([`7e0e9fb`](https://github.com/UCSD-E4E/e4e-data-management/commit/7e0e9fb88a7e097389e0ef9b532e83725c002879))

* Merge remote-tracking branch 'origin/main' into 42-implement-e4edm-validate ([`23a95b9`](https://github.com/UCSD-E4E/e4e-data-management/commit/23a95b96eeb332a6365ead70e4f3451aefe50de4))

* Merge pull request #46 from UCSD-E4E/43-include-full-date-in-dataset

43 include full date in dataset ([`e034866`](https://github.com/UCSD-E4E/e4e-data-management/commit/e03486620bfa968b6d5604ea677a2aa6858e6d3c))
Expand All @@ -122,6 +208,10 @@ Added self.save() call to dataset activation ([`cb5168d`](https://github.com/UCS

* Updated Tests ([`b0663e6`](https://github.com/UCSD-E4E/e4e-data-management/commit/b0663e6a544a85c1ef885424f5e4bb231875b640))

* Added remaining logic ([`3223992`](https://github.com/UCSD-E4E/e4e-data-management/commit/32239925c01f1ad06714528d2f741170911df7b9))

* Adding command changes ([`bc12b34`](https://github.com/UCSD-E4E/e4e-data-management/commit/bc12b34fa0a5803087f6d033e7c5c9d572dcff48))

* Merge pull request #36 from UCSD-E4E/34-remove-dataset-after-pushing-to-server

34 remove dataset after pushing to server ([`aad84c9`](https://github.com/UCSD-E4E/e4e-data-management/commit/aad84c96ebc1129ed9a4300d783b6c1ff021b02a))
Expand Down
2 changes: 1 addition & 1 deletion e4e_data_management/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
'''E4E Data Management Tools
'''
__version__ = '0.1.4'
__version__ = '0.2.0'
157 changes: 90 additions & 67 deletions e4e_data_management/cli.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from e4e_data_management import __version__
from e4e_data_management.core import DataManager
from e4e_data_management.metadata import Metadata

from e4e_data_management.data import Dataset
T = TypeVar('T')
@dataclass
class Parameter:
Expand All @@ -28,71 +28,90 @@ class Parameter:
validator: Callable[[T], bool]


class DataMangerCLI:
class DataManagerCLI:
"""Data Manager Command Line Interface
"""
def __init__(self):
self.__configure_logging()
self._log = logging.getLogger('e4edm.cli')
self._log.debug('Invoking version %s from %s', __version__, __file__)
self.app = DataManager.load()
commands = [
'init_dataset',
'init_mission',
'status',
'list',
'config',
'activate',
'add',
'commit',
'duplicate',
'validate',
'push',
'zip',
'unzip',
'prune',
'ls',
]
self.parameters = [
Parameter(
name='dataset_dir',
getter=lambda: getattr(self.app, 'dataset_dir'),
setter=lambda x: setattr(self.app, 'dataset_dir', x),
parser=Path,
formatter=Path.as_posix,
validator=Path.is_dir
),
Parameter(
name='version',
getter=lambda: getattr(self.app, 'version'),
setter=None,
parser=int,
formatter=str,
validator=None
)
]
self.parser = argparse.ArgumentParser()
subparsers = self.parser.add_subparsers()
parsers = {cmd:subparsers.add_parser(cmd) for cmd in commands}

self.__configure_init_dataset_parser(parsers['init_dataset'])
self.__configure_init_mission_parser(parsers['init_mission'])
self.__configure_status_parser(parsers['status'])
self.__configure_list_parser(parsers['list'])
self.__configure_add_parser(parsers['add'])
self.__configure_commit_parser(parsers['commit'])
self.__configure_duplicate_parser(parsers['duplicate'])
self.__configure_push_parser(parsers['push'])
self.__configure_prune_parser(parsers['prune'])
self.__configure_config_parser(parsers['config'])
self.__configure_activate_parser(parsers['activate'])
self.__configure_ls_parser(parsers['ls'])
# self.__configure_validate_parser(parsers['validate'])
# self.__configure_zip_parser(parsers['zip'])
# self.__configure_unzip_parser(parsers['unzip'])

self.parser.add_argument('--version', action='version', version=f'e4edm {__version__}')
self.parser.set_defaults(func=self.parser.print_help)
try:
self.app = DataManager.load()
commands = [
'init_dataset',
'init_mission',
'status',
'list',
'config',
'activate',
'add',
'commit',
'duplicate',
'validate',
'push',
'zip',
'unzip',
'prune',
'ls',
]
self.parameters = [
Parameter(
name='dataset_dir',
getter=lambda: getattr(self.app, 'dataset_dir'),
setter=lambda x: setattr(self.app, 'dataset_dir', x),
parser=Path,
formatter=Path.as_posix,
validator=Path.is_dir
),
Parameter(
name='version',
getter=lambda: getattr(self.app, 'version'),
setter=None,
parser=int,
formatter=str,
validator=None
)
]
self.parser = argparse.ArgumentParser()
subparsers = self.parser.add_subparsers()
parsers = {cmd:subparsers.add_parser(cmd) for cmd in commands}

self.__configure_init_dataset_parser(parsers['init_dataset'])
self.__configure_init_mission_parser(parsers['init_mission'])
self.__configure_status_parser(parsers['status'])
self.__configure_list_parser(parsers['list'])
self.__configure_add_parser(parsers['add'])
self.__configure_commit_parser(parsers['commit'])
self.__configure_duplicate_parser(parsers['duplicate'])
self.__configure_push_parser(parsers['push'])
self.__configure_prune_parser(parsers['prune'])
self.__configure_config_parser(parsers['config'])
self.__configure_activate_parser(parsers['activate'])
self.__configure_ls_parser(parsers['ls'])
self.__configure_validate_parser(parsers['validate'])
# self.__configure_zip_parser(parsers['zip'])
# self.__configure_unzip_parser(parsers['unzip'])

self.parser.add_argument('--version', action='version', version=f'e4edm {__version__}')
self.parser.set_defaults(func=self.parser.print_help)
except Exception as exc:
self._log.exception('Exception during application load/configuration')
raise exc

def __configure_validate_parser(self, parser: argparse.ArgumentParser):
parser.add_argument('root_dir', nargs='?', default=None, type=Path)
parser.set_defaults(func=self.__external_validate)

def __external_validate(self, root_dir: Optional[Path]):
if root_dir is None:
dataset = self.app.active_dataset
else:
dataset = Dataset.load(path=root_dir)

if not dataset.validate():
print('Dataset validation failed')
else:
print('Dataset valid')

def __configure_logging(self) -> None:
log_dir = Path(DataManager.dirs.user_log_dir).resolve()
Expand Down Expand Up @@ -253,14 +272,18 @@ def prune_cmd(self):
def main(self):
"""Main function
"""
self._log.info("Invoked with %s", ' '.join(sys.argv))
args = self.parser.parse_args()
arg_dict = vars(args)
try:
self._log.info("Invoked with %s", ' '.join(sys.argv))
args = self.parser.parse_args()
arg_dict = vars(args)

arg_fn = args.func
arg_dict.pop('func')
arg_fn = args.func
arg_dict.pop('func')

arg_fn(**arg_dict)
arg_fn(**arg_dict)
except Exception as exc:
self._log.exception('Exception during main execution')
raise exc

def __configure_ls_parser(self, parser: argparse.ArgumentParser):
parser.add_argument('path', type=Path, default=Path('.'))
Expand Down Expand Up @@ -398,7 +421,7 @@ def __configure_init_dataset_parser(self, parser: argparse.ArgumentParser):
def main():
"""Main bootstrap
"""
DataMangerCLI().main()
DataManagerCLI().main()

if __name__ == '__main__':
main()
29 changes: 9 additions & 20 deletions e4e_data_management/core.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,8 @@
from __future__ import annotations

import datetime as dt
import fnmatch
import logging
import pickle
import re
from pathlib import Path
from shutil import copy2, rmtree
from typing import Dict, Iterable, List, Optional, Set
Expand Down Expand Up @@ -284,24 +282,7 @@ def push(self, path: Path) -> None:
Args:
path (Path): Destination to push completed dataset to
"""
if any(len(mission.staged_files) != 0
for mission in self.active_dataset.missions.values()) or \
len(self.active_dataset.staged_files) != 0:
raise RuntimeError('Files still in staging')

# Check that the README is present
readmes = [file
for file in list(self.active_dataset.root.glob('*'))
if re.match(fnmatch.translate('readme.*'), file.name, re.IGNORECASE)]

if len(readmes) == 0:
raise RuntimeError('Readme not found')
acceptable_exts = ['.md', '.docx']
if not any(readme.suffix.lower() in acceptable_exts for readme in readmes):
raise RuntimeError('Illegal README format')

# validate self
self.active_dataset.validate()
self.active_dataset.check_complete()

# Duplicate to destination
destination = path.joinpath(self.active_dataset.name)
Expand All @@ -319,6 +300,14 @@ def zip(self, output_path: Path) -> None:
Args:
output_path (Path): Output path
"""
if output_path.suffix.lower() != '.zip':
output_path = output_path.joinpath(
self.active_dataset.name + '.zip')

output_path.parent.mkdir(parents=True, exist_ok=True)
self.active_dataset.check_complete()

self.active_dataset.create_zip(output_path)

def unzip(self, input_file: Path, output_path: Path) -> None:
"""This will unzip the archived dataset to the specified root
Expand Down
Loading

0 comments on commit 68d1e86

Please sign in to comment.