Merge pull request #57 from InfuseAI/feature/sc-21471/datasets

add admin datasets group

docs/CLI/apps.md

@@ -214,4 +214,4 @@ stop:              False
 status:            Ready
 message:           Deployment is ready
 pods:              [{'logEndpoint': 'http://primehub-python-sdk.primehub.io/api/logs/pods/app-code-server-26fcc-765bf579c5-srcft'}]
-```
+```

docs/CLI/deployments.md

@@ -364,4 +364,4 @@ endpoint:            http://primehub-python-sdk.primehub.io/deployment/quickstar
 availableReplicas:   1
 message:             Deployment is being deployed and not available now
 pods:                [{'name': 'deploy-quickstart-iris-cfmnh-68889b97cc-8qgvf'}, {'name': 'deploy-quickstart-iris-cfmnh-68889b97cc-wxzs8'}]
-```
+```

docs/notebook/admin/datasets.ipynb

@@ -0,0 +1,228 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "3365d96c",
+   "metadata": {},
+   "source": [
+    "# [admin] Datasets command\n",
+    "\n",
+    "\n",
+    "The `datasets` command in `admin` scope could help you manage datasets.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0e684e90",
+   "metadata": {},
+   "source": [
+    "## Setup PrimeHub Python SDK\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2ed0bf22",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from primehub import PrimeHub, PrimeHubConfig\n",
+    "ph = PrimeHub(PrimeHubConfig())\n",
+    "\n",
+    "if ph.is_ready():\n",
+    "    print(\"PrimeHub Python SDK setup successfully\")\n",
+    "else:\n",
+    "    print(\"PrimeHub Python SDK couldn't get the group information, follow the 00-getting-started.ipynb to complete it\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3720ad82",
+   "metadata": {},
+   "source": [
+    "## Help documentation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2055801e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "help(ph.admin.datasets)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ac0e2516",
+   "metadata": {},
+   "source": [
+    "## Dataset management\n",
+    "\n",
+    "---\n",
+    "\n",
+    "\n",
+    "```\n",
+    "$ primehub admin datasets\n",
+    "\n",
+    "Usage:\n",
+    "  primehub admin datasets <command>\n",
+    "\n",
+    "Manage datasets\n",
+    "\n",
+    "Available Commands:\n",
+    "  create               Create a dataset\n",
+    "  delete               Delete a dataset by id\n",
+    "  get                  Get a dataset by name\n",
+    "  list                 Delete a dataset by id\n",
+    "  update               Update the dataset\n",
+    "  upload_secret        Regenerate the secret of the upload server\n",
+    "```\n",
+    "\n",
+    "---\n",
+    "\n",
+    "\n",
+    "## Dataset configuration\n",
+    "\n",
+    "You need a configuration `create` and `update` to operate. Here is an example to create a `pv-dataset`:\n",
+    "\n",
+    "```json\n",
+    "{\n",
+    "  \"name\": \"pv-dataset\",\n",
+    "  \"displayName\": \"the dataset created by SDK\",\n",
+    "  \"description\": \"It is a PV dataset\",\n",
+    "  \"type\": \"pv\",\n",
+    "  \"global\": true,\n",
+    "  \"pvProvisioning\": \"auto\",\n",
+    "  \"volumeSize\": 1\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "In our system, there are 5 types for datasets: `['pv', 'nfs', 'hostPath', 'git', 'env']`. Please check the fields reference to give a proper configuration to create your own dataset.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Fields for creating or updating\n",
+    "\n",
+    "| field | required | type | description |\n",
+    "| --- | --- | --- | --- |\n",
+    "| name | required | string | it should be a valid resource name for kubernetes |\n",
+    "| displayName | optional | string | display name for this dataset |\n",
+    "| description | optional | string | |\n",
+    "| global | optional | boolean | when a dataset is global, it could be seen for each group |\n",
+    "| type | required | string | one of ['pv', 'nfs', 'hostPath', 'git', 'env'] |\n",
+    "| url | conditional | string | **MUST** use with `git` type |\n",
+    "| pvProvisioning | conditional | string | onf of ['auto', 'manual'], **MUST** use with `pv` type. This field only uses in `CREATE` action |\n",
+    "| nfsServer | conditional | string | **MUST** use with `nfs` type |\n",
+    "| nfsPath | conditional | string | **MUST** use with `nfs` type |\n",
+    "| hostPath | conditional | string | **MUST** use with `hostPath` type  |\n",
+    "| variables | optional | dict | **MAY** use with `env` type. It is key value pairs. All values have to a string value. For example: `{\"key1\":\"value1\",\"key2\":\"value2\"}`. |\n",
+    "| groups | optional | list of connected groups (dict) | please see the `connect` examples |\n",
+    "| secret | optional | dict | **MAY** use with `git` type. it binds a `secret` to the `git` dataset |\n",
+    "| volumeSize | conditional | integer | **MUST** use with `pv` type. The unit is `GB`.|\n",
+    "| enableUploadServer | optional | boolean | it only works with one of ['pv', 'nfs', 'hostPath'] writable types |\n",
+    "\n",
+    "> There is a simple rule to use fields for `UPDATE`. All required fields should not be in the payload.\n",
+    "\n",
+    "For example, there is a configuration for creating env dataset:\n",
+    "\n",
+    "```bash\n",
+    "primehub admin datasets create <<EOF\n",
+    "{\n",
+    "  \"name\": \"env-dataset\",\n",
+    "  \"description\": \"\",\n",
+    "  \"type\": \"env\",\n",
+    "  \"variables\": {\n",
+    "    \"ENV\": \"prod\",\n",
+    "    \"LUCKY_NUMBER\": \"7\"\n",
+    "  }\n",
+    "}\n",
+    "EOF\n",
+    "```\n",
+    "\n",
+    "After removing required `name` and `type` fields, it could be used with updating:\n",
+    "\n",
+    "```bash\n",
+    "primehub admin datasets update env-dataset <<EOF\n",
+    "{\n",
+    "  \"description\": \"make changes to the description\",\n",
+    "  \"variables\": {\n",
+    "    \"ENV\": \"prod\",\n",
+    "    \"LUCKY_NUMBER\": \"8\"\n",
+    "  }\n",
+    "}\n",
+    "EOF\n",
+    "```\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dcb77e3d",
+   "metadata": {},
+   "source": [
+    "## Examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b4eb5746",
+   "metadata": {},
+   "source": [
+    "You could find [more examples on our github](https://github.com/InfuseAI/primehub-python-sdk/blob/main/docs/CLI/admin/datasets.md)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "639e6281",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# List datasets\n",
+    "list(ph.admin.datasets.list())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "908759a6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get a dataset\n",
+    "ph.admin.datasets.get('primehub')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ef3c1a91",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "PrimeHub SDK",
+   "language": "python",
+   "name": "myenv"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/notebook/groups.ipynb

@@ -98,4 +98,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

primehub/__init__.py

@@ -192,7 +192,9 @@ class PrimeHub(object):
     def __init__(self, config: PrimeHubConfig):
         self.primehub_config = config
         self.json_output = True
+        self.usage_role = 'user'
         self.commands: Dict[str, Module] = dict()
+        self.admin_commands: Dict[str, Module] = dict()
         self._stderr = sys.stderr
         self._stdout = sys.stdout
 
@@ -212,6 +214,9 @@ def __init__(self, config: PrimeHubConfig):
         self.register_command('apptemplates', 'AppTemplate')
         self.register_command('apps', 'Apps')
 
+        # register admin commands
+        self.register_admin_command('admin_datasets', 'AdminDatasets', 'datasets')
+
         # initial
         self._ensure_config_details(config)
 
@@ -236,19 +241,42 @@ def request_logs(self, endpint: str, follow: bool, tail: int):
     def request_file(self, endpint: str, dest: str):
         return Client(self.primehub_config).request_file(endpint, dest)
 
-    def register_command(self, module_name: str, command_class: Union[str, Callable], command_name=None):
-        if not command_name:
-            command_name = module_name
-
+    def _find_command_class(self, command_class, module_name):
         # create command instance
         if isinstance(command_class, str):
             clazz = importlib.import_module('primehub.' + module_name).__getattribute__(command_class)
         else:
             clazz = command_class
+        return clazz
+
+    def register_command(self, module_name: str, command_class: Union[str, Callable], command_name=None):
+        if not command_name:
+            command_name = module_name
+
+        clazz = self._find_command_class(command_class, module_name)
 
         # register to the commands table
         self.commands[command_name] = clazz(self)
 
+    def register_admin_command(self, module_name: str, command_class: Union[str, Callable], command_name=None):
+        if not command_name:
+            command_name = module_name
+
+        clazz = self._find_command_class(command_class, module_name)
+
+        # register to the commands table
+        self.admin_commands[command_name] = clazz(self)
+
+    def switch_admin_role(self):
+        self.usage_role = 'admin'
+        self.commands = self.admin_commands
+
+    @property
+    def admin(self):
+        admin_primehub = PrimeHub(self.primehub_config)
+        admin_primehub.commands = self.admin_commands
+        return admin_primehub
+
     def __getattr__(self, item):
         if item in self.commands:
             return self.commands[item]