AzureML - Iris Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook de exemplo Azure Machine Learning\n",
    "\n",
    "Este é um notebook de exemplo de como criar um pipeline no Azure Machine Learning pensando em reprodutibilidade de resultados em treino de modelos de machine learning. Neste exemplo, é apresentado como treinar um modelo para classificar flores usando o Iris Dataset. Ao final, teremos um pipeline automático que permitirá retreino com grande facilidade de troca de dados ou parâmetros. Além disso, todos os hiperparâmetros, dataset, modelo e artefatos são registrados durante o experimento.\n",
    "\n",
    "Como requerimentos para usar esse notebook:\n",
    "- obrigatório: ter conta no Portal do Azure\n",
    "- obrigatório: ter um workspace AzureMl\n",
    "- obrigatório: adicionar o arquivo de config.json do workspace em configuration\n",
    "- obrigatório: ter compute target\n",
    "\n",
    "\n",
    "- opcional: ter datastore criado (pode ser usado o default)\n",
    "- opcional: ter um yaml de um ambiente virtual (conda env export --name azureml > environment.yml) ou arquivo de requirements.txt\n",
    "\n",
    "------------------------------\n",
    "\n",
    "### Passos:\n",
    "- Importando pacotes\n",
    "- Definindo variáveis\n",
    "- Acessando o workspace\n",
    "------------------------------\n",
    "- Criando um datastore\n",
    "- Criando um ambiente\n",
    "- Fazendo o upload de dataset no datastore\n",
    "- Registrando o dataset\n",
    "- Registrando o ambiente\n",
    "------------------------------\n",
    "- Definindo os steps do pipeline\n",
    "- Criando o pipeline\n",
    "- Criando o experimento com o pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importando pacotes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azureml.core import Workspace, Datastore, Experiment, Dataset, ScriptRunConfig\n",
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.environment import Environment\n",
    "from azureml.pipeline.core import Pipeline, PipelineData\n",
    "from azureml.pipeline.steps import PythonScriptStep, EstimatorStep\n",
    "from azureml.pipeline.core.graph import PipelineParameter\n",
    "from azureml.train.estimator import Estimator\n",
    "from azureml.core.runconfig import RunConfiguration\n",
    "from msrest.exceptions import HttpOperationError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definindo variáveis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace_config_path = './azureml_files/configuration/workspace_config.json'\n",
    "environment_requirements_path = './azureml_files/configuration/requirements.txt'\n",
    "environment_name = 'iris_env'\n",
    "datastore_name = 'iris'\n",
    "dataset_path = './dataset/'\n",
    "dataset_file_name = 'iris.csv'\n",
    "dataset_register_name = 'iris_dataset'\n",
    "dataset_description = 'iris dataset'\n",
    "train_file_name = 'train.py'\n",
    "experiment_name = 'iris_training_pipeline'\n",
    "experiment_description = 'Iris training pipeline with decision tree'\n",
    "project_folder = 'azureml_files'\n",
    "model_name = 'iris_decision_tree_model'\n",
    "compute_target_name = 'training-compute'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acessando o workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "azureml_workspace = Workspace.from_config(workspace_config_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando um ambiente"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environment = Environment.from_pip_requirements(name = environment_name, file_path = environment_requirements_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando um datastore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    datastore = Datastore.get(azureml_workspace, datastore_name)\n",
    "except HttpOperationError:\n",
    "    error_message = 'Datastore \"{}\" not found in the \"{}\" workspace. Using default datastore.'\n",
    "    print(error_message.format(datastore_name, azureml_workspace.name))\n",
    "    datastore = azureml_workspace.get_default_datastore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fazendo o upload de dataset no datastore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datastore.upload_files(\n",
    "    files = [os.path.join(dataset_path, dataset_file_name)],\n",
    "    target_path = dataset_register_name,\n",
    "    overwrite = True,\n",
    "    show_progress=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Registrando o dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_path = (datastore, os.path.join(dataset_register_name, dataset_file_name))\n",
    "tabular_dataset = Dataset.Tabular.from_delimited_files(path = dataset_path)\n",
    "\n",
    "tabular_dataset.register(\n",
    "    workspace = azureml_workspace, \n",
    "    name = dataset_register_name,\n",
    "    description = dataset_description,\n",
    "    tags = {'format' : 'CSV'},\n",
    "    create_new_version = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Registrando o ambiente"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "environment.register(workspace = azureml_workspace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acessando o cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_compute_target = ComputeTarget(workspace = azureml_workspace, name = compute_target_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Definindo os steps do pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_size_param = PipelineParameter(name = 'test_size', default_value = 0.1)\n",
    "max_leaf_nodes_param = PipelineParameter(name = 'max_leaf_nodes_param', default_value = 4)\n",
    "dataset_name_param = PipelineParameter(name = 'dataset_name_param', default_value = dataset_register_name)\n",
    "model_folder = PipelineData('model_folder', datastore = datastore, output_name = 'model_folder')\n",
    "dataset_input = tabular_dataset.as_named_input(dataset_register_name)\n",
    "\n",
    "arguments_lst = [\n",
    "    '--model-name', model_name,\n",
    "    '--dataset-name', dataset_name_param,\n",
    "    '--output-folder', model_folder,\n",
    "    '--random-state-test', 0,\n",
    "    '--random-state-model', 0,\n",
    "    '--max-leaf-nodes', max_leaf_nodes_param,\n",
    "    '--test-size', test_size_param\n",
    "]\n",
    "\n",
    "estimator = Estimator(\n",
    "    source_directory = project_folder,\n",
    "    environment_definition = environment,\n",
    "    compute_target = pipeline_compute_target,\n",
    "    entry_script = train_file_name\n",
    ")\n",
    "\n",
    "train_step = EstimatorStep(\n",
    "    name = 'Train model',\n",
    "    estimator = estimator, \n",
    "    estimator_entry_script_arguments = arguments_lst,\n",
    "    compute_target = pipeline_compute_target,\n",
    "    inputs = [dataset_input],\n",
    "    outputs = [model_folder],\n",
    "    allow_reuse = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando o pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline(workspace = azureml_workspace, steps = [train_step])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando o experimento com o pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run = Experiment(azureml_workspace, experiment_name).submit(pipeline)\n",
    "pipeline_run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publicando o pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "published_pipeline = pipeline_run.publish_pipeline(\n",
    "     name = experiment_name,\n",
    "     description = experiment_description,\n",
    "     version = \"1.0\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "published_pipeline"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}