WebScrappingML.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0e6f4d45c5424fa29faf88ce4b633919": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fe80c0fdb8154e9f8cd0e6c04542130f",
              "IPY_MODEL_f0627a81cc8746098951c75e2a99c6ac",
              "IPY_MODEL_b936a57982d7475b909a8d5bfb47c132"
            ],
            "layout": "IPY_MODEL_482c72ff6d2d443bbd3f07f33aa53e73"
          }
        },
        "fe80c0fdb8154e9f8cd0e6c04542130f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e4ad9e8777648f1bf6d54b5b53bc5ff",
            "placeholder": "​",
            "style": "IPY_MODEL_236eb4c47198471787432449cffdeebc",
            "value": "Downloading: 100%"
          }
        },
        "f0627a81cc8746098951c75e2a99c6ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e71860f04c9f413091cc9dffd27eca9a",
            "max": 571,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3434d43a13544415abb99adf21c55521",
            "value": 571
          }
        },
        "b936a57982d7475b909a8d5bfb47c132": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e297b5139f944a0cbf90372cbf987e37",
            "placeholder": "​",
            "style": "IPY_MODEL_53008621bd684e0e868c1495e2cee027",
            "value": " 571/571 [00:00&lt;00:00, 7.29kB/s]"
          }
        },
        "482c72ff6d2d443bbd3f07f33aa53e73": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3e4ad9e8777648f1bf6d54b5b53bc5ff": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "236eb4c47198471787432449cffdeebc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e71860f04c9f413091cc9dffd27eca9a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3434d43a13544415abb99adf21c55521": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e297b5139f944a0cbf90372cbf987e37": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "53008621bd684e0e868c1495e2cee027": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "59b0a5d4a36d40079bf9709c3cf3294a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8f0f2a2260364b5c859869dd01fc7d11",
              "IPY_MODEL_ec502d4c2e974e2d98fd0cf6acb7135f",
              "IPY_MODEL_75f5d675c3654a33ad2f2dcbe5141022"
            ],
            "layout": "IPY_MODEL_2c67ca59297f481197bc7439102b8975"
          }
        },
        "8f0f2a2260364b5c859869dd01fc7d11": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e432ad0920304c7c9ce47e47841cdbc4",
            "placeholder": "​",
            "style": "IPY_MODEL_70d860cdf00c4fbd8ce0570dcc435d75",
            "value": "Downloading: 100%"
          }
        },
        "ec502d4c2e974e2d98fd0cf6acb7135f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d099563b01f64ca7bcb9ada9a3bfe919",
            "max": 1344997306,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_49ff316f5632459ea6cca519c70aa5f7",
            "value": 1344997306
          }
        },
        "75f5d675c3654a33ad2f2dcbe5141022": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_024dd09abd4d4c65b10e75c37b214241",
            "placeholder": "​",
            "style": "IPY_MODEL_eb9003e88279404cbc94fc22189758a0",
            "value": " 1.34G/1.34G [01:11&lt;00:00, 21.7MB/s]"
          }
        },
        "2c67ca59297f481197bc7439102b8975": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e432ad0920304c7c9ce47e47841cdbc4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "70d860cdf00c4fbd8ce0570dcc435d75": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d099563b01f64ca7bcb9ada9a3bfe919": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "49ff316f5632459ea6cca519c70aa5f7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "024dd09abd4d4c65b10e75c37b214241": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eb9003e88279404cbc94fc22189758a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ceb972b115e3409897d3cb07dc239767": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bd560499fdea491b8621eb302295619c",
              "IPY_MODEL_d3b7084a4caa4997bbe202d51c468df2",
              "IPY_MODEL_fccde74cb2444f16b9564e2936e1b3da"
            ],
            "layout": "IPY_MODEL_1f14e14eb41c4e409b671e2d10bd51bf"
          }
        },
        "bd560499fdea491b8621eb302295619c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7f3a8cd7256d414598338dbf9b277604",
            "placeholder": "​",
            "style": "IPY_MODEL_82c06d5d33fd408cbb373682b6edb799",
            "value": "Downloading: 100%"
          }
        },
        "d3b7084a4caa4997bbe202d51c468df2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e1c257a6f09047acb83bdcdb7f499be4",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_14819e1dc63846ea92f77b75a712eeb5",
            "value": 231508
          }
        },
        "fccde74cb2444f16b9564e2936e1b3da": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6aab4f29557447b7bb52ef19daa74124",
            "placeholder": "​",
            "style": "IPY_MODEL_0c0342ca388e45a8bad5a97eeedd35d0",
            "value": " 232k/232k [00:00&lt;00:00, 308kB/s]"
          }
        },
        "1f14e14eb41c4e409b671e2d10bd51bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7f3a8cd7256d414598338dbf9b277604": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82c06d5d33fd408cbb373682b6edb799": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e1c257a6f09047acb83bdcdb7f499be4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "14819e1dc63846ea92f77b75a712eeb5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6aab4f29557447b7bb52ef19daa74124": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0c0342ca388e45a8bad5a97eeedd35d0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "df31d0890aa748c48237b374e5eecbcf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9de98b99c91046a09bfe006932aa9eae",
              "IPY_MODEL_4f8b0c276e1e41e2be2f044b303a7ad0",
              "IPY_MODEL_c94d065654b548568d64ec5d27b9077d"
            ],
            "layout": "IPY_MODEL_1d1d985c647b421c855bd555d4edef52"
          }
        },
        "9de98b99c91046a09bfe006932aa9eae": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3cbd7cea7ac54665847ecf3bed64af56",
            "placeholder": "​",
            "style": "IPY_MODEL_bd81acf5684f48c89cddfa40cd178930",
            "value": "Downloading: 100%"
          }
        },
        "4f8b0c276e1e41e2be2f044b303a7ad0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f4b3d126f1114a45b663a9c8aa0fdfc7",
            "max": 28,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_48699a6d213b402a9c770369f3ffcc8f",
            "value": 28
          }
        },
        "c94d065654b548568d64ec5d27b9077d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e846a309fd0436baa14dcd87b1c5efd",
            "placeholder": "​",
            "style": "IPY_MODEL_07ac5e1727384642a102cc22438adcfb",
            "value": " 28.0/28.0 [00:00&lt;00:00, 1.24kB/s]"
          }
        },
        "1d1d985c647b421c855bd555d4edef52": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3cbd7cea7ac54665847ecf3bed64af56": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd81acf5684f48c89cddfa40cd178930": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f4b3d126f1114a45b663a9c8aa0fdfc7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "48699a6d213b402a9c770369f3ffcc8f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3e846a309fd0436baa14dcd87b1c5efd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "07ac5e1727384642a102cc22438adcfb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Al3xGROS/WebScappingML/blob/main/WebScrappingML.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Web Scrapping & Article Summarization\n",
        "\n",
        "## Introduction\n",
        "\n",
        "The goal of this project is to be able to scrappe any Wikipedia page (in English) the user wants and summarize it.\n",
        "First we are going to talk a little bit about the scrapping part of the project and then we will see the how to summarize a text in python.\n",
        "\n",
        "\n",
        "## Web Scrapping\n",
        "\n",
        "To scrappe a page in python, there are a lot of librairies and differents methods but we chose the librairies ```Requests``` and ```BeautifulSoup```.\n",
        "The first one is used to get the html content of any web page from the url of this same page. And the second one is to naviguate the html page more easily.\n",
        "\n",
        "First we need to import the librairies :"
      ],
      "metadata": {
        "id": "2k30YfA1LWOf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "iqJJEDu0ysHB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# LIBRAIRIES\n",
        "from bs4.element import PageElement\n",
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import re"
      ],
      "metadata": {
        "id": "RdLSbUBIQMHU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And then we have the function that scrappe the wikipedia page and get all the paragraphe of the page in one single string.\n",
        "We get all the text from the ```<p>``` nodes. And then create a regex to keep only the text after the beginning of the article."
      ],
      "metadata": {
        "id": "JTdcHqRuQX5x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "URL = \"https://en.wikipedia.org/wiki/Margaret_Hamilton_(software_engineer)\"\n",
        "\n",
        "# SCRAPPING\n",
        "def scrape(url):\n",
        "  page = requests.get(url)\n",
        "\n",
        "  soup = BeautifulSoup(page.content, 'html.parser')\n",
        "\n",
        "  title = soup.find(\"b\")\n",
        "  realTitle = title.get_text()\n",
        "\n",
        "  data = '' \n",
        "  text = \"\"\n",
        "  for data in soup.find_all(\"p\"):\n",
        "    text += data.get_text()\n",
        "\n",
        "  result = text[text.find(realTitle):]\n",
        "  patn = re.sub(r\"\\[[^\\]]*\\]\", \"\", result)\n",
        "  return patn\n",
        "\n",
        "scrape(URL)"
      ],
      "metadata": {
        "id": "RdQ80AIGj9Z5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "ae66875c-5cb1-413e-8a59-263242a9995c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner. She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA\\'s Apollo program. She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.\\nHamilton has published more than 130 papers, proceedings, and reports, about sixty projects, and six major programs. She is one of the people credited with coining the term \"software engineering\".\\nOn November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA\\'s Apollo Moon missions.\\nMargaret Elaine Heafield was born August 17, 1936, in Paoli, Indiana, to Kenneth Heafield and Ruth Esther Heafield (née Partington). The family later moved to Michigan, where Margaret graduated from Hancock High School in 1954. She studied mathematics at the University of Michigan in 1955 before transferring to Earlham College, where her mother was a student; she earned a BA in mathematics with a minor in philosophy in 1958. She cites Florence Long, the head of the math department at Earlham, as helping with her desire to pursue abstract mathematics and become a mathematics professor. She says her poet father and headmaster grandfather inspired her to include a minor in philosophy in her studies.\\nIn Boston, Hamilton initially intended to enroll in graduate study in abstract mathematics at Brandeis. However, in mid-1959, Hamilton began working for Edward Norton Lorenz, in the meteorology department at MIT. She developed software for predicting weather, programming on the LGP-30 and the PDP-1 computers at Marvin Minsky\\'s Project MAC. Her work contributed to Lorenz\\'s publications on chaos theory. At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience. She moved on to another project in the summer of 1961, and hired and trained Ellen Fetter as her replacement.\\nFrom 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft. She also wrote software for a satellite tracking project at the Air Force Cambridge Research Laboratories. The SAGE Project was an extension of Project Whirlwind, started by MIT to create a computer system that could predict weather systems and track their movements using simulators. SAGE was soon developed for military use in anti-aircraft air defense. Hamilton said:\\nWhat they used to do when you came into this organization as a beginner, was to assign you this program which nobody was able to ever figure out or get to run. When I was the beginner they gave it to me as well. And what had happened was it was tricky programming, and the person who wrote it took delight in the fact that all of his comments were in Greek and Latin. So I was assigned this program and I actually got it to work. It even printed out its answers in Latin and Greek. I was the first one to get it to work.It was her efforts on this project that made her a candidate for the position at NASA as the lead developer for Apollo flight software.\\nHamilton then joined the Charles Stark Draper Laboratory at MIT, which worked on the Apollo Space Mission. Hamilton was initially hired as a programmer for this process but moved on into systems design. By 1965, she was in charge of all Command Module software, which was all the software for navigation and lunar landing guidance. Hamilton\\'s team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab. Another part of her team designed and developed the systems software. This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed. She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist. Hamilton also served as Director of the Software Engineering Division.\\nHer areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques. These made her code incredibly reliable because they helped programmers identify and fix anomalies before they became major problems.\\nIn one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon. Three minutes before the lunar lander reached the Moon\\'s surface, several computer alarms were triggered. The on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.\\nHamilton had prepared for just this situation years before:\\nThere was one other failsafe that Hamilton likes to remember. Her \"priority display\" innovation had created a knock-on risk that astronaut and computer would slip out of synch just when it mattered most. As the alarms went off and priority displays replaced normal ones, the actual switchover to new programmes behind the screens was happening \"a step slower\" than it would today.\\nHamilton had thought long and hard about this. It meant that if Aldrin, say, hit a button on the priority display too quickly, he might still get a \"normal\" response. Her solution: when you see a priority display, first count to\\nfive.\\nBy some accounts, the astronauts had inadvertently left the rendezvous radar switch on, causing these alarms to be triggered (the claim that the radar was left on inadvertently by the astronauts is disputed by Robert Wills with the National Museum of Computing). The computer was overloaded with interrupts caused by incorrectly phased power supplied to the lander\\'s rendezvous radar. The program alarms indicated \"executive overflows\", meaning the guidance computer could not complete all of its tasks in real time and had to postpone some of them. The asynchronous executive designed by J. Halcombe Laning was used by Hamilton\\'s team to develop asynchronous flight software:\\nBecause of the flight software\\'s system-software\\'s error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts\\' normal mission displays with priority displays of critical alarms in case of an emergency. This depended on our assigning a unique priority to every process in the software in order to ensure that all of its events would take place in the correct order and at the right time relative to everything else that was going on.Hamilton\\'s priority alarm displays interrupted the astronauts\\' normal displays to warn them that there was an emergency \"giving the astronauts a go/no go decision (to land or not to land)\". Jack Garman, a NASA computer engineer in mission control, recognized the meaning of the errors that were presented to the astronauts by the priority displays and shouted, \"Go, go!\" and they continued. Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton\\'s work \"the foundation for ultra-reliable software design\".\\nHamilton later wrote of the incident:\\nThe computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing. It then sent out an alarm, which meant to the astronaut, \\'I\\'m overloaded with more tasks than I should be doing at this time and I\\'m going to keep only the more important tasks\\'; i.e., the ones needed for landing\\xa0... Actually, the computer was programmed to do more than recognize error conditions. A complete set of recovery programs was incorporated into the software. The software\\'s action, in this case, was to eliminate lower priority tasks and re-establish the more important ones\\xa0... If the computer hadn\\'t recognized this problem and taken recovery action, I doubt if Apollo 11 would have been the successful moon landing it was.\\nIn 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program. They created a product called USE.IT, based on the HOS methodology they developed at MIT. It was successfully used in numerous government programs including a project to formalize and implement C-IDEF, an automated version of IDEF, a modeling language developed by the U.S. Air Force in the Integrated Computer-Aided Manufacturing (ICAM) project. In 1980, British-Israeli computer scientist David Harel published a proposal for a structured programming language derived from HOS from the viewpoint of and/or subgoals. Others have used HOS to formalize the semantics of linguistic quantifiers, and to formalize the design of reliable real-time embedded systems.\\nHamilton was the CEO of HOS through 1984 and left the company in 1985. In March 1986, she founded Hamilton Technologies, Inc. in Cambridge, Massachusetts. The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.\\nAnthony Oettinger, Barry Boehm, and Hamilton have been credited with naming the discipline of \"software engineering\". Hamilton details how she came to make up the term \"software engineering\":\\nWhen I first came up with the term, no one had heard of it before, at least in our world. It was an ongoing joke for a long time. They liked to kid me about my radical ideas. It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware. Not because of his acceptance of the new \\'term\\' per se, but because we had earned his and the acceptance of the others in the room as being in an engineering field in its own right.When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science. Hamilton was concerned with legitimizing software development as an engineering discipline. Over time the term \"software engineering\" gained the same respect as any other technical discipline. The IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering. Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system. Writing in Wired, Robert McMillan noted: \"At MIT she assisted in the creation of the core principles in computer programming as she worked with her colleagues in writing code for the world\\'s first portable computer\". Hamilton\\'s innovations go beyond the feats of playing an important role in getting humans to the Moon. According to Wired\\'s Karen Tegan Padir: \"She, along with that other early programming pioneer, COBOL inventor Grace Hopper, also deserve tremendous credit for helping to open the door for more women to enter and succeed in STEM fields like software.\"\\nIn 2019, to celebrate 50 years to the Apollo landing, Google decided to make a tribute to Hamilton. The mirrors at the Ivanpah plant were configured to create a picture of Hamilton and the Apollo 11 by moonlight.\\nHamilton has a sister Kathryn.\\nShe met her first husband, James Cox Hamilton, in the mid-1950s while attending college. They were married on June 15, 1958, the summer after she graduated from Earlham. She briefly taught high school mathematics and French at a public school in Boston, Indiana. The couple then moved to Boston, Massachusetts, where they had a daughter, Lauren, born on November 10, 1959. They divorced in 1967 and Margaret married Dan Lickly two years later.\\n'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "But if you ever want to scrappe a wikipedia page there exist a librairy called ```Wikipedia``` that is perfect for it.\n",
        "\n",
        "Let see how it works.\n",
        "\n",
        "We install the librairy."
      ],
      "metadata": {
        "id": "VfBwuh6eRZNY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install wikipedia"
      ],
      "metadata": {
        "id": "fGvoyu5oR1tQ",
        "outputId": "08e9efb9-f168-496c-a2d2-073f40890432",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting wikipedia\n",
            "  Downloading wikipedia-1.4.0.tar.gz (27 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.8/dist-packages (from wikipedia) (4.6.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from wikipedia) (2.25.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.24.3)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (4.0.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2.10)\n",
            "Building wheels for collected packages: wikipedia\n",
            "  Building wheel for wikipedia (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wikipedia: filename=wikipedia-1.4.0-py3-none-any.whl size=11695 sha256=e36f70eea026ae41dda9c86d4de317e79cb285c7a455ba17fc7aad376a8792d7\n",
            "  Stored in directory: /root/.cache/pip/wheels/07/93/05/72c05349177dca2e0ba31a33ba4f7907606f7ddef303517c6a\n",
            "Successfully built wikipedia\n",
            "Installing collected packages: wikipedia\n",
            "Successfully installed wikipedia-1.4.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We import it.\n",
        "And we can get the content of the page we want by using ```wikipedia.page``` and then ```variable.content``` (variable with the content of wikipedia.page)."
      ],
      "metadata": {
        "id": "PqYS_ut9YNPf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import wikipedia\n",
        "wikisearch = wikipedia.page(\"Margaret_Hamilton_(software_engineer)\")\n",
        "wikicontent = wikisearch.content\n",
        "\n",
        "wikicontent"
      ],
      "metadata": {
        "id": "DOay9GSVYQon",
        "outputId": "d7209334-7a8f-44c5-b0cb-127ea8bff11f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner. She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA\\'s Apollo program. She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.\\nHamilton has published more than 130 papers, proceedings, and reports, about sixty projects, and six major programs. She is one of the people credited with coining the term \"software engineering\".On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA\\'s Apollo Moon missions.\\n\\n\\n== Early life and education ==\\nMargaret Elaine Heafield was born August 17, 1936, in Paoli, Indiana, to Kenneth Heafield and Ruth Esther Heafield (née Partington). The family later moved to Michigan, where Margaret graduated from Hancock High School in 1954. She studied mathematics at the University of Michigan in 1955 before transferring to Earlham College, where her mother was a student; she earned a BA in mathematics with a minor in philosophy in 1958. She cites Florence Long, the head of the math department at Earlham, as helping with her desire to pursue abstract mathematics and become a mathematics professor. She says her poet father and headmaster grandfather inspired her to include a minor in philosophy in her studies.\\n\\n\\n== Career ==\\nIn Boston, Hamilton initially intended to enroll in graduate study in abstract mathematics at Brandeis. However, in mid-1959, Hamilton began working for Edward Norton Lorenz, in the meteorology department at MIT. She developed software for predicting weather, programming on the LGP-30 and the PDP-1 computers at Marvin Minsky\\'s Project MAC. Her work contributed to Lorenz\\'s publications on chaos theory. At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience. She moved on to another project in the summer of 1961, and hired and trained Ellen Fetter as her replacement.\\n\\n\\n=== SAGE Project ===\\nFrom 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft. She also wrote software for a satellite tracking project at the Air Force Cambridge Research Laboratories. The SAGE Project was an extension of Project Whirlwind, started by MIT to create a computer system that could predict weather systems and track their movements using simulators. SAGE was soon developed for military use in anti-aircraft air defense. Hamilton said:\\n\\nWhat they used to do when you came into this organization as a beginner, was to assign you this program which nobody was able to ever figure out or get to run. When I was the beginner they gave it to me as well. And what had happened was it was tricky programming, and the person who wrote it took delight in the fact that all of his comments were in Greek and Latin. So I was assigned this program and I actually got it to work. It even printed out its answers in Latin and Greek. I was the first one to get it to work.\\nIt was her efforts on this project that made her a candidate for the position at NASA as the lead developer for Apollo flight software.\\n\\n\\n=== Draper Laboratory ===\\n\\nHamilton then joined the Charles Stark Draper Laboratory at MIT, which worked on the Apollo Space Mission. Hamilton was initially hired as a programmer for this process but moved on into systems design. By 1965, she was in charge of all Command Module software, which was all the software for navigation and lunar landing guidance. Hamilton\\'s team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab. Another part of her team designed and developed the systems software. This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed. She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist. Hamilton also served as Director of the Software Engineering Division.Her areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques. These made her code incredibly reliable because they helped programmers identify and fix anomalies before they became major problems.\\n\\n\\n=== Apollo program ===\\nIn one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon. Three minutes before the lunar lander reached the Moon\\'s surface, several computer alarms were triggered. The on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.\\nHamilton had prepared for just this situation years before:\\n\\nThere was one other failsafe that Hamilton likes to remember. Her \"priority display\" innovation had created a knock-on risk that astronaut and computer would slip out of synch just when it mattered most. As the alarms went off and priority displays replaced normal ones, the actual switchover to new programmes behind the screens was happening \"a step slower\" than it would today.\\nHamilton had thought long and hard about this. It meant that if Aldrin, say, hit a button on the priority display too quickly, he might still get a \"normal\" response. Her solution: when you see a priority display, first count to\\nfive.\\n\\nBy some accounts, the astronauts had inadvertently left the rendezvous radar switch on, causing these alarms to be triggered (the claim that the radar was left on inadvertently by the astronauts is disputed by Robert Wills with the National Museum of Computing). The computer was overloaded with interrupts caused by incorrectly phased power supplied to the lander\\'s rendezvous radar. The program alarms indicated \"executive overflows\", meaning the guidance computer could not complete all of its tasks in real time and had to postpone some of them. The asynchronous executive designed by J. Halcombe Laning was used by Hamilton\\'s team to develop asynchronous flight software:\\n\\nBecause of the flight software\\'s system-software\\'s error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts\\' normal mission displays with priority displays of critical alarms in case of an emergency. This depended on our assigning a unique priority to every process in the software in order to ensure that all of its events would take place in the correct order and at the right time relative to everything else that was going on.\\nHamilton\\'s priority alarm displays interrupted the astronauts\\' normal displays to warn them that there was an emergency \"giving the astronauts a go/no go decision (to land or not to land)\". Jack Garman, a NASA computer engineer in mission control, recognized the meaning of the errors that were presented to the astronauts by the priority displays and shouted, \"Go, go!\" and they continued. Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton\\'s work \"the foundation for ultra-reliable software design\".Hamilton later wrote of the incident:\\n\\nThe computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing. It then sent out an alarm, which meant to the astronaut, \\'I\\'m overloaded with more tasks than I should be doing at this time and I\\'m going to keep only the more important tasks\\'; i.e., the ones needed for landing ... Actually, the computer was programmed to do more than recognize error conditions. A complete set of recovery programs was incorporated into the software. The software\\'s action, in this case, was to eliminate lower priority tasks and re-establish the more important ones ... If the computer hadn\\'t recognized this problem and taken recovery action, I doubt if Apollo 11 would have been the successful moon landing it was.\\n\\n\\n=== Businesses ===\\nIn 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program. They created a product called USE.IT, based on the HOS methodology they developed at MIT. It was successfully used in numerous government programs including a project to formalize and implement C-IDEF, an automated version of IDEF, a modeling language developed by the U.S. Air Force in the Integrated Computer-Aided Manufacturing (ICAM) project. In 1980, British-Israeli computer scientist David Harel published a proposal for a structured programming language derived from HOS from the viewpoint of and/or subgoals. Others have used HOS to formalize the semantics of linguistic quantifiers, and to formalize the design of reliable real-time embedded systems.Hamilton was the CEO of HOS through 1984 and left the company in 1985. In March 1986, she founded Hamilton Technologies, Inc. in Cambridge, Massachusetts. The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.\\n\\n\\n== Legacy ==\\n\\nAnthony Oettinger, Barry Boehm, and Hamilton have been credited with naming the discipline of \"software engineering\". Hamilton details how she came to make up the term \"software engineering\":\\n\\nWhen I first came up with the term, no one had heard of it before, at least in our world. It was an ongoing joke for a long time. They liked to kid me about my radical ideas. It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware. Not because of his acceptance of the new \\'term\\' per se, but because we had earned his and the acceptance of the others in the room as being in an engineering field in its own right.\\nWhen Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science. Hamilton was concerned with legitimizing software development as an engineering discipline. Over time the term \"software engineering\" gained the same respect as any other technical discipline. The IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering. Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system. Writing in Wired, Robert McMillan noted: \"At MIT she assisted in the creation of the core principles in computer programming as she worked with her colleagues in writing code for the world\\'s first portable computer\". Hamilton\\'s innovations go beyond the feats of playing an important role in getting humans to the Moon. According to Wired\\'s Karen Tegan Padir: \"She, along with that other early programming pioneer, COBOL inventor Grace Hopper, also deserve tremendous credit for helping to open the door for more women to enter and succeed in STEM fields like software.\"In 2019, to celebrate 50 years to the Apollo landing, Google decided to make a tribute to Hamilton. The mirrors at the Ivanpah plant were configured to create a picture of Hamilton and the Apollo 11 by moonlight.\\n\\n\\n== Awards ==\\n\\nIn 1986, Hamilton received the Augusta Ada Lovelace Award by the Association for Women in Computing.\\nIn 2003, she was given the NASA Exceptional Space Act Award for scientific and technical contributions. The award included $37,200, the largest amount awarded to any individual in NASA\\'s history.\\nIn 2009, she received the Outstanding Alumni Award by Earlham College.\\nIn 2016, she received the Presidential Medal of Freedom from Barack Obama, the highest civilian honor in the United States.\\nOn April 28, 2017, she received the Computer History Museum Fellow Award, which honors exceptional men and women whose computing ideas have changed the world.\\nIn 2017, a \"Women of NASA\" LEGO set went on sale featuring minifigures of Hamilton, Mae Jemison, Sally Ride, and Nancy Grace Roman.\\nIn 2018, she was awarded an honorary doctorate degree by the Polytechnic University of Catalonia.\\nIn 2019, she was awarded The Washington Award.\\nIn 2019, she was awarded an honorary doctorate degree by Bard College.\\nIn 2019, she was awarded the Intrepid Lifetime Achievement Award.\\n\\n\\n== Publications ==\\nHamilton, M.; Zeldin, S. (March 1976). \"Higher Order Software—A Methodology for Defining Software\". IEEE Transactions on Software Engineering. SE-2 (1): 9–32. doi:10.1109/TSE.1976.233798. S2CID 7799553.\\nHamilton, M.; Zeldin, S. (January 1, 1979). \"The relationship between design and verification\". Journal of Systems and Software. 1: 29–56. doi:10.1016/0164-1212(79)90004-9.\\nHamilton, M. (April 1994). \"Inside Development Before the Fact\". (Cover story). Special Editorial Supplement. 8ES-24ES. Electronic Design.\\nHamilton, M. (June 1994). \"001: A Full Life Cycle Systems Engineering and Software Development Environment\". (Cover story). Special Editorial Supplement. 22ES-30ES. Electronic Design.\\nHamilton, M.; Hackler, W. R. (2004). \"Deeply Integrated Guidance Navigation Unit (DI-GNU) Common Software Architecture Principles\". (Revised December 29, 2004). DAAAE30-02-D-1020 and DAAB07-98-D-H502/0180, Picatinny Arsenal, NJ, 2003–2004.\\nHamilton, M.; Hackler, W. R. (2007). \"Universal Systems Language for Preventative Systems Engineering\", Proc. 5th Ann. Conf. Systems Eng. Res. (CSER), Stevens Institute of Technology, Mar. 2007, paper #36.\\nHamilton, M.; Hackler, W. R. (2007). \"A Formal Universal Systems Semantics for SysML\". 17th Annual International Symposium, INCOSE 2007, San Diego, CA, Jun. 2007.\\nHamilton, M.; Hackler, W. R. (2008). \"Universal Systems Language: Lessons Learned from Apollo\". IEEE Computer, Dec. 2008.\\nHamilton, M. H. (September 2018). \"What the Errors Tell Us\". IEEE Software. 35 (5): 32–37. doi:10.1109/MS.2018.290110447. S2CID 52896962.\\n\\n\\n== Personal life ==\\nHamilton has a sister Kathryn.She met her first husband, James Cox Hamilton, in the mid-1950s while attending college. They were married on June 15, 1958, the summer after she graduated from Earlham. She briefly taught high school mathematics and French at a public school in Boston, Indiana. The couple then moved to Boston, Massachusetts, where they had a daughter, Lauren, born on November 10, 1959. They divorced in 1967 and Margaret married Dan Lickly two years later.\\n\\n\\n== See also ==\\nList of pioneers in computer science\\n\\n\\n== References ==\\n\\n\\n== Further reading ==\\nSteafel, Eleanor (July 20, 2019). \"One woman in a room full of men\". The Telegraph Magazine. London: Daily Telegraph plc. pp. 56–59, 61. OCLC 69022829.\\n\\n\\n== External links ==\\n Media related to Margaret Hamilton (scientist) at Wikimedia Commons\\nHamilton Technologies, Inc.\\nMIT News\\nMargaret Hamilton Archived September 5, 2017, at the Wayback Machine Video produced by Makers: Women Who Make America\\nMargaret Hamilton ’58 – Presidential Medal of Freedom Recipient Archived July 30, 2019, at the Wayback Machine: Earlham College profile'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Context\n",
        "\n",
        "It is very difficult for machines to summarize text. As humans we usually read the whole text to keep the important points. Computers not having such developed language skills makes summarizing text difficult and non-trivial for the machine.\n",
        "\n",
        "There are many Machine Learning models developed for this specific task. Most of them are based on a classification system to determine which part of the analyzed document is relevant or not. \n",
        "There are two main methods for automatic text summarization: extraction and abstraction.\n",
        "\n",
        "The extractive method retrieves sentences directly from the analyzed document based on a score principle. In this approach, important parts of the document are identified, then cut out and reassembled to form a summary. The sentences are classified by order of importance (Top K algorithm) and by link with the subject.\n",
        "\n",
        "This is the most common summarization method and the one most used so far by automatic summarization models.\n",
        "The abstractive method generates grammatically correct sentences from the input text. This model rewrites large amounts of text which is further processed again and summarized using NLP. This method is the precursor of the AI writing tools that appeared recently.\n"
      ],
      "metadata": {
        "id": "wCabtGVSr0RN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Spacy\n",
        "\n",
        "The first library we are going to use is called ```Spacy```.\n",
        "```Spacy``` is a free open-source library for Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more.\n",
        "We use this library because it is one of the most basic in the field of word processing. It does not rewrite a text but just compile part of a text you choose.\n",
        "We are going to use it to do a summary of the text from Wikipedia by evaluting words from their occurrences and only keeping the top 30% of the sentences of the text.\n"
      ],
      "metadata": {
        "id": "N-_KSFagqMnE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# imports\n",
        "import spacy\n",
        "from spacy.lang.en.stop_words import STOP_WORDS\n",
        "from string import punctuation\n",
        "from heapq import nlargest"
      ],
      "metadata": {
        "id": "GNYoMNUWahQX",
        "outputId": "8c4812ce-2dc6-423e-c053-11fc525ffe76",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:497: UserWarning: Can't initialize NVML\n",
            "  warnings.warn(\"Can't initialize NVML\")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# preparation of the text\n",
        "stopwords=list(STOP_WORDS)\n",
        "punctuation=punctuation+ '\\n'\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "doc= nlp(scrape(URL))\n",
        "tokens=[token.text for token in doc]\n",
        "\n",
        "tokens"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zJKzupPEpECN",
        "outputId": "99c73265-a9c5-48e9-cf79-81c4325a2007"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Margaret',\n",
              " 'Heafield',\n",
              " 'Hamilton',\n",
              " '(',\n",
              " 'born',\n",
              " 'August',\n",
              " '17',\n",
              " ',',\n",
              " '1936',\n",
              " ')',\n",
              " 'is',\n",
              " 'an',\n",
              " 'American',\n",
              " 'computer',\n",
              " 'scientist',\n",
              " ',',\n",
              " 'systems',\n",
              " 'engineer',\n",
              " ',',\n",
              " 'and',\n",
              " 'business',\n",
              " 'owner',\n",
              " '.',\n",
              " 'She',\n",
              " 'was',\n",
              " 'director',\n",
              " 'of',\n",
              " 'the',\n",
              " 'Software',\n",
              " 'Engineering',\n",
              " 'Division',\n",
              " 'of',\n",
              " 'the',\n",
              " 'MIT',\n",
              " 'Instrumentation',\n",
              " 'Laboratory',\n",
              " ',',\n",
              " 'which',\n",
              " 'developed',\n",
              " 'on',\n",
              " '-',\n",
              " 'board',\n",
              " 'flight',\n",
              " 'software',\n",
              " 'for',\n",
              " 'NASA',\n",
              " \"'s\",\n",
              " 'Apollo',\n",
              " 'program',\n",
              " '.',\n",
              " 'She',\n",
              " 'later',\n",
              " 'founded',\n",
              " 'two',\n",
              " 'software',\n",
              " 'companies',\n",
              " '—',\n",
              " 'Higher',\n",
              " 'Order',\n",
              " 'Software',\n",
              " 'in',\n",
              " '1976',\n",
              " 'and',\n",
              " 'Hamilton',\n",
              " 'Technologies',\n",
              " 'in',\n",
              " '1986',\n",
              " ',',\n",
              " 'both',\n",
              " 'in',\n",
              " 'Cambridge',\n",
              " ',',\n",
              " 'Massachusetts',\n",
              " '.',\n",
              " '\\n',\n",
              " 'Hamilton',\n",
              " 'has',\n",
              " 'published',\n",
              " 'more',\n",
              " 'than',\n",
              " '130',\n",
              " 'papers',\n",
              " ',',\n",
              " 'proceedings',\n",
              " ',',\n",
              " 'and',\n",
              " 'reports',\n",
              " ',',\n",
              " 'about',\n",
              " 'sixty',\n",
              " 'projects',\n",
              " ',',\n",
              " 'and',\n",
              " 'six',\n",
              " 'major',\n",
              " 'programs',\n",
              " '.',\n",
              " 'She',\n",
              " 'is',\n",
              " 'one',\n",
              " 'of',\n",
              " 'the',\n",
              " 'people',\n",
              " 'credited',\n",
              " 'with',\n",
              " 'coining',\n",
              " 'the',\n",
              " 'term',\n",
              " '\"',\n",
              " 'software',\n",
              " 'engineering',\n",
              " '\"',\n",
              " '.',\n",
              " '\\n',\n",
              " 'On',\n",
              " 'November',\n",
              " '22',\n",
              " ',',\n",
              " '2016',\n",
              " ',',\n",
              " 'Hamilton',\n",
              " 'received',\n",
              " 'the',\n",
              " 'Presidential',\n",
              " 'Medal',\n",
              " 'of',\n",
              " 'Freedom',\n",
              " 'from',\n",
              " 'president',\n",
              " 'Barack',\n",
              " 'Obama',\n",
              " 'for',\n",
              " 'her',\n",
              " 'work',\n",
              " 'leading',\n",
              " 'to',\n",
              " 'the',\n",
              " 'development',\n",
              " 'of',\n",
              " 'on',\n",
              " '-',\n",
              " 'board',\n",
              " 'flight',\n",
              " 'software',\n",
              " 'for',\n",
              " 'NASA',\n",
              " \"'s\",\n",
              " 'Apollo',\n",
              " 'Moon',\n",
              " 'missions',\n",
              " '.',\n",
              " '\\n',\n",
              " 'Margaret',\n",
              " 'Elaine',\n",
              " 'Heafield',\n",
              " 'was',\n",
              " 'born',\n",
              " 'August',\n",
              " '17',\n",
              " ',',\n",
              " '1936',\n",
              " ',',\n",
              " 'in',\n",
              " 'Paoli',\n",
              " ',',\n",
              " 'Indiana',\n",
              " ',',\n",
              " 'to',\n",
              " 'Kenneth',\n",
              " 'Heafield',\n",
              " 'and',\n",
              " 'Ruth',\n",
              " 'Esther',\n",
              " 'Heafield',\n",
              " '(',\n",
              " 'née',\n",
              " 'Partington',\n",
              " ')',\n",
              " '.',\n",
              " 'The',\n",
              " 'family',\n",
              " 'later',\n",
              " 'moved',\n",
              " 'to',\n",
              " 'Michigan',\n",
              " ',',\n",
              " 'where',\n",
              " 'Margaret',\n",
              " 'graduated',\n",
              " 'from',\n",
              " 'Hancock',\n",
              " 'High',\n",
              " 'School',\n",
              " 'in',\n",
              " '1954',\n",
              " '.',\n",
              " 'She',\n",
              " 'studied',\n",
              " 'mathematics',\n",
              " 'at',\n",
              " 'the',\n",
              " 'University',\n",
              " 'of',\n",
              " 'Michigan',\n",
              " 'in',\n",
              " '1955',\n",
              " 'before',\n",
              " 'transferring',\n",
              " 'to',\n",
              " 'Earlham',\n",
              " 'College',\n",
              " ',',\n",
              " 'where',\n",
              " 'her',\n",
              " 'mother',\n",
              " 'was',\n",
              " 'a',\n",
              " 'student',\n",
              " ';',\n",
              " 'she',\n",
              " 'earned',\n",
              " 'a',\n",
              " 'BA',\n",
              " 'in',\n",
              " 'mathematics',\n",
              " 'with',\n",
              " 'a',\n",
              " 'minor',\n",
              " 'in',\n",
              " 'philosophy',\n",
              " 'in',\n",
              " '1958',\n",
              " '.',\n",
              " 'She',\n",
              " 'cites',\n",
              " 'Florence',\n",
              " 'Long',\n",
              " ',',\n",
              " 'the',\n",
              " 'head',\n",
              " 'of',\n",
              " 'the',\n",
              " 'math',\n",
              " 'department',\n",
              " 'at',\n",
              " 'Earlham',\n",
              " ',',\n",
              " 'as',\n",
              " 'helping',\n",
              " 'with',\n",
              " 'her',\n",
              " 'desire',\n",
              " 'to',\n",
              " 'pursue',\n",
              " 'abstract',\n",
              " 'mathematics',\n",
              " 'and',\n",
              " 'become',\n",
              " 'a',\n",
              " 'mathematics',\n",
              " 'professor',\n",
              " '.',\n",
              " 'She',\n",
              " 'says',\n",
              " 'her',\n",
              " 'poet',\n",
              " 'father',\n",
              " 'and',\n",
              " 'headmaster',\n",
              " 'grandfather',\n",
              " 'inspired',\n",
              " 'her',\n",
              " 'to',\n",
              " 'include',\n",
              " 'a',\n",
              " 'minor',\n",
              " 'in',\n",
              " 'philosophy',\n",
              " 'in',\n",
              " 'her',\n",
              " 'studies',\n",
              " '.',\n",
              " '\\n',\n",
              " 'In',\n",
              " 'Boston',\n",
              " ',',\n",
              " 'Hamilton',\n",
              " 'initially',\n",
              " 'intended',\n",
              " 'to',\n",
              " 'enroll',\n",
              " 'in',\n",
              " 'graduate',\n",
              " 'study',\n",
              " 'in',\n",
              " 'abstract',\n",
              " 'mathematics',\n",
              " 'at',\n",
              " 'Brandeis',\n",
              " '.',\n",
              " 'However',\n",
              " ',',\n",
              " 'in',\n",
              " 'mid-1959',\n",
              " ',',\n",
              " 'Hamilton',\n",
              " 'began',\n",
              " 'working',\n",
              " 'for',\n",
              " 'Edward',\n",
              " 'Norton',\n",
              " 'Lorenz',\n",
              " ',',\n",
              " 'in',\n",
              " 'the',\n",
              " 'meteorology',\n",
              " 'department',\n",
              " 'at',\n",
              " 'MIT',\n",
              " '.',\n",
              " 'She',\n",
              " 'developed',\n",
              " 'software',\n",
              " 'for',\n",
              " 'predicting',\n",
              " 'weather',\n",
              " ',',\n",
              " 'programming',\n",
              " 'on',\n",
              " 'the',\n",
              " 'LGP-30',\n",
              " 'and',\n",
              " 'the',\n",
              " 'PDP-1',\n",
              " 'computers',\n",
              " 'at',\n",
              " 'Marvin',\n",
              " 'Minsky',\n",
              " \"'s\",\n",
              " 'Project',\n",
              " 'MAC',\n",
              " '.',\n",
              " 'Her',\n",
              " 'work',\n",
              " 'contributed',\n",
              " 'to',\n",
              " 'Lorenz',\n",
              " \"'s\",\n",
              " 'publications',\n",
              " 'on',\n",
              " 'chaos',\n",
              " 'theory',\n",
              " '.',\n",
              " 'At',\n",
              " 'the',\n",
              " 'time',\n",
              " ',',\n",
              " 'computer',\n",
              " 'science',\n",
              " 'and',\n",
              " 'software',\n",
              " 'engineering',\n",
              " 'were',\n",
              " 'not',\n",
              " 'yet',\n",
              " 'established',\n",
              " 'disciplines',\n",
              " ';',\n",
              " 'instead',\n",
              " ',',\n",
              " 'programmers',\n",
              " 'learned',\n",
              " 'on',\n",
              " 'the',\n",
              " 'job',\n",
              " 'with',\n",
              " 'hands',\n",
              " '-',\n",
              " 'on',\n",
              " 'experience',\n",
              " '.',\n",
              " 'She',\n",
              " 'moved',\n",
              " 'on',\n",
              " 'to',\n",
              " 'another',\n",
              " 'project',\n",
              " 'in',\n",
              " 'the',\n",
              " 'summer',\n",
              " 'of',\n",
              " '1961',\n",
              " ',',\n",
              " 'and',\n",
              " 'hired',\n",
              " 'and',\n",
              " 'trained',\n",
              " 'Ellen',\n",
              " 'Fetter',\n",
              " 'as',\n",
              " 'her',\n",
              " 'replacement',\n",
              " '.',\n",
              " '\\n',\n",
              " 'From',\n",
              " '1961',\n",
              " 'to',\n",
              " '1963',\n",
              " ',',\n",
              " 'Hamilton',\n",
              " 'worked',\n",
              " 'on',\n",
              " 'the',\n",
              " 'Semi',\n",
              " '-',\n",
              " 'Automatic',\n",
              " 'Ground',\n",
              " 'Environment',\n",
              " '(',\n",
              " 'SAGE',\n",
              " ')',\n",
              " 'Project',\n",
              " 'at',\n",
              " 'the',\n",
              " 'MIT',\n",
              " 'Lincoln',\n",
              " 'Lab',\n",
              " ',',\n",
              " 'where',\n",
              " 'she',\n",
              " 'was',\n",
              " 'one',\n",
              " 'of',\n",
              " 'the',\n",
              " 'programmers',\n",
              " 'who',\n",
              " 'wrote',\n",
              " 'software',\n",
              " 'for',\n",
              " 'the',\n",
              " 'prototype',\n",
              " 'AN',\n",
              " '/',\n",
              " 'FSQ-7',\n",
              " 'computer',\n",
              " '(',\n",
              " 'the',\n",
              " 'XD-1',\n",
              " ')',\n",
              " ',',\n",
              " 'used',\n",
              " 'by',\n",
              " 'the',\n",
              " 'U.S.',\n",
              " 'Air',\n",
              " 'Force',\n",
              " 'to',\n",
              " 'search',\n",
              " 'for',\n",
              " 'possibly',\n",
              " 'unfriendly',\n",
              " 'aircraft',\n",
              " '.',\n",
              " 'She',\n",
              " 'also',\n",
              " 'wrote',\n",
              " 'software',\n",
              " 'for',\n",
              " 'a',\n",
              " 'satellite',\n",
              " 'tracking',\n",
              " 'project',\n",
              " 'at',\n",
              " 'the',\n",
              " 'Air',\n",
              " 'Force',\n",
              " 'Cambridge',\n",
              " 'Research',\n",
              " 'Laboratories',\n",
              " '.',\n",
              " 'The',\n",
              " 'SAGE',\n",
              " 'Project',\n",
              " 'was',\n",
              " 'an',\n",
              " 'extension',\n",
              " 'of',\n",
              " 'Project',\n",
              " 'Whirlwind',\n",
              " ',',\n",
              " 'started',\n",
              " 'by',\n",
              " 'MIT',\n",
              " 'to',\n",
              " 'create',\n",
              " 'a',\n",
              " 'computer',\n",
              " 'system',\n",
              " 'that',\n",
              " 'could',\n",
              " 'predict',\n",
              " 'weather',\n",
              " 'systems',\n",
              " 'and',\n",
              " 'track',\n",
              " 'their',\n",
              " 'movements',\n",
              " 'using',\n",
              " 'simulators',\n",
              " '.',\n",
              " 'SAGE',\n",
              " 'was',\n",
              " 'soon',\n",
              " 'developed',\n",
              " 'for',\n",
              " 'military',\n",
              " 'use',\n",
              " 'in',\n",
              " 'anti',\n",
              " '-',\n",
              " 'aircraft',\n",
              " 'air',\n",
              " 'defense',\n",
              " '.',\n",
              " 'Hamilton',\n",
              " 'said',\n",
              " ':',\n",
              " '\\n',\n",
              " 'What',\n",
              " 'they',\n",
              " 'used',\n",
              " 'to',\n",
              " 'do',\n",
              " 'when',\n",
              " 'you',\n",
              " 'came',\n",
              " 'into',\n",
              " 'this',\n",
              " 'organization',\n",
              " 'as',\n",
              " 'a',\n",
              " 'beginner',\n",
              " ',',\n",
              " 'was',\n",
              " 'to',\n",
              " 'assign',\n",
              " 'you',\n",
              " 'this',\n",
              " 'program',\n",
              " 'which',\n",
              " 'nobody',\n",
              " 'was',\n",
              " 'able',\n",
              " 'to',\n",
              " 'ever',\n",
              " 'figure',\n",
              " 'out',\n",
              " 'or',\n",
              " 'get',\n",
              " 'to',\n",
              " 'run',\n",
              " '.',\n",
              " 'When',\n",
              " 'I',\n",
              " 'was',\n",
              " 'the',\n",
              " 'beginner',\n",
              " 'they',\n",
              " 'gave',\n",
              " 'it',\n",
              " 'to',\n",
              " 'me',\n",
              " 'as',\n",
              " 'well',\n",
              " '.',\n",
              " 'And',\n",
              " 'what',\n",
              " 'had',\n",
              " 'happened',\n",
              " 'was',\n",
              " 'it',\n",
              " 'was',\n",
              " 'tricky',\n",
              " 'programming',\n",
              " ',',\n",
              " 'and',\n",
              " 'the',\n",
              " 'person',\n",
              " 'who',\n",
              " 'wrote',\n",
              " 'it',\n",
              " 'took',\n",
              " 'delight',\n",
              " 'in',\n",
              " 'the',\n",
              " 'fact',\n",
              " 'that',\n",
              " 'all',\n",
              " 'of',\n",
              " 'his',\n",
              " 'comments',\n",
              " 'were',\n",
              " 'in',\n",
              " 'Greek',\n",
              " 'and',\n",
              " 'Latin',\n",
              " '.',\n",
              " 'So',\n",
              " 'I',\n",
              " 'was',\n",
              " 'assigned',\n",
              " 'this',\n",
              " 'program',\n",
              " 'and',\n",
              " 'I',\n",
              " 'actually',\n",
              " 'got',\n",
              " 'it',\n",
              " 'to',\n",
              " 'work',\n",
              " '.',\n",
              " 'It',\n",
              " 'even',\n",
              " 'printed',\n",
              " 'out',\n",
              " 'its',\n",
              " 'answers',\n",
              " 'in',\n",
              " 'Latin',\n",
              " 'and',\n",
              " 'Greek',\n",
              " '.',\n",
              " 'I',\n",
              " 'was',\n",
              " 'the',\n",
              " 'first',\n",
              " 'one',\n",
              " 'to',\n",
              " 'get',\n",
              " 'it',\n",
              " 'to',\n",
              " 'work',\n",
              " '.',\n",
              " 'It',\n",
              " 'was',\n",
              " 'her',\n",
              " 'efforts',\n",
              " 'on',\n",
              " 'this',\n",
              " 'project',\n",
              " 'that',\n",
              " 'made',\n",
              " 'her',\n",
              " 'a',\n",
              " 'candidate',\n",
              " 'for',\n",
              " 'the',\n",
              " 'position',\n",
              " 'at',\n",
              " 'NASA',\n",
              " 'as',\n",
              " 'the',\n",
              " 'lead',\n",
              " 'developer',\n",
              " 'for',\n",
              " 'Apollo',\n",
              " 'flight',\n",
              " 'software',\n",
              " '.',\n",
              " '\\n',\n",
              " 'Hamilton',\n",
              " 'then',\n",
              " 'joined',\n",
              " 'the',\n",
              " 'Charles',\n",
              " 'Stark',\n",
              " 'Draper',\n",
              " 'Laboratory',\n",
              " 'at',\n",
              " 'MIT',\n",
              " ',',\n",
              " 'which',\n",
              " 'worked',\n",
              " 'on',\n",
              " 'the',\n",
              " 'Apollo',\n",
              " 'Space',\n",
              " 'Mission',\n",
              " '.',\n",
              " 'Hamilton',\n",
              " 'was',\n",
              " 'initially',\n",
              " 'hired',\n",
              " 'as',\n",
              " 'a',\n",
              " 'programmer',\n",
              " 'for',\n",
              " 'this',\n",
              " 'process',\n",
              " 'but',\n",
              " 'moved',\n",
              " 'on',\n",
              " 'into',\n",
              " 'systems',\n",
              " 'design',\n",
              " '.',\n",
              " 'By',\n",
              " '1965',\n",
              " ',',\n",
              " 'she',\n",
              " 'was',\n",
              " 'in',\n",
              " 'charge',\n",
              " 'of',\n",
              " 'all',\n",
              " 'Command',\n",
              " 'Module',\n",
              " 'software',\n",
              " ',',\n",
              " 'which',\n",
              " 'was',\n",
              " 'all',\n",
              " 'the',\n",
              " 'software',\n",
              " 'for',\n",
              " 'navigation',\n",
              " 'and',\n",
              " 'lunar',\n",
              " 'landing',\n",
              " 'guidance',\n",
              " '.',\n",
              " 'Hamilton',\n",
              " \"'s\",\n",
              " 'team',\n",
              " 'was',\n",
              " 'responsible',\n",
              " 'for',\n",
              " 'developing',\n",
              " 'in',\n",
              " '-',\n",
              " 'flight',\n",
              " 'software',\n",
              " ',',\n",
              " 'which',\n",
              " 'included',\n",
              " 'algorithms',\n",
              " 'designed',\n",
              " 'by',\n",
              " 'various',\n",
              " 'senior',\n",
              " 'scientists',\n",
              " 'for',\n",
              " 'the',\n",
              " 'Apollo',\n",
              " 'command',\n",
              " 'module',\n",
              " ',',\n",
              " 'Apollo',\n",
              " 'lunar',\n",
              " 'module',\n",
              " ',',\n",
              " 'and',\n",
              " 'the',\n",
              " 'subsequent',\n",
              " 'Skylab',\n",
              " '.',\n",
              " 'Another',\n",
              " 'part',\n",
              " 'of',\n",
              " 'her',\n",
              " 'team',\n",
              " 'designed',\n",
              " 'and',\n",
              " 'developed',\n",
              " 'the',\n",
              " 'systems',\n",
              " 'software',\n",
              " '.',\n",
              " 'This',\n",
              " 'included',\n",
              " 'error',\n",
              " 'detection',\n",
              " 'and',\n",
              " 'recovery',\n",
              " 'software',\n",
              " 'such',\n",
              " 'as',\n",
              " 'restarts',\n",
              " 'and',\n",
              " 'the',\n",
              " 'Display',\n",
              " 'Interface',\n",
              " 'Routines',\n",
              " '(',\n",
              " 'also',\n",
              " 'known',\n",
              " 'as',\n",
              " 'the',\n",
              " 'Priority',\n",
              " 'Displays',\n",
              " ')',\n",
              " ',',\n",
              " 'which',\n",
              " 'Hamilton',\n",
              " 'designed',\n",
              " 'and',\n",
              " 'developed',\n",
              " '.',\n",
              " 'She',\n",
              " 'worked',\n",
              " 'to',\n",
              " 'gain',\n",
              " 'hands',\n",
              " '-',\n",
              " 'on',\n",
              " 'experience',\n",
              " 'during',\n",
              " 'a',\n",
              " 'time',\n",
              " 'when',\n",
              " 'computer',\n",
              " 'science',\n",
              " 'courses',\n",
              " 'were',\n",
              " 'uncommon',\n",
              " 'and',\n",
              " 'software',\n",
              " 'engineering',\n",
              " 'courses',\n",
              " 'did',\n",
              " 'not',\n",
              " 'exist',\n",
              " '.',\n",
              " 'Hamilton',\n",
              " 'also',\n",
              " 'served',\n",
              " 'as',\n",
              " 'Director',\n",
              " 'of',\n",
              " 'the',\n",
              " 'Software',\n",
              " 'Engineering',\n",
              " 'Division',\n",
              " '.',\n",
              " '\\n',\n",
              " 'Her',\n",
              " 'areas',\n",
              " 'of',\n",
              " 'expertise',\n",
              " 'include',\n",
              " ':',\n",
              " 'systems',\n",
              " 'design',\n",
              " 'and',\n",
              " 'software',\n",
              " 'development',\n",
              " ',',\n",
              " 'enterprise',\n",
              " 'and',\n",
              " 'process',\n",
              " 'modeling',\n",
              " ',',\n",
              " 'development',\n",
              " 'paradigm',\n",
              " ',',\n",
              " 'formal',\n",
              " 'systems',\n",
              " 'modeling',\n",
              " 'languages',\n",
              " ',',\n",
              " 'system',\n",
              " '-',\n",
              " 'oriented',\n",
              " 'objects',\n",
              " 'for',\n",
              " 'systems',\n",
              " 'modeling',\n",
              " 'and',\n",
              " 'development',\n",
              " ',',\n",
              " 'automated',\n",
              " 'life',\n",
              " '-',\n",
              " 'cycle',\n",
              " 'environments',\n",
              " ',',\n",
              " 'methods',\n",
              " 'for',\n",
              " 'maximizing',\n",
              " 'software',\n",
              " 'reliability',\n",
              " 'and',\n",
              " 'reuse',\n",
              " ',',\n",
              " 'domain',\n",
              " 'analysis',\n",
              " ',',\n",
              " 'correctness',\n",
              " 'by',\n",
              " 'built',\n",
              " '-',\n",
              " 'in',\n",
              " 'language',\n",
              " 'properties',\n",
              " ',',\n",
              " 'open',\n",
              " '-',\n",
              " 'architecture',\n",
              " 'techniques',\n",
              " 'for',\n",
              " 'robust',\n",
              " 'systems',\n",
              " ',',\n",
              " 'full',\n",
              " 'life',\n",
              " '-',\n",
              " 'cycle',\n",
              " 'automation',\n",
              " ',',\n",
              " 'quality',\n",
              " 'assurance',\n",
              " ',',\n",
              " 'seamless',\n",
              " 'integration',\n",
              " ',',\n",
              " 'error',\n",
              " 'detection',\n",
              " 'and',\n",
              " 'recovery',\n",
              " 'techniques',\n",
              " ',',\n",
              " 'human',\n",
              " '-',\n",
              " 'machine',\n",
              " 'interface',\n",
              " 'systems',\n",
              " ',',\n",
              " 'operating',\n",
              " 'systems',\n",
              " ',',\n",
              " 'end',\n",
              " '-',\n",
              " 'to',\n",
              " '-',\n",
              " 'end',\n",
              " 'testing',\n",
              " 'techniques',\n",
              " ',',\n",
              " 'and',\n",
              " 'life',\n",
              " '-',\n",
              " 'cycle',\n",
              " 'management',\n",
              " 'techniques',\n",
              " '.',\n",
              " 'These',\n",
              " 'made',\n",
              " 'her',\n",
              " 'code',\n",
              " 'incredibly',\n",
              " 'reliable',\n",
              " 'because',\n",
              " 'they',\n",
              " 'helped',\n",
              " 'programmers',\n",
              " 'identify',\n",
              " 'and',\n",
              " 'fix',\n",
              " 'anomalies',\n",
              " 'before',\n",
              " 'they',\n",
              " 'became',\n",
              " 'major',\n",
              " 'problems',\n",
              " '.',\n",
              " '\\n',\n",
              " 'In',\n",
              " 'one',\n",
              " 'of',\n",
              " 'the',\n",
              " 'critical',\n",
              " 'moments',\n",
              " 'of',\n",
              " 'the',\n",
              " 'Apollo',\n",
              " '11',\n",
              " 'mission',\n",
              " ',',\n",
              " 'the',\n",
              " 'Apollo',\n",
              " 'Guidance',\n",
              " 'Computer',\n",
              " ',',\n",
              " 'together',\n",
              " 'with',\n",
              " 'the',\n",
              " 'on',\n",
              " '-',\n",
              " 'board',\n",
              " 'flight',\n",
              " ...]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# word frequencies\n",
        "word_frequencies={}\n",
        "for word in doc:\n",
        "    if word.text.lower() not in stopwords:\n",
        "        if word.text.lower() not in punctuation:\n",
        "            if word.text not in word_frequencies.keys():\n",
        "                word_frequencies[word.text] = 1\n",
        "            else:\n",
        "                word_frequencies[word.text] += 1\n",
        "\n",
        "word_frequencies"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3_CJmJuXpdey",
        "outputId": "84cb1352-d47c-4f3a-eb76-7802fd235c11"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Margaret': 4,\n",
              " 'Heafield': 4,\n",
              " 'Hamilton': 34,\n",
              " 'born': 3,\n",
              " 'August': 2,\n",
              " '17': 2,\n",
              " '1936': 2,\n",
              " 'American': 1,\n",
              " 'computer': 16,\n",
              " 'scientist': 2,\n",
              " 'systems': 12,\n",
              " 'engineer': 2,\n",
              " 'business': 1,\n",
              " 'owner': 1,\n",
              " 'director': 1,\n",
              " 'Software': 5,\n",
              " 'Engineering': 2,\n",
              " 'Division': 2,\n",
              " 'MIT': 8,\n",
              " 'Instrumentation': 1,\n",
              " 'Laboratory': 2,\n",
              " 'developed': 8,\n",
              " 'board': 4,\n",
              " 'flight': 8,\n",
              " 'software': 38,\n",
              " 'NASA': 5,\n",
              " 'Apollo': 13,\n",
              " 'program': 5,\n",
              " 'later': 4,\n",
              " 'founded': 3,\n",
              " 'companies': 1,\n",
              " '—': 1,\n",
              " 'Higher': 2,\n",
              " 'Order': 2,\n",
              " '1976': 2,\n",
              " 'Technologies': 2,\n",
              " '1986': 2,\n",
              " 'Cambridge': 3,\n",
              " 'Massachusetts': 3,\n",
              " 'published': 2,\n",
              " '130': 1,\n",
              " 'papers': 1,\n",
              " 'proceedings': 1,\n",
              " 'reports': 1,\n",
              " 'projects': 1,\n",
              " 'major': 2,\n",
              " 'programs': 3,\n",
              " 'people': 1,\n",
              " 'credited': 2,\n",
              " 'coining': 1,\n",
              " 'term': 6,\n",
              " 'engineering': 13,\n",
              " 'November': 2,\n",
              " '22': 1,\n",
              " '2016': 1,\n",
              " 'received': 1,\n",
              " 'Presidential': 1,\n",
              " 'Medal': 1,\n",
              " 'Freedom': 1,\n",
              " 'president': 1,\n",
              " 'Barack': 1,\n",
              " 'Obama': 1,\n",
              " 'work': 6,\n",
              " 'leading': 1,\n",
              " 'development': 8,\n",
              " 'Moon': 4,\n",
              " 'missions': 2,\n",
              " 'Elaine': 1,\n",
              " 'Paoli': 1,\n",
              " 'Indiana': 2,\n",
              " 'Kenneth': 1,\n",
              " 'Ruth': 1,\n",
              " 'Esther': 1,\n",
              " 'née': 1,\n",
              " 'Partington': 1,\n",
              " 'family': 1,\n",
              " 'moved': 4,\n",
              " 'Michigan': 2,\n",
              " 'graduated': 2,\n",
              " 'Hancock': 1,\n",
              " 'High': 1,\n",
              " 'School': 1,\n",
              " '1954': 1,\n",
              " 'studied': 1,\n",
              " 'mathematics': 6,\n",
              " 'University': 1,\n",
              " '1955': 1,\n",
              " 'transferring': 1,\n",
              " 'Earlham': 3,\n",
              " 'College': 1,\n",
              " 'mother': 1,\n",
              " 'student': 1,\n",
              " 'earned': 2,\n",
              " 'BA': 1,\n",
              " 'minor': 2,\n",
              " 'philosophy': 2,\n",
              " '1958': 2,\n",
              " 'cites': 1,\n",
              " 'Florence': 1,\n",
              " 'Long': 1,\n",
              " 'head': 1,\n",
              " 'math': 1,\n",
              " 'department': 2,\n",
              " 'helping': 2,\n",
              " 'desire': 1,\n",
              " 'pursue': 1,\n",
              " 'abstract': 2,\n",
              " 'professor': 1,\n",
              " 'says': 1,\n",
              " 'poet': 1,\n",
              " 'father': 1,\n",
              " 'headmaster': 1,\n",
              " 'grandfather': 1,\n",
              " 'inspired': 1,\n",
              " 'include': 2,\n",
              " 'studies': 1,\n",
              " 'Boston': 3,\n",
              " 'initially': 2,\n",
              " 'intended': 1,\n",
              " 'enroll': 1,\n",
              " 'graduate': 1,\n",
              " 'study': 1,\n",
              " 'Brandeis': 1,\n",
              " 'mid-1959': 1,\n",
              " 'began': 1,\n",
              " 'working': 2,\n",
              " 'Edward': 1,\n",
              " 'Norton': 1,\n",
              " 'Lorenz': 2,\n",
              " 'meteorology': 1,\n",
              " 'predicting': 1,\n",
              " 'weather': 2,\n",
              " 'programming': 5,\n",
              " 'LGP-30': 1,\n",
              " 'PDP-1': 1,\n",
              " 'computers': 1,\n",
              " 'Marvin': 1,\n",
              " 'Minsky': 1,\n",
              " 'Project': 4,\n",
              " 'MAC': 1,\n",
              " 'contributed': 1,\n",
              " 'publications': 1,\n",
              " 'chaos': 1,\n",
              " 'theory': 1,\n",
              " 'time': 8,\n",
              " 'science': 3,\n",
              " 'established': 1,\n",
              " 'disciplines': 1,\n",
              " 'instead': 1,\n",
              " 'programmers': 3,\n",
              " 'learned': 1,\n",
              " 'job': 1,\n",
              " 'hands': 2,\n",
              " 'experience': 3,\n",
              " 'project': 5,\n",
              " 'summer': 2,\n",
              " '1961': 2,\n",
              " 'hired': 2,\n",
              " 'trained': 1,\n",
              " 'Ellen': 1,\n",
              " 'Fetter': 1,\n",
              " 'replacement': 1,\n",
              " '1963': 1,\n",
              " 'worked': 4,\n",
              " 'Semi': 1,\n",
              " 'Automatic': 1,\n",
              " 'Ground': 1,\n",
              " 'Environment': 1,\n",
              " 'SAGE': 3,\n",
              " 'Lincoln': 1,\n",
              " 'Lab': 1,\n",
              " 'wrote': 4,\n",
              " 'prototype': 1,\n",
              " 'FSQ-7': 1,\n",
              " 'XD-1': 1,\n",
              " 'U.S.': 2,\n",
              " 'Air': 3,\n",
              " 'Force': 3,\n",
              " 'search': 1,\n",
              " 'possibly': 1,\n",
              " 'unfriendly': 1,\n",
              " 'aircraft': 2,\n",
              " 'satellite': 1,\n",
              " 'tracking': 1,\n",
              " 'Research': 1,\n",
              " 'Laboratories': 1,\n",
              " 'extension': 1,\n",
              " 'Whirlwind': 1,\n",
              " 'started': 2,\n",
              " 'create': 2,\n",
              " 'system': 5,\n",
              " 'predict': 1,\n",
              " 'track': 1,\n",
              " 'movements': 1,\n",
              " 'simulators': 1,\n",
              " 'soon': 1,\n",
              " 'military': 1,\n",
              " 'use': 1,\n",
              " 'anti': 1,\n",
              " 'air': 1,\n",
              " 'defense': 1,\n",
              " 'said': 1,\n",
              " 'came': 3,\n",
              " 'organization': 1,\n",
              " 'beginner': 2,\n",
              " 'assign': 1,\n",
              " 'able': 2,\n",
              " 'figure': 1,\n",
              " 'run': 1,\n",
              " 'gave': 1,\n",
              " 'happened': 1,\n",
              " 'tricky': 1,\n",
              " 'person': 1,\n",
              " 'took': 1,\n",
              " 'delight': 1,\n",
              " 'fact': 2,\n",
              " 'comments': 1,\n",
              " 'Greek': 2,\n",
              " 'Latin': 2,\n",
              " 'assigned': 1,\n",
              " 'actually': 1,\n",
              " 'got': 1,\n",
              " 'printed': 1,\n",
              " 'answers': 1,\n",
              " 'efforts': 1,\n",
              " 'candidate': 1,\n",
              " 'position': 1,\n",
              " 'lead': 1,\n",
              " 'developer': 1,\n",
              " 'joined': 1,\n",
              " 'Charles': 1,\n",
              " 'Stark': 1,\n",
              " 'Draper': 1,\n",
              " 'Space': 2,\n",
              " 'Mission': 1,\n",
              " 'programmer': 1,\n",
              " 'process': 4,\n",
              " 'design': 5,\n",
              " '1965': 1,\n",
              " 'charge': 1,\n",
              " 'Command': 1,\n",
              " 'Module': 1,\n",
              " 'navigation': 1,\n",
              " 'lunar': 3,\n",
              " 'landing': 5,\n",
              " 'guidance': 2,\n",
              " 'team': 3,\n",
              " 'responsible': 1,\n",
              " 'developing': 1,\n",
              " 'included': 3,\n",
              " 'algorithms': 1,\n",
              " 'designed': 4,\n",
              " 'senior': 2,\n",
              " 'scientists': 1,\n",
              " 'command': 1,\n",
              " 'module': 2,\n",
              " 'subsequent': 1,\n",
              " 'Skylab': 1,\n",
              " 'error': 5,\n",
              " 'detection': 3,\n",
              " 'recovery': 5,\n",
              " 'restarts': 1,\n",
              " 'Display': 2,\n",
              " 'Interface': 2,\n",
              " 'Routines': 2,\n",
              " 'known': 1,\n",
              " 'Priority': 1,\n",
              " 'Displays': 1,\n",
              " 'gain': 1,\n",
              " 'courses': 2,\n",
              " 'uncommon': 1,\n",
              " 'exist': 1,\n",
              " 'served': 1,\n",
              " 'Director': 1,\n",
              " 'areas': 1,\n",
              " 'expertise': 1,\n",
              " 'enterprise': 1,\n",
              " 'modeling': 4,\n",
              " 'paradigm': 2,\n",
              " 'formal': 1,\n",
              " 'languages': 1,\n",
              " 'oriented': 1,\n",
              " 'objects': 1,\n",
              " 'automated': 3,\n",
              " 'life': 3,\n",
              " 'cycle': 3,\n",
              " 'environments': 1,\n",
              " 'methods': 1,\n",
              " 'maximizing': 1,\n",
              " 'reliability': 1,\n",
              " 'reuse': 1,\n",
              " 'domain': 1,\n",
              " 'analysis': 1,\n",
              " 'correctness': 1,\n",
              " 'built': 1,\n",
              " 'language': 4,\n",
              " 'properties': 1,\n",
              " 'open': 2,\n",
              " 'architecture': 1,\n",
              " 'techniques': 6,\n",
              " 'robust': 1,\n",
              " 'automation': 1,\n",
              " 'quality': 1,\n",
              " 'assurance': 1,\n",
              " 'seamless': 1,\n",
              " 'integration': 1,\n",
              " 'human': 1,\n",
              " 'machine': 1,\n",
              " 'interface': 1,\n",
              " 'operating': 1,\n",
              " 'end': 2,\n",
              " 'testing': 1,\n",
              " 'management': 1,\n",
              " 'code': 2,\n",
              " 'incredibly': 1,\n",
              " 'reliable': 3,\n",
              " 'helped': 1,\n",
              " 'identify': 1,\n",
              " 'fix': 1,\n",
              " 'anomalies': 1,\n",
              " 'problems': 1,\n",
              " 'critical': 2,\n",
              " 'moments': 1,\n",
              " '11': 3,\n",
              " 'mission': 3,\n",
              " 'Guidance': 1,\n",
              " 'Computer': 2,\n",
              " 'averted': 1,\n",
              " 'abort': 1,\n",
              " 'minutes': 1,\n",
              " 'lander': 2,\n",
              " 'reached': 1,\n",
              " 'surface': 1,\n",
              " 'alarms': 6,\n",
              " 'triggered': 2,\n",
              " 'captured': 1,\n",
              " 'supposed': 1,\n",
              " 'happen': 1,\n",
              " 'displays': 9,\n",
              " 'interrupting': 1,\n",
              " 'astronauts': 7,\n",
              " 'priority': 11,\n",
              " 'alarm': 3,\n",
              " 'prepared': 1,\n",
              " 'situation': 1,\n",
              " 'years': 3,\n",
              " 'failsafe': 1,\n",
              " 'likes': 1,\n",
              " 'remember': 1,\n",
              " 'display': 3,\n",
              " 'innovation': 1,\n",
              " 'created': 3,\n",
              " 'knock': 1,\n",
              " 'risk': 1,\n",
              " 'astronaut': 2,\n",
              " 'slip': 1,\n",
              " 'synch': 1,\n",
              " 'mattered': 1,\n",
              " 'went': 1,\n",
              " 'replaced': 1,\n",
              " 'normal': 4,\n",
              " 'ones': 3,\n",
              " 'actual': 1,\n",
              " 'switchover': 1,\n",
              " 'new': 2,\n",
              " 'programmes': 1,\n",
              " 'screens': 1,\n",
              " 'happening': 1,\n",
              " 'step': 1,\n",
              " 'slower': 1,\n",
              " 'today': 1,\n",
              " 'thought': 1,\n",
              " 'long': 2,\n",
              " 'hard': 1,\n",
              " 'meant': 2,\n",
              " 'Aldrin': 1,\n",
              " 'hit': 1,\n",
              " 'button': 1,\n",
              " 'quickly': 1,\n",
              " 'response': 1,\n",
              " 'solution': 1,\n",
              " 'count': 1,\n",
              " 'accounts': 1,\n",
              " 'inadvertently': 2,\n",
              " 'left': 3,\n",
              " 'rendezvous': 2,\n",
              " 'radar': 3,\n",
              " 'switch': 1,\n",
              " 'causing': 1,\n",
              " 'claim': 1,\n",
              " 'disputed': 1,\n",
              " 'Robert': 2,\n",
              " 'Wills': 1,\n",
              " 'National': 1,\n",
              " 'Museum': 1,\n",
              " 'Computing': 1,\n",
              " 'overloaded': 2,\n",
              " 'interrupts': 1,\n",
              " 'caused': 1,\n",
              " 'incorrectly': 1,\n",
              " 'phased': 1,\n",
              " 'power': 1,\n",
              " 'supplied': 1,\n",
              " 'indicated': 1,\n",
              " 'executive': 2,\n",
              " 'overflows': 1,\n",
              " 'meaning': 2,\n",
              " 'complete': 2,\n",
              " 'tasks': 5,\n",
              " 'real': 2,\n",
              " 'postpone': 1,\n",
              " 'asynchronous': 2,\n",
              " 'J.': 1,\n",
              " 'Halcombe': 1,\n",
              " 'Laning': 1,\n",
              " 'develop': 2,\n",
              " 'wide': 1,\n",
              " 'kill': 1,\n",
              " 'recompute': 1,\n",
              " 'safe': 1,\n",
              " 'place': 2,\n",
              " 'restart': 1,\n",
              " 'approach': 1,\n",
              " 'snapshot': 1,\n",
              " 'rollback': 1,\n",
              " 'AKA': 1,\n",
              " 'man': 1,\n",
              " 'loop': 1,\n",
              " 'capabilities': 1,\n",
              " 'order': 3,\n",
              " 'capability': 1,\n",
              " 'interrupt': 1,\n",
              " 'case': 2,\n",
              " 'emergency': 2,\n",
              " 'depended': 1,\n",
              " 'assigning': 1,\n",
              " 'unique': 1,\n",
              " 'ensure': 1,\n",
              " 'events': 1,\n",
              " 'correct': 1,\n",
              " 'right': 2,\n",
              " 'relative': 1,\n",
              " 'going': 2,\n",
              " 'interrupted': 1,\n",
              " 'warn': 1,\n",
              " 'giving': 1,\n",
              " 'decision': 1,\n",
              " 'land': 2,\n",
              " 'Jack': 1,\n",
              " 'Garman': 1,\n",
              " 'control': 1,\n",
              " 'recognized': 2,\n",
              " 'errors': 1,\n",
              " 'presented': 1,\n",
              " 'shouted': 1,\n",
              " 'continued': 1,\n",
              " 'Paul': 1,\n",
              " 'Curto': 1,\n",
              " 'technologist': 1,\n",
              " 'nominated': 1,\n",
              " 'Act': 1,\n",
              " 'Award': 1,\n",
              " 'called': 3,\n",
              " 'foundation': 1,\n",
              " 'ultra': 1,\n",
              " 'incident': 1,\n",
              " 'smart': 1,\n",
              " 'recognize': 2,\n",
              " 'asked': 1,\n",
              " 'perform': 1,\n",
              " 'performing': 1,\n",
              " 'sent': 1,\n",
              " 'important': 3,\n",
              " 'i.e.': 1,\n",
              " 'needed': 1,\n",
              " '\\xa0': 2,\n",
              " '...': 2,\n",
              " 'Actually': 1,\n",
              " 'programmed': 1,\n",
              " 'conditions': 1,\n",
              " 'set': 1,\n",
              " 'incorporated': 1,\n",
              " 'action': 2,\n",
              " 'eliminate': 1,\n",
              " 'lower': 1,\n",
              " 'establish': 1,\n",
              " 'problem': 1,\n",
              " 'taken': 2,\n",
              " 'doubt': 1,\n",
              " 'successful': 1,\n",
              " 'moon': 1,\n",
              " 'co': 1,\n",
              " 'Saydean': 1,\n",
              " 'Zeldin': 1,\n",
              " 'company': 3,\n",
              " 'HOS': 5,\n",
              " 'ideas': 2,\n",
              " 'prevention': 1,\n",
              " 'fault': 1,\n",
              " 'tolerance': 1,\n",
              " 'emerging': 1,\n",
              " 'product': 1,\n",
              " 'USE.IT': 1,\n",
              " 'based': 2,\n",
              " 'methodology': 1,\n",
              " 'successfully': 1,\n",
              " 'numerous': 1,\n",
              " 'government': 1,\n",
              " 'including': 1,\n",
              " 'formalize': 3,\n",
              " 'implement': 1,\n",
              " 'C': 1,\n",
              " 'IDEF': 2,\n",
              " 'version': 1,\n",
              " 'Integrated': 1,\n",
              " 'Aided': 1,\n",
              " 'Manufacturing': 1,\n",
              " 'ICAM': 1,\n",
              " '1980': 1,\n",
              " 'British': 1,\n",
              " 'Israeli': 1,\n",
              " 'David': 1,\n",
              " 'Harel': 1,\n",
              " 'proposal': 1,\n",
              " 'structured': 1,\n",
              " 'derived': 1,\n",
              " 'viewpoint': 1,\n",
              " 'and/or': 1,\n",
              " 'subgoals': 1,\n",
              " 'semantics': 1,\n",
              " 'linguistic': 1,\n",
              " 'quantifiers': 1,\n",
              " 'embedded': 1,\n",
              " 'CEO': 1,\n",
              " '1984': 1,\n",
              " '1985': 1,\n",
              " 'March': 1,\n",
              " 'Inc.': 1,\n",
              " 'Universal': 1,\n",
              " 'Systems': 1,\n",
              " 'Language': 1,\n",
              " 'USL': 2,\n",
              " 'associated': 1,\n",
              " 'environment': 1,\n",
              " '001': 1,\n",
              " 'Tool': 1,\n",
              " 'Suite': 1,\n",
              " 'Anthony': 1,\n",
              " 'Oettinger': 1,\n",
              " 'Barry': 1,\n",
              " 'Boehm': 1,\n",
              " 'naming': 1,\n",
              " 'discipline': 4,\n",
              " 'details': 1,\n",
              " 'heard': 1,\n",
              " 'world': 2,\n",
              " 'ongoing': 1,\n",
              " 'joke': 1,\n",
              " 'liked': 1,\n",
              " 'kid': 1,\n",
              " 'radical': 1,\n",
              " 'memorable': 1,\n",
              " 'day': 1,\n",
              " 'respected': 1,\n",
              " 'hardware': 2,\n",
              " 'gurus': 1,\n",
              " 'explained': 1,\n",
              " 'meeting': 1,\n",
              " 'agreed': 1,\n",
              " 'building': 1,\n",
              " 'considered': 1,\n",
              " 'like': 2,\n",
              " 'acceptance': 2,\n",
              " 'se': 1,\n",
              " 'room': 1,\n",
              " 'field': 1,\n",
              " 'early': 2,\n",
              " 'seriously': 1,\n",
              " 'compared': 1,\n",
              " 'regarded': 1,\n",
              " 'concerned': 1,\n",
              " 'legitimizing': 1,\n",
              " 'gained': 1,\n",
              " 'respect': 1,\n",
              " 'technical': 1,\n",
              " 'IEEE': 1,\n",
              " 'September': 1,\n",
              " 'October': 1,\n",
              " '2018': 1,\n",
              " 'issue': 1,\n",
              " 'celebrates': 1,\n",
              " '50th': 1,\n",
              " 'anniversary': 1,\n",
              " 'talks': 1,\n",
              " 'Errors': 2,\n",
              " 'influenced': 1,\n",
              " 'related': 1,\n",
              " 'prevent': 1,\n",
              " 'majority': 1,\n",
              " 'Writing': 1,\n",
              " 'Wired': 2,\n",
              " 'McMillan': 1,\n",
              " 'noted': 1,\n",
              " 'assisted': 1,\n",
              " 'creation': 1,\n",
              " 'core': 1,\n",
              " 'principles': 1,\n",
              " 'colleagues': 1,\n",
              " 'writing': 1,\n",
              " 'portable': 1,\n",
              " 'innovations': 1,\n",
              " 'feats': 1,\n",
              " 'playing': 1,\n",
              " 'role': 1,\n",
              " 'getting': 1,\n",
              " 'humans': 1,\n",
              " 'According': 1,\n",
              " 'Karen': 1,\n",
              " 'Tegan': 1,\n",
              " 'Padir': 1,\n",
              " 'pioneer': 1,\n",
              " 'COBOL': 1,\n",
              " 'inventor': 1,\n",
              " 'Grace': 1,\n",
              " 'Hopper': 1,\n",
              " 'deserve': 1,\n",
              " 'tremendous': 1,\n",
              " 'credit': 1,\n",
              " 'door': 1,\n",
              " 'women': 1,\n",
              " 'enter': 1,\n",
              " 'succeed': 1,\n",
              " 'STEM': 1,\n",
              " 'fields': 1,\n",
              " '2019': 1,\n",
              " 'celebrate': 1,\n",
              " '50': 1,\n",
              " 'Google': 1,\n",
              " 'decided': 1,\n",
              " 'tribute': 1,\n",
              " 'mirrors': 1,\n",
              " 'Ivanpah': 1,\n",
              " 'plant': 1,\n",
              " 'configured': 1,\n",
              " 'picture': 1,\n",
              " 'moonlight': 1,\n",
              " 'sister': 1,\n",
              " 'Kathryn': 1,\n",
              " 'met': 1,\n",
              " 'husband': 1,\n",
              " 'James': 1,\n",
              " 'Cox': 1,\n",
              " 'mid-1950s': 1,\n",
              " 'attending': 1,\n",
              " 'college': 1,\n",
              " 'married': 2,\n",
              " 'June': 1,\n",
              " '15': 1,\n",
              " 'briefly': 1,\n",
              " 'taught': 1,\n",
              " 'high': 1,\n",
              " 'school': 2,\n",
              " 'French': 1,\n",
              " 'public': 1,\n",
              " 'couple': 1,\n",
              " 'daughter': 1,\n",
              " 'Lauren': 1,\n",
              " '10': 1,\n",
              " '1959': 1,\n",
              " 'divorced': 1,\n",
              " '1967': 1,\n",
              " 'Dan': 1,\n",
              " 'Lickly': 1}"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# normalized word frequencies\n",
        "max_frequency=max(word_frequencies.values())\n",
        "for word in word_frequencies.keys():\n",
        "    word_frequencies[word]=word_frequencies[word]/max_frequency\n",
        "\n",
        "word_frequencies"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HrMbNQAzpmdn",
        "outputId": "67ab6ae5-2589-4974-f9e3-c54a9c251459"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Margaret': 0.10526315789473684,\n",
              " 'Heafield': 0.10526315789473684,\n",
              " 'Hamilton': 0.8947368421052632,\n",
              " 'born': 0.07894736842105263,\n",
              " 'August': 0.05263157894736842,\n",
              " '17': 0.05263157894736842,\n",
              " '1936': 0.05263157894736842,\n",
              " 'American': 0.02631578947368421,\n",
              " 'computer': 0.42105263157894735,\n",
              " 'scientist': 0.05263157894736842,\n",
              " 'systems': 0.3157894736842105,\n",
              " 'engineer': 0.05263157894736842,\n",
              " 'business': 0.02631578947368421,\n",
              " 'owner': 0.02631578947368421,\n",
              " 'director': 0.02631578947368421,\n",
              " 'Software': 0.13157894736842105,\n",
              " 'Engineering': 0.05263157894736842,\n",
              " 'Division': 0.05263157894736842,\n",
              " 'MIT': 0.21052631578947367,\n",
              " 'Instrumentation': 0.02631578947368421,\n",
              " 'Laboratory': 0.05263157894736842,\n",
              " 'developed': 0.21052631578947367,\n",
              " 'board': 0.10526315789473684,\n",
              " 'flight': 0.21052631578947367,\n",
              " 'software': 1.0,\n",
              " 'NASA': 0.13157894736842105,\n",
              " 'Apollo': 0.34210526315789475,\n",
              " 'program': 0.13157894736842105,\n",
              " 'later': 0.10526315789473684,\n",
              " 'founded': 0.07894736842105263,\n",
              " 'companies': 0.02631578947368421,\n",
              " '—': 0.02631578947368421,\n",
              " 'Higher': 0.05263157894736842,\n",
              " 'Order': 0.05263157894736842,\n",
              " '1976': 0.05263157894736842,\n",
              " 'Technologies': 0.05263157894736842,\n",
              " '1986': 0.05263157894736842,\n",
              " 'Cambridge': 0.07894736842105263,\n",
              " 'Massachusetts': 0.07894736842105263,\n",
              " 'published': 0.05263157894736842,\n",
              " '130': 0.02631578947368421,\n",
              " 'papers': 0.02631578947368421,\n",
              " 'proceedings': 0.02631578947368421,\n",
              " 'reports': 0.02631578947368421,\n",
              " 'projects': 0.02631578947368421,\n",
              " 'major': 0.05263157894736842,\n",
              " 'programs': 0.07894736842105263,\n",
              " 'people': 0.02631578947368421,\n",
              " 'credited': 0.05263157894736842,\n",
              " 'coining': 0.02631578947368421,\n",
              " 'term': 0.15789473684210525,\n",
              " 'engineering': 0.34210526315789475,\n",
              " 'November': 0.05263157894736842,\n",
              " '22': 0.02631578947368421,\n",
              " '2016': 0.02631578947368421,\n",
              " 'received': 0.02631578947368421,\n",
              " 'Presidential': 0.02631578947368421,\n",
              " 'Medal': 0.02631578947368421,\n",
              " 'Freedom': 0.02631578947368421,\n",
              " 'president': 0.02631578947368421,\n",
              " 'Barack': 0.02631578947368421,\n",
              " 'Obama': 0.02631578947368421,\n",
              " 'work': 0.15789473684210525,\n",
              " 'leading': 0.02631578947368421,\n",
              " 'development': 0.21052631578947367,\n",
              " 'Moon': 0.10526315789473684,\n",
              " 'missions': 0.05263157894736842,\n",
              " 'Elaine': 0.02631578947368421,\n",
              " 'Paoli': 0.02631578947368421,\n",
              " 'Indiana': 0.05263157894736842,\n",
              " 'Kenneth': 0.02631578947368421,\n",
              " 'Ruth': 0.02631578947368421,\n",
              " 'Esther': 0.02631578947368421,\n",
              " 'née': 0.02631578947368421,\n",
              " 'Partington': 0.02631578947368421,\n",
              " 'family': 0.02631578947368421,\n",
              " 'moved': 0.10526315789473684,\n",
              " 'Michigan': 0.05263157894736842,\n",
              " 'graduated': 0.05263157894736842,\n",
              " 'Hancock': 0.02631578947368421,\n",
              " 'High': 0.02631578947368421,\n",
              " 'School': 0.02631578947368421,\n",
              " '1954': 0.02631578947368421,\n",
              " 'studied': 0.02631578947368421,\n",
              " 'mathematics': 0.15789473684210525,\n",
              " 'University': 0.02631578947368421,\n",
              " '1955': 0.02631578947368421,\n",
              " 'transferring': 0.02631578947368421,\n",
              " 'Earlham': 0.07894736842105263,\n",
              " 'College': 0.02631578947368421,\n",
              " 'mother': 0.02631578947368421,\n",
              " 'student': 0.02631578947368421,\n",
              " 'earned': 0.05263157894736842,\n",
              " 'BA': 0.02631578947368421,\n",
              " 'minor': 0.05263157894736842,\n",
              " 'philosophy': 0.05263157894736842,\n",
              " '1958': 0.05263157894736842,\n",
              " 'cites': 0.02631578947368421,\n",
              " 'Florence': 0.02631578947368421,\n",
              " 'Long': 0.02631578947368421,\n",
              " 'head': 0.02631578947368421,\n",
              " 'math': 0.02631578947368421,\n",
              " 'department': 0.05263157894736842,\n",
              " 'helping': 0.05263157894736842,\n",
              " 'desire': 0.02631578947368421,\n",
              " 'pursue': 0.02631578947368421,\n",
              " 'abstract': 0.05263157894736842,\n",
              " 'professor': 0.02631578947368421,\n",
              " 'says': 0.02631578947368421,\n",
              " 'poet': 0.02631578947368421,\n",
              " 'father': 0.02631578947368421,\n",
              " 'headmaster': 0.02631578947368421,\n",
              " 'grandfather': 0.02631578947368421,\n",
              " 'inspired': 0.02631578947368421,\n",
              " 'include': 0.05263157894736842,\n",
              " 'studies': 0.02631578947368421,\n",
              " 'Boston': 0.07894736842105263,\n",
              " 'initially': 0.05263157894736842,\n",
              " 'intended': 0.02631578947368421,\n",
              " 'enroll': 0.02631578947368421,\n",
              " 'graduate': 0.02631578947368421,\n",
              " 'study': 0.02631578947368421,\n",
              " 'Brandeis': 0.02631578947368421,\n",
              " 'mid-1959': 0.02631578947368421,\n",
              " 'began': 0.02631578947368421,\n",
              " 'working': 0.05263157894736842,\n",
              " 'Edward': 0.02631578947368421,\n",
              " 'Norton': 0.02631578947368421,\n",
              " 'Lorenz': 0.05263157894736842,\n",
              " 'meteorology': 0.02631578947368421,\n",
              " 'predicting': 0.02631578947368421,\n",
              " 'weather': 0.05263157894736842,\n",
              " 'programming': 0.13157894736842105,\n",
              " 'LGP-30': 0.02631578947368421,\n",
              " 'PDP-1': 0.02631578947368421,\n",
              " 'computers': 0.02631578947368421,\n",
              " 'Marvin': 0.02631578947368421,\n",
              " 'Minsky': 0.02631578947368421,\n",
              " 'Project': 0.10526315789473684,\n",
              " 'MAC': 0.02631578947368421,\n",
              " 'contributed': 0.02631578947368421,\n",
              " 'publications': 0.02631578947368421,\n",
              " 'chaos': 0.02631578947368421,\n",
              " 'theory': 0.02631578947368421,\n",
              " 'time': 0.21052631578947367,\n",
              " 'science': 0.07894736842105263,\n",
              " 'established': 0.02631578947368421,\n",
              " 'disciplines': 0.02631578947368421,\n",
              " 'instead': 0.02631578947368421,\n",
              " 'programmers': 0.07894736842105263,\n",
              " 'learned': 0.02631578947368421,\n",
              " 'job': 0.02631578947368421,\n",
              " 'hands': 0.05263157894736842,\n",
              " 'experience': 0.07894736842105263,\n",
              " 'project': 0.13157894736842105,\n",
              " 'summer': 0.05263157894736842,\n",
              " '1961': 0.05263157894736842,\n",
              " 'hired': 0.05263157894736842,\n",
              " 'trained': 0.02631578947368421,\n",
              " 'Ellen': 0.02631578947368421,\n",
              " 'Fetter': 0.02631578947368421,\n",
              " 'replacement': 0.02631578947368421,\n",
              " '1963': 0.02631578947368421,\n",
              " 'worked': 0.10526315789473684,\n",
              " 'Semi': 0.02631578947368421,\n",
              " 'Automatic': 0.02631578947368421,\n",
              " 'Ground': 0.02631578947368421,\n",
              " 'Environment': 0.02631578947368421,\n",
              " 'SAGE': 0.07894736842105263,\n",
              " 'Lincoln': 0.02631578947368421,\n",
              " 'Lab': 0.02631578947368421,\n",
              " 'wrote': 0.10526315789473684,\n",
              " 'prototype': 0.02631578947368421,\n",
              " 'FSQ-7': 0.02631578947368421,\n",
              " 'XD-1': 0.02631578947368421,\n",
              " 'U.S.': 0.05263157894736842,\n",
              " 'Air': 0.07894736842105263,\n",
              " 'Force': 0.07894736842105263,\n",
              " 'search': 0.02631578947368421,\n",
              " 'possibly': 0.02631578947368421,\n",
              " 'unfriendly': 0.02631578947368421,\n",
              " 'aircraft': 0.05263157894736842,\n",
              " 'satellite': 0.02631578947368421,\n",
              " 'tracking': 0.02631578947368421,\n",
              " 'Research': 0.02631578947368421,\n",
              " 'Laboratories': 0.02631578947368421,\n",
              " 'extension': 0.02631578947368421,\n",
              " 'Whirlwind': 0.02631578947368421,\n",
              " 'started': 0.05263157894736842,\n",
              " 'create': 0.05263157894736842,\n",
              " 'system': 0.13157894736842105,\n",
              " 'predict': 0.02631578947368421,\n",
              " 'track': 0.02631578947368421,\n",
              " 'movements': 0.02631578947368421,\n",
              " 'simulators': 0.02631578947368421,\n",
              " 'soon': 0.02631578947368421,\n",
              " 'military': 0.02631578947368421,\n",
              " 'use': 0.02631578947368421,\n",
              " 'anti': 0.02631578947368421,\n",
              " 'air': 0.02631578947368421,\n",
              " 'defense': 0.02631578947368421,\n",
              " 'said': 0.02631578947368421,\n",
              " 'came': 0.07894736842105263,\n",
              " 'organization': 0.02631578947368421,\n",
              " 'beginner': 0.05263157894736842,\n",
              " 'assign': 0.02631578947368421,\n",
              " 'able': 0.05263157894736842,\n",
              " 'figure': 0.02631578947368421,\n",
              " 'run': 0.02631578947368421,\n",
              " 'gave': 0.02631578947368421,\n",
              " 'happened': 0.02631578947368421,\n",
              " 'tricky': 0.02631578947368421,\n",
              " 'person': 0.02631578947368421,\n",
              " 'took': 0.02631578947368421,\n",
              " 'delight': 0.02631578947368421,\n",
              " 'fact': 0.05263157894736842,\n",
              " 'comments': 0.02631578947368421,\n",
              " 'Greek': 0.05263157894736842,\n",
              " 'Latin': 0.05263157894736842,\n",
              " 'assigned': 0.02631578947368421,\n",
              " 'actually': 0.02631578947368421,\n",
              " 'got': 0.02631578947368421,\n",
              " 'printed': 0.02631578947368421,\n",
              " 'answers': 0.02631578947368421,\n",
              " 'efforts': 0.02631578947368421,\n",
              " 'candidate': 0.02631578947368421,\n",
              " 'position': 0.02631578947368421,\n",
              " 'lead': 0.02631578947368421,\n",
              " 'developer': 0.02631578947368421,\n",
              " 'joined': 0.02631578947368421,\n",
              " 'Charles': 0.02631578947368421,\n",
              " 'Stark': 0.02631578947368421,\n",
              " 'Draper': 0.02631578947368421,\n",
              " 'Space': 0.05263157894736842,\n",
              " 'Mission': 0.02631578947368421,\n",
              " 'programmer': 0.02631578947368421,\n",
              " 'process': 0.10526315789473684,\n",
              " 'design': 0.13157894736842105,\n",
              " '1965': 0.02631578947368421,\n",
              " 'charge': 0.02631578947368421,\n",
              " 'Command': 0.02631578947368421,\n",
              " 'Module': 0.02631578947368421,\n",
              " 'navigation': 0.02631578947368421,\n",
              " 'lunar': 0.07894736842105263,\n",
              " 'landing': 0.13157894736842105,\n",
              " 'guidance': 0.05263157894736842,\n",
              " 'team': 0.07894736842105263,\n",
              " 'responsible': 0.02631578947368421,\n",
              " 'developing': 0.02631578947368421,\n",
              " 'included': 0.07894736842105263,\n",
              " 'algorithms': 0.02631578947368421,\n",
              " 'designed': 0.10526315789473684,\n",
              " 'senior': 0.05263157894736842,\n",
              " 'scientists': 0.02631578947368421,\n",
              " 'command': 0.02631578947368421,\n",
              " 'module': 0.05263157894736842,\n",
              " 'subsequent': 0.02631578947368421,\n",
              " 'Skylab': 0.02631578947368421,\n",
              " 'error': 0.13157894736842105,\n",
              " 'detection': 0.07894736842105263,\n",
              " 'recovery': 0.13157894736842105,\n",
              " 'restarts': 0.02631578947368421,\n",
              " 'Display': 0.05263157894736842,\n",
              " 'Interface': 0.05263157894736842,\n",
              " 'Routines': 0.05263157894736842,\n",
              " 'known': 0.02631578947368421,\n",
              " 'Priority': 0.02631578947368421,\n",
              " 'Displays': 0.02631578947368421,\n",
              " 'gain': 0.02631578947368421,\n",
              " 'courses': 0.05263157894736842,\n",
              " 'uncommon': 0.02631578947368421,\n",
              " 'exist': 0.02631578947368421,\n",
              " 'served': 0.02631578947368421,\n",
              " 'Director': 0.02631578947368421,\n",
              " 'areas': 0.02631578947368421,\n",
              " 'expertise': 0.02631578947368421,\n",
              " 'enterprise': 0.02631578947368421,\n",
              " 'modeling': 0.10526315789473684,\n",
              " 'paradigm': 0.05263157894736842,\n",
              " 'formal': 0.02631578947368421,\n",
              " 'languages': 0.02631578947368421,\n",
              " 'oriented': 0.02631578947368421,\n",
              " 'objects': 0.02631578947368421,\n",
              " 'automated': 0.07894736842105263,\n",
              " 'life': 0.07894736842105263,\n",
              " 'cycle': 0.07894736842105263,\n",
              " 'environments': 0.02631578947368421,\n",
              " 'methods': 0.02631578947368421,\n",
              " 'maximizing': 0.02631578947368421,\n",
              " 'reliability': 0.02631578947368421,\n",
              " 'reuse': 0.02631578947368421,\n",
              " 'domain': 0.02631578947368421,\n",
              " 'analysis': 0.02631578947368421,\n",
              " 'correctness': 0.02631578947368421,\n",
              " 'built': 0.02631578947368421,\n",
              " 'language': 0.10526315789473684,\n",
              " 'properties': 0.02631578947368421,\n",
              " 'open': 0.05263157894736842,\n",
              " 'architecture': 0.02631578947368421,\n",
              " 'techniques': 0.15789473684210525,\n",
              " 'robust': 0.02631578947368421,\n",
              " 'automation': 0.02631578947368421,\n",
              " 'quality': 0.02631578947368421,\n",
              " 'assurance': 0.02631578947368421,\n",
              " 'seamless': 0.02631578947368421,\n",
              " 'integration': 0.02631578947368421,\n",
              " 'human': 0.02631578947368421,\n",
              " 'machine': 0.02631578947368421,\n",
              " 'interface': 0.02631578947368421,\n",
              " 'operating': 0.02631578947368421,\n",
              " 'end': 0.05263157894736842,\n",
              " 'testing': 0.02631578947368421,\n",
              " 'management': 0.02631578947368421,\n",
              " 'code': 0.05263157894736842,\n",
              " 'incredibly': 0.02631578947368421,\n",
              " 'reliable': 0.07894736842105263,\n",
              " 'helped': 0.02631578947368421,\n",
              " 'identify': 0.02631578947368421,\n",
              " 'fix': 0.02631578947368421,\n",
              " 'anomalies': 0.02631578947368421,\n",
              " 'problems': 0.02631578947368421,\n",
              " 'critical': 0.05263157894736842,\n",
              " 'moments': 0.02631578947368421,\n",
              " '11': 0.07894736842105263,\n",
              " 'mission': 0.07894736842105263,\n",
              " 'Guidance': 0.02631578947368421,\n",
              " 'Computer': 0.05263157894736842,\n",
              " 'averted': 0.02631578947368421,\n",
              " 'abort': 0.02631578947368421,\n",
              " 'minutes': 0.02631578947368421,\n",
              " 'lander': 0.05263157894736842,\n",
              " 'reached': 0.02631578947368421,\n",
              " 'surface': 0.02631578947368421,\n",
              " 'alarms': 0.15789473684210525,\n",
              " 'triggered': 0.05263157894736842,\n",
              " 'captured': 0.02631578947368421,\n",
              " 'supposed': 0.02631578947368421,\n",
              " 'happen': 0.02631578947368421,\n",
              " 'displays': 0.23684210526315788,\n",
              " 'interrupting': 0.02631578947368421,\n",
              " 'astronauts': 0.18421052631578946,\n",
              " 'priority': 0.2894736842105263,\n",
              " 'alarm': 0.07894736842105263,\n",
              " 'prepared': 0.02631578947368421,\n",
              " 'situation': 0.02631578947368421,\n",
              " 'years': 0.07894736842105263,\n",
              " 'failsafe': 0.02631578947368421,\n",
              " 'likes': 0.02631578947368421,\n",
              " 'remember': 0.02631578947368421,\n",
              " 'display': 0.07894736842105263,\n",
              " 'innovation': 0.02631578947368421,\n",
              " 'created': 0.07894736842105263,\n",
              " 'knock': 0.02631578947368421,\n",
              " 'risk': 0.02631578947368421,\n",
              " 'astronaut': 0.05263157894736842,\n",
              " 'slip': 0.02631578947368421,\n",
              " 'synch': 0.02631578947368421,\n",
              " 'mattered': 0.02631578947368421,\n",
              " 'went': 0.02631578947368421,\n",
              " 'replaced': 0.02631578947368421,\n",
              " 'normal': 0.10526315789473684,\n",
              " 'ones': 0.07894736842105263,\n",
              " 'actual': 0.02631578947368421,\n",
              " 'switchover': 0.02631578947368421,\n",
              " 'new': 0.05263157894736842,\n",
              " 'programmes': 0.02631578947368421,\n",
              " 'screens': 0.02631578947368421,\n",
              " 'happening': 0.02631578947368421,\n",
              " 'step': 0.02631578947368421,\n",
              " 'slower': 0.02631578947368421,\n",
              " 'today': 0.02631578947368421,\n",
              " 'thought': 0.02631578947368421,\n",
              " 'long': 0.05263157894736842,\n",
              " 'hard': 0.02631578947368421,\n",
              " 'meant': 0.05263157894736842,\n",
              " 'Aldrin': 0.02631578947368421,\n",
              " 'hit': 0.02631578947368421,\n",
              " 'button': 0.02631578947368421,\n",
              " 'quickly': 0.02631578947368421,\n",
              " 'response': 0.02631578947368421,\n",
              " 'solution': 0.02631578947368421,\n",
              " 'count': 0.02631578947368421,\n",
              " 'accounts': 0.02631578947368421,\n",
              " 'inadvertently': 0.05263157894736842,\n",
              " 'left': 0.07894736842105263,\n",
              " 'rendezvous': 0.05263157894736842,\n",
              " 'radar': 0.07894736842105263,\n",
              " 'switch': 0.02631578947368421,\n",
              " 'causing': 0.02631578947368421,\n",
              " 'claim': 0.02631578947368421,\n",
              " 'disputed': 0.02631578947368421,\n",
              " 'Robert': 0.05263157894736842,\n",
              " 'Wills': 0.02631578947368421,\n",
              " 'National': 0.02631578947368421,\n",
              " 'Museum': 0.02631578947368421,\n",
              " 'Computing': 0.02631578947368421,\n",
              " 'overloaded': 0.05263157894736842,\n",
              " 'interrupts': 0.02631578947368421,\n",
              " 'caused': 0.02631578947368421,\n",
              " 'incorrectly': 0.02631578947368421,\n",
              " 'phased': 0.02631578947368421,\n",
              " 'power': 0.02631578947368421,\n",
              " 'supplied': 0.02631578947368421,\n",
              " 'indicated': 0.02631578947368421,\n",
              " 'executive': 0.05263157894736842,\n",
              " 'overflows': 0.02631578947368421,\n",
              " 'meaning': 0.05263157894736842,\n",
              " 'complete': 0.05263157894736842,\n",
              " 'tasks': 0.13157894736842105,\n",
              " 'real': 0.05263157894736842,\n",
              " 'postpone': 0.02631578947368421,\n",
              " 'asynchronous': 0.05263157894736842,\n",
              " 'J.': 0.02631578947368421,\n",
              " 'Halcombe': 0.02631578947368421,\n",
              " 'Laning': 0.02631578947368421,\n",
              " 'develop': 0.05263157894736842,\n",
              " 'wide': 0.02631578947368421,\n",
              " 'kill': 0.02631578947368421,\n",
              " 'recompute': 0.02631578947368421,\n",
              " 'safe': 0.02631578947368421,\n",
              " 'place': 0.05263157894736842,\n",
              " 'restart': 0.02631578947368421,\n",
              " 'approach': 0.02631578947368421,\n",
              " 'snapshot': 0.02631578947368421,\n",
              " 'rollback': 0.02631578947368421,\n",
              " 'AKA': 0.02631578947368421,\n",
              " 'man': 0.02631578947368421,\n",
              " 'loop': 0.02631578947368421,\n",
              " 'capabilities': 0.02631578947368421,\n",
              " 'order': 0.07894736842105263,\n",
              " 'capability': 0.02631578947368421,\n",
              " 'interrupt': 0.02631578947368421,\n",
              " 'case': 0.05263157894736842,\n",
              " 'emergency': 0.05263157894736842,\n",
              " 'depended': 0.02631578947368421,\n",
              " 'assigning': 0.02631578947368421,\n",
              " 'unique': 0.02631578947368421,\n",
              " 'ensure': 0.02631578947368421,\n",
              " 'events': 0.02631578947368421,\n",
              " 'correct': 0.02631578947368421,\n",
              " 'right': 0.05263157894736842,\n",
              " 'relative': 0.02631578947368421,\n",
              " 'going': 0.05263157894736842,\n",
              " 'interrupted': 0.02631578947368421,\n",
              " 'warn': 0.02631578947368421,\n",
              " 'giving': 0.02631578947368421,\n",
              " 'decision': 0.02631578947368421,\n",
              " 'land': 0.05263157894736842,\n",
              " 'Jack': 0.02631578947368421,\n",
              " 'Garman': 0.02631578947368421,\n",
              " 'control': 0.02631578947368421,\n",
              " 'recognized': 0.05263157894736842,\n",
              " 'errors': 0.02631578947368421,\n",
              " 'presented': 0.02631578947368421,\n",
              " 'shouted': 0.02631578947368421,\n",
              " 'continued': 0.02631578947368421,\n",
              " 'Paul': 0.02631578947368421,\n",
              " 'Curto': 0.02631578947368421,\n",
              " 'technologist': 0.02631578947368421,\n",
              " 'nominated': 0.02631578947368421,\n",
              " 'Act': 0.02631578947368421,\n",
              " 'Award': 0.02631578947368421,\n",
              " 'called': 0.07894736842105263,\n",
              " 'foundation': 0.02631578947368421,\n",
              " 'ultra': 0.02631578947368421,\n",
              " 'incident': 0.02631578947368421,\n",
              " 'smart': 0.02631578947368421,\n",
              " 'recognize': 0.05263157894736842,\n",
              " 'asked': 0.02631578947368421,\n",
              " 'perform': 0.02631578947368421,\n",
              " 'performing': 0.02631578947368421,\n",
              " 'sent': 0.02631578947368421,\n",
              " 'important': 0.07894736842105263,\n",
              " 'i.e.': 0.02631578947368421,\n",
              " 'needed': 0.02631578947368421,\n",
              " '\\xa0': 0.05263157894736842,\n",
              " '...': 0.05263157894736842,\n",
              " 'Actually': 0.02631578947368421,\n",
              " 'programmed': 0.02631578947368421,\n",
              " 'conditions': 0.02631578947368421,\n",
              " 'set': 0.02631578947368421,\n",
              " 'incorporated': 0.02631578947368421,\n",
              " 'action': 0.05263157894736842,\n",
              " 'eliminate': 0.02631578947368421,\n",
              " 'lower': 0.02631578947368421,\n",
              " 'establish': 0.02631578947368421,\n",
              " 'problem': 0.02631578947368421,\n",
              " 'taken': 0.05263157894736842,\n",
              " 'doubt': 0.02631578947368421,\n",
              " 'successful': 0.02631578947368421,\n",
              " 'moon': 0.02631578947368421,\n",
              " 'co': 0.02631578947368421,\n",
              " 'Saydean': 0.02631578947368421,\n",
              " 'Zeldin': 0.02631578947368421,\n",
              " 'company': 0.07894736842105263,\n",
              " 'HOS': 0.13157894736842105,\n",
              " 'ideas': 0.05263157894736842,\n",
              " 'prevention': 0.02631578947368421,\n",
              " 'fault': 0.02631578947368421,\n",
              " 'tolerance': 0.02631578947368421,\n",
              " 'emerging': 0.02631578947368421,\n",
              " 'product': 0.02631578947368421,\n",
              " 'USE.IT': 0.02631578947368421,\n",
              " 'based': 0.05263157894736842,\n",
              " 'methodology': 0.02631578947368421,\n",
              " 'successfully': 0.02631578947368421,\n",
              " 'numerous': 0.02631578947368421,\n",
              " 'government': 0.02631578947368421,\n",
              " 'including': 0.02631578947368421,\n",
              " 'formalize': 0.07894736842105263,\n",
              " 'implement': 0.02631578947368421,\n",
              " 'C': 0.02631578947368421,\n",
              " 'IDEF': 0.05263157894736842,\n",
              " 'version': 0.02631578947368421,\n",
              " 'Integrated': 0.02631578947368421,\n",
              " 'Aided': 0.02631578947368421,\n",
              " 'Manufacturing': 0.02631578947368421,\n",
              " 'ICAM': 0.02631578947368421,\n",
              " '1980': 0.02631578947368421,\n",
              " 'British': 0.02631578947368421,\n",
              " 'Israeli': 0.02631578947368421,\n",
              " 'David': 0.02631578947368421,\n",
              " 'Harel': 0.02631578947368421,\n",
              " 'proposal': 0.02631578947368421,\n",
              " 'structured': 0.02631578947368421,\n",
              " 'derived': 0.02631578947368421,\n",
              " 'viewpoint': 0.02631578947368421,\n",
              " 'and/or': 0.02631578947368421,\n",
              " 'subgoals': 0.02631578947368421,\n",
              " 'semantics': 0.02631578947368421,\n",
              " 'linguistic': 0.02631578947368421,\n",
              " 'quantifiers': 0.02631578947368421,\n",
              " 'embedded': 0.02631578947368421,\n",
              " 'CEO': 0.02631578947368421,\n",
              " '1984': 0.02631578947368421,\n",
              " '1985': 0.02631578947368421,\n",
              " 'March': 0.02631578947368421,\n",
              " 'Inc.': 0.02631578947368421,\n",
              " 'Universal': 0.02631578947368421,\n",
              " 'Systems': 0.02631578947368421,\n",
              " 'Language': 0.02631578947368421,\n",
              " 'USL': 0.05263157894736842,\n",
              " 'associated': 0.02631578947368421,\n",
              " 'environment': 0.02631578947368421,\n",
              " '001': 0.02631578947368421,\n",
              " 'Tool': 0.02631578947368421,\n",
              " 'Suite': 0.02631578947368421,\n",
              " 'Anthony': 0.02631578947368421,\n",
              " 'Oettinger': 0.02631578947368421,\n",
              " 'Barry': 0.02631578947368421,\n",
              " 'Boehm': 0.02631578947368421,\n",
              " 'naming': 0.02631578947368421,\n",
              " 'discipline': 0.10526315789473684,\n",
              " 'details': 0.02631578947368421,\n",
              " 'heard': 0.02631578947368421,\n",
              " 'world': 0.05263157894736842,\n",
              " 'ongoing': 0.02631578947368421,\n",
              " 'joke': 0.02631578947368421,\n",
              " 'liked': 0.02631578947368421,\n",
              " 'kid': 0.02631578947368421,\n",
              " 'radical': 0.02631578947368421,\n",
              " 'memorable': 0.02631578947368421,\n",
              " 'day': 0.02631578947368421,\n",
              " 'respected': 0.02631578947368421,\n",
              " 'hardware': 0.05263157894736842,\n",
              " 'gurus': 0.02631578947368421,\n",
              " 'explained': 0.02631578947368421,\n",
              " 'meeting': 0.02631578947368421,\n",
              " 'agreed': 0.02631578947368421,\n",
              " 'building': 0.02631578947368421,\n",
              " 'considered': 0.02631578947368421,\n",
              " 'like': 0.05263157894736842,\n",
              " 'acceptance': 0.05263157894736842,\n",
              " 'se': 0.02631578947368421,\n",
              " 'room': 0.02631578947368421,\n",
              " 'field': 0.02631578947368421,\n",
              " 'early': 0.05263157894736842,\n",
              " 'seriously': 0.02631578947368421,\n",
              " 'compared': 0.02631578947368421,\n",
              " 'regarded': 0.02631578947368421,\n",
              " 'concerned': 0.02631578947368421,\n",
              " 'legitimizing': 0.02631578947368421,\n",
              " 'gained': 0.02631578947368421,\n",
              " 'respect': 0.02631578947368421,\n",
              " 'technical': 0.02631578947368421,\n",
              " 'IEEE': 0.02631578947368421,\n",
              " 'September': 0.02631578947368421,\n",
              " 'October': 0.02631578947368421,\n",
              " '2018': 0.02631578947368421,\n",
              " 'issue': 0.02631578947368421,\n",
              " 'celebrates': 0.02631578947368421,\n",
              " '50th': 0.02631578947368421,\n",
              " 'anniversary': 0.02631578947368421,\n",
              " 'talks': 0.02631578947368421,\n",
              " 'Errors': 0.05263157894736842,\n",
              " 'influenced': 0.02631578947368421,\n",
              " 'related': 0.02631578947368421,\n",
              " 'prevent': 0.02631578947368421,\n",
              " 'majority': 0.02631578947368421,\n",
              " 'Writing': 0.02631578947368421,\n",
              " 'Wired': 0.05263157894736842,\n",
              " 'McMillan': 0.02631578947368421,\n",
              " 'noted': 0.02631578947368421,\n",
              " 'assisted': 0.02631578947368421,\n",
              " 'creation': 0.02631578947368421,\n",
              " 'core': 0.02631578947368421,\n",
              " 'principles': 0.02631578947368421,\n",
              " 'colleagues': 0.02631578947368421,\n",
              " 'writing': 0.02631578947368421,\n",
              " 'portable': 0.02631578947368421,\n",
              " 'innovations': 0.02631578947368421,\n",
              " 'feats': 0.02631578947368421,\n",
              " 'playing': 0.02631578947368421,\n",
              " 'role': 0.02631578947368421,\n",
              " 'getting': 0.02631578947368421,\n",
              " 'humans': 0.02631578947368421,\n",
              " 'According': 0.02631578947368421,\n",
              " 'Karen': 0.02631578947368421,\n",
              " 'Tegan': 0.02631578947368421,\n",
              " 'Padir': 0.02631578947368421,\n",
              " 'pioneer': 0.02631578947368421,\n",
              " 'COBOL': 0.02631578947368421,\n",
              " 'inventor': 0.02631578947368421,\n",
              " 'Grace': 0.02631578947368421,\n",
              " 'Hopper': 0.02631578947368421,\n",
              " 'deserve': 0.02631578947368421,\n",
              " 'tremendous': 0.02631578947368421,\n",
              " 'credit': 0.02631578947368421,\n",
              " 'door': 0.02631578947368421,\n",
              " 'women': 0.02631578947368421,\n",
              " 'enter': 0.02631578947368421,\n",
              " 'succeed': 0.02631578947368421,\n",
              " 'STEM': 0.02631578947368421,\n",
              " 'fields': 0.02631578947368421,\n",
              " '2019': 0.02631578947368421,\n",
              " 'celebrate': 0.02631578947368421,\n",
              " '50': 0.02631578947368421,\n",
              " 'Google': 0.02631578947368421,\n",
              " 'decided': 0.02631578947368421,\n",
              " 'tribute': 0.02631578947368421,\n",
              " 'mirrors': 0.02631578947368421,\n",
              " 'Ivanpah': 0.02631578947368421,\n",
              " 'plant': 0.02631578947368421,\n",
              " 'configured': 0.02631578947368421,\n",
              " 'picture': 0.02631578947368421,\n",
              " 'moonlight': 0.02631578947368421,\n",
              " 'sister': 0.02631578947368421,\n",
              " 'Kathryn': 0.02631578947368421,\n",
              " 'met': 0.02631578947368421,\n",
              " 'husband': 0.02631578947368421,\n",
              " 'James': 0.02631578947368421,\n",
              " 'Cox': 0.02631578947368421,\n",
              " 'mid-1950s': 0.02631578947368421,\n",
              " 'attending': 0.02631578947368421,\n",
              " 'college': 0.02631578947368421,\n",
              " 'married': 0.05263157894736842,\n",
              " 'June': 0.02631578947368421,\n",
              " '15': 0.02631578947368421,\n",
              " 'briefly': 0.02631578947368421,\n",
              " 'taught': 0.02631578947368421,\n",
              " 'high': 0.02631578947368421,\n",
              " 'school': 0.05263157894736842,\n",
              " 'French': 0.02631578947368421,\n",
              " 'public': 0.02631578947368421,\n",
              " 'couple': 0.02631578947368421,\n",
              " 'daughter': 0.02631578947368421,\n",
              " 'Lauren': 0.02631578947368421,\n",
              " '10': 0.02631578947368421,\n",
              " '1959': 0.02631578947368421,\n",
              " 'divorced': 0.02631578947368421,\n",
              " '1967': 0.02631578947368421,\n",
              " 'Dan': 0.02631578947368421,\n",
              " 'Lickly': 0.02631578947368421}"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# sentence tokens\n",
        "sentence_tokens= [sent for sent in doc.sents]\n",
        "\n",
        "sentence_tokens"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ama_gtLhp2Ny",
        "outputId": "4c54e459-57a9-4f9b-cd12-1108e41726bc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner.,\n",
              " She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program.,\n",
              " She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.,\n",
              " Hamilton has published more than 130 papers, proceedings, and reports, about sixty projects, and six major programs.,\n",
              " She is one of the people credited with coining the term \"software engineering\".,\n",
              " On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA's Apollo Moon missions.,\n",
              " Margaret Elaine Heafield was born August 17, 1936, in Paoli, Indiana, to Kenneth Heafield and Ruth Esther Heafield (née Partington).,\n",
              " The family later moved to Michigan, where Margaret graduated from Hancock High School in 1954.,\n",
              " She studied mathematics at the University of Michigan in 1955 before transferring to Earlham College, where her mother was a student; she earned a BA in mathematics with a minor in philosophy in 1958.,\n",
              " She cites Florence Long, the head of the math department at Earlham, as helping with her desire to pursue abstract mathematics and become a mathematics professor.,\n",
              " She says her poet father and headmaster grandfather inspired her to include a minor in philosophy in her studies.,\n",
              " In Boston, Hamilton initially intended to enroll in graduate study in abstract mathematics at Brandeis.,\n",
              " However, in mid-1959, Hamilton began working for Edward Norton Lorenz, in the meteorology department at MIT.,\n",
              " She developed software for predicting weather, programming on the LGP-30 and the PDP-1 computers at Marvin Minsky's Project MAC.,\n",
              " Her work contributed to Lorenz's publications on chaos theory.,\n",
              " At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience.,\n",
              " She moved on to another project in the summer of 1961, and hired and trained Ellen Fetter as her replacement.,\n",
              " From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft.,\n",
              " She also wrote software for a satellite tracking project at the Air Force Cambridge Research Laboratories.,\n",
              " The SAGE Project was an extension of Project Whirlwind, started by MIT to create a computer system that could predict weather systems and track their movements using simulators.,\n",
              " SAGE was soon developed for military use in anti-aircraft air defense.,\n",
              " Hamilton said:\n",
              " What they used to do when you came into this organization as a beginner, was to assign you this program which nobody was able to ever figure out or get to run.,\n",
              " When I was the beginner they gave it to me as well.,\n",
              " And what had happened was it was tricky programming, and the person who wrote it took delight in the fact that all of his comments were in Greek and Latin.,\n",
              " So I was assigned this program and I actually got it to work.,\n",
              " It even printed out its answers in Latin and Greek.,\n",
              " I was the first one to get it to work.,\n",
              " It was her efforts on this project that made her a candidate for the position at NASA as the lead developer for Apollo flight software.,\n",
              " Hamilton then joined the Charles Stark Draper Laboratory at MIT, which worked on the Apollo Space Mission.,\n",
              " Hamilton was initially hired as a programmer for this process but moved on into systems design.,\n",
              " By 1965, she was in charge of all Command Module software, which was all the software for navigation and lunar landing guidance.,\n",
              " Hamilton's team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab.,\n",
              " Another part of her team designed and developed the systems software.,\n",
              " This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed.,\n",
              " She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist.,\n",
              " Hamilton also served as Director of the Software Engineering Division.,\n",
              " Her areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques.,\n",
              " These made her code incredibly reliable because they helped programmers identify and fix anomalies before they became major problems.,\n",
              " In one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon.,\n",
              " Three minutes before the lunar lander reached the Moon's surface, several computer alarms were triggered.,\n",
              " The on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.,\n",
              " Hamilton had prepared for just this situation years before:\n",
              " There was one other failsafe that Hamilton likes to remember.,\n",
              " Her \"priority display\" innovation had created a knock-on risk that astronaut and computer would slip out of synch just when it mattered most.,\n",
              " As the alarms went off and priority displays replaced normal ones, the actual switchover to new programmes behind the screens was happening \"a step slower\" than it would today.,\n",
              " Hamilton had thought long and hard about this.,\n",
              " It meant that if Aldrin, say, hit a button on the priority display too quickly, he might still get a \"normal\" response.,\n",
              " Her solution: when you see a priority display, first count to\n",
              " five.,\n",
              " By some accounts, the astronauts had inadvertently left the rendezvous radar switch on, causing these alarms to be triggered (the claim that the radar was left on inadvertently by the astronauts is disputed by Robert Wills with the National Museum of Computing).,\n",
              " The computer was overloaded with interrupts caused by incorrectly phased power supplied to the lander's rendezvous radar.,\n",
              " The program alarms indicated \"executive overflows\", meaning the guidance computer could not complete all of its tasks in real time and had to postpone some of them.,\n",
              " The asynchronous executive designed by J. Halcombe Laning was used by Hamilton's team to develop asynchronous flight software:\n",
              " Because of the flight software's system-software's error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts' normal mission displays with priority displays of critical alarms in case of an emergency.,\n",
              " This depended on our assigning a unique priority to every process in the software in order to ensure that all of its events would take place in the correct order and at the right time relative to everything else that was going on.,\n",
              " Hamilton's priority alarm displays interrupted the astronauts' normal displays to warn them that there was an emergency \"giving the astronauts a go/no go decision (to land or not to land)\".,\n",
              " Jack Garman, a NASA computer engineer in mission control, recognized the meaning of the errors that were presented to the astronauts by the priority displays and shouted, \"Go, go!\",\n",
              " and they continued.,\n",
              " Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton's work \"the foundation for ultra-reliable software design\".\n",
              " Hamilton later wrote of the incident:\n",
              " The computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing.,\n",
              " It then sent out an alarm, which meant to the astronaut, 'I'm overloaded with more tasks than I should be doing at this time and I'm going to keep only the more important tasks'; i.e., the ones needed for landing ...,\n",
              " Actually, the computer was programmed to do more than recognize error conditions.,\n",
              " A complete set of recovery programs was incorporated into the software.,\n",
              " The software's action, in this case, was to eliminate lower priority tasks and re-establish the more important ones ...,\n",
              " If the computer hadn't recognized this problem and taken recovery action, I doubt if Apollo 11 would have been the successful moon landing it was.,\n",
              " In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.,\n",
              " They created a product called USE.IT, based on the HOS methodology they developed at MIT.,\n",
              " It was successfully used in numerous government programs including a project to formalize and implement C-IDEF, an automated version of IDEF, a modeling language developed by the U.S. Air Force in the Integrated Computer-Aided Manufacturing (ICAM) project.,\n",
              " In 1980, British-Israeli computer scientist David Harel published a proposal for a structured programming language derived from HOS from the viewpoint of and/or subgoals.,\n",
              " Others have used HOS to formalize the semantics of linguistic quantifiers, and to formalize the design of reliable real-time embedded systems.,\n",
              " Hamilton was the CEO of HOS through 1984 and left the company in 1985.,\n",
              " In March 1986, she founded Hamilton Technologies, Inc. in Cambridge, Massachusetts.,\n",
              " The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.,\n",
              " Anthony Oettinger, Barry Boehm, and Hamilton have been credited with naming the discipline of \"software engineering\".,\n",
              " Hamilton details how she came to make up the term \"software engineering\":\n",
              " When I first came up with the term, no one had heard of it before, at least in our world.,\n",
              " It was an ongoing joke for a long time.,\n",
              " They liked to kid me about my radical ideas.,\n",
              " It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware.,\n",
              " Not because of his acceptance of the new 'term' per se, but because we had earned his and the acceptance of the others in the room as being in an engineering field in its own right.,\n",
              " When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science.,\n",
              " Hamilton was concerned with legitimizing software development as an engineering discipline.,\n",
              " Over time the term \"software engineering\" gained the same respect as any other technical discipline.,\n",
              " The IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering.,\n",
              " Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system.,\n",
              " Writing in Wired, Robert McMillan noted: \"At MIT she assisted in the creation of the core principles in computer programming as she worked with her colleagues in writing code for the world's first portable computer\".,\n",
              " Hamilton's innovations go beyond the feats of playing an important role in getting humans to the Moon.,\n",
              " According to Wired's Karen Tegan Padir: \"She, along with that other early programming pioneer, COBOL inventor Grace Hopper, also deserve tremendous credit for helping to open the door for more women to enter and succeed in STEM fields like software.,\n",
              " \"\n",
              " In 2019, to celebrate 50 years to the Apollo landing, Google decided to make a tribute to Hamilton.,\n",
              " The mirrors at the Ivanpah plant were configured to create a picture of Hamilton and the Apollo 11 by moonlight.,\n",
              " Hamilton has a sister Kathryn.,\n",
              " She met her first husband, James Cox Hamilton, in the mid-1950s while attending college.,\n",
              " They were married on June 15, 1958, the summer after she graduated from Earlham.,\n",
              " She briefly taught high school mathematics and French at a public school in Boston, Indiana.,\n",
              " The couple then moved to Boston, Massachusetts, where they had a daughter, Lauren, born on November 10, 1959.,\n",
              " They divorced in 1967 and Margaret married Dan Lickly two years later.]"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# evaluate the sentences by giving them a score\n",
        "sentence_scores = {}\n",
        "for sent in sentence_tokens:\n",
        "    for word in sent:\n",
        "        if word.text.lower() in word_frequencies.keys():\n",
        "            if sent not in sentence_scores.keys():                            \n",
        "             sentence_scores[sent]=word_frequencies[word.text.lower()]\n",
        "            else:\n",
        "             sentence_scores[sent]+=word_frequencies[word.text.lower()]\n",
        "\n",
        "sentence_scores"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vjVlJW6BqP16",
        "outputId": "d91520d6-6b06-4b33-9ba8-106fccc636ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner.: 1.0789473684210529,\n",
              " She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program.: 3.026315789473684,\n",
              " She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.: 2.421052631578948,\n",
              " Hamilton has published more than 130 papers, proceedings, and reports, about sixty projects, and six major programs.: 0.3157894736842105,\n",
              " She is one of the people credited with coining the term \"software engineering\".: 1.6052631578947367,\n",
              " On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA's Apollo Moon missions.: 1.8947368421052633,\n",
              " Margaret Elaine Heafield was born August 17, 1936, in Paoli, Indiana, to Kenneth Heafield and Ruth Esther Heafield (née Partington).: 0.21052631578947367,\n",
              " The family later moved to Michigan, where Margaret graduated from Hancock High School in 1954.: 0.39473684210526316,\n",
              " She studied mathematics at the University of Michigan in 1955 before transferring to Earlham College, where her mother was a student; she earned a BA in mathematics with a minor in philosophy in 1958.: 0.6842105263157894,\n",
              " She cites Florence Long, the head of the math department at Earlham, as helping with her desire to pursue abstract mathematics and become a mathematics professor.: 0.6842105263157895,\n",
              " She says her poet father and headmaster grandfather inspired her to include a minor in philosophy in her studies.: 0.3421052631578947,\n",
              " In Boston, Hamilton initially intended to enroll in graduate study in abstract mathematics at Brandeis.: 0.3684210526315789,\n",
              " However, in mid-1959, Hamilton began working for Edward Norton Lorenz, in the meteorology department at MIT.: 0.18421052631578946,\n",
              " She developed software for predicting weather, programming on the LGP-30 and the PDP-1 computers at Marvin Minsky's Project MAC.: 1.5789473684210527,\n",
              " Her work contributed to Lorenz's publications on chaos theory.: 0.2631578947368421,\n",
              " At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience.: 2.394736842105263,\n",
              " She moved on to another project in the summer of 1961, and hired and trained Ellen Fetter as her replacement.: 0.4473684210526315,\n",
              " From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft.: 2.131578947368421,\n",
              " She also wrote software for a satellite tracking project at the Air Force Cambridge Research Laboratories.: 1.3157894736842106,\n",
              " The SAGE Project was an extension of Project Whirlwind, started by MIT to create a computer system that could predict weather systems and track their movements using simulators.: 1.4210526315789476,\n",
              " SAGE was soon developed for military use in anti-aircraft air defense.: 0.42105263157894735,\n",
              " Hamilton said:\n",
              " What they used to do when you came into this organization as a beginner, was to assign you this program which nobody was able to ever figure out or get to run.: 0.4473684210526315,\n",
              " When I was the beginner they gave it to me as well.: 0.07894736842105263,\n",
              " And what had happened was it was tricky programming, and the person who wrote it took delight in the fact that all of his comments were in Greek and Latin.: 0.4473684210526315,\n",
              " So I was assigned this program and I actually got it to work.: 0.3684210526315789,\n",
              " It even printed out its answers in Latin and Greek.: 0.05263157894736842,\n",
              " I was the first one to get it to work.: 0.15789473684210525,\n",
              " It was her efforts on this project that made her a candidate for the position at NASA as the lead developer for Apollo flight software.: 1.4736842105263157,\n",
              " Hamilton then joined the Charles Stark Draper Laboratory at MIT, which worked on the Apollo Space Mission.: 0.21052631578947367,\n",
              " Hamilton was initially hired as a programmer for this process but moved on into systems design.: 0.7894736842105262,\n",
              " By 1965, she was in charge of all Command Module software, which was all the software for navigation and lunar landing guidance.: 2.421052631578948,\n",
              " Hamilton's team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab.: 1.8684210526315792,\n",
              " Another part of her team designed and developed the systems software.: 1.7105263157894737,\n",
              " This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed.: 2.421052631578948,\n",
              " She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist.: 2.4736842105263155,\n",
              " Hamilton also served as Director of the Software Engineering Division.: 1.394736842105263,\n",
              " Her areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques.: 7.894736842105258,\n",
              " These made her code incredibly reliable because they helped programmers identify and fix anomalies before they became major problems.: 0.42105263157894735,\n",
              " In one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon.: 2.2368421052631575,\n",
              " Three minutes before the lunar lander reached the Moon's surface, several computer alarms were triggered.: 0.8684210526315788,\n",
              " The on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.: 2.605263157894737,\n",
              " Hamilton had prepared for just this situation years before:\n",
              " There was one other failsafe that Hamilton likes to remember.: 0.21052631578947367,\n",
              " Her \"priority display\" innovation had created a knock-on risk that astronaut and computer would slip out of synch just when it mattered most.: 1.0789473684210529,\n",
              " As the alarms went off and priority displays replaced normal ones, the actual switchover to new programmes behind the screens was happening \"a step slower\" than it would today.: 1.1842105263157898,\n",
              " Hamilton had thought long and hard about this.: 0.10526315789473684,\n",
              " It meant that if Aldrin, say, hit a button on the priority display too quickly, he might still get a \"normal\" response.: 0.631578947368421,\n",
              " Her solution: when you see a priority display, first count to\n",
              " five.: 0.42105263157894735,\n",
              " By some accounts, the astronauts had inadvertently left the rendezvous radar switch on, causing these alarms to be triggered (the claim that the radar was left on inadvertently by the astronauts is disputed by Robert Wills with the National Museum of Computing).: 1.1842105263157894,\n",
              " The computer was overloaded with interrupts caused by incorrectly phased power supplied to the lander's rendezvous radar.: 0.8157894736842104,\n",
              " The program alarms indicated \"executive overflows\", meaning the guidance computer could not complete all of its tasks in real time and had to postpone some of them.: 1.394736842105263,\n",
              " The asynchronous executive designed by J. Halcombe Laning was used by Hamilton's team to develop asynchronous flight software:\n",
              " Because of the flight software's system-software's error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts' normal mission displays with priority displays of critical alarms in case of an emergency.: 7.499999999999995,\n",
              " This depended on our assigning a unique priority to every process in the software in order to ensure that all of its events would take place in the correct order and at the right time relative to everything else that was going on.: 2.105263157894737,\n",
              " Hamilton's priority alarm displays interrupted the astronauts' normal displays to warn them that there was an emergency \"giving the astronauts a go/no go decision (to land or not to land)\".: 1.5789473684210527,\n",
              " Jack Garman, a NASA computer engineer in mission control, recognized the meaning of the errors that were presented to the astronauts by the priority displays and shouted, \"Go, go!\": 1.4736842105263157,\n",
              " and they continued.: 0.02631578947368421,\n",
              " Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton's work \"the foundation for ultra-reliable software design\".\n",
              " Hamilton later wrote of the incident:\n",
              " The computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing.: 3.5526315789473677,\n",
              " It then sent out an alarm, which meant to the astronaut, 'I'm overloaded with more tasks than I should be doing at this time and I'm going to keep only the more important tasks'; i.e., the ones needed for landing ...: 1.2368421052631577,\n",
              " Actually, the computer was programmed to do more than recognize error conditions.: 0.6842105263157893,\n",
              " A complete set of recovery programs was incorporated into the software.: 1.3157894736842106,\n",
              " The software's action, in this case, was to eliminate lower priority tasks and re-establish the more important ones ...: 1.868421052631579,\n",
              " If the computer hadn't recognized this problem and taken recovery action, I doubt if Apollo 11 would have been the successful moon landing it was.: 1.026315789473684,\n",
              " In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.: 2.0000000000000004,\n",
              " They created a product called USE.IT, based on the HOS methodology they developed at MIT.: 0.47368421052631576,\n",
              " It was successfully used in numerous government programs including a project to formalize and implement C-IDEF, an automated version of IDEF, a modeling language developed by the U.S. Air Force in the Integrated Computer-Aided Manufacturing (ICAM) project.: 1.526315789473684,\n",
              " In 1980, British-Israeli computer scientist David Harel published a proposal for a structured programming language derived from HOS from the viewpoint of and/or subgoals.: 0.9473684210526313,\n",
              " Others have used HOS to formalize the semantics of linguistic quantifiers, and to formalize the design of reliable real-time embedded systems.: 1.0526315789473684,\n",
              " Hamilton was the CEO of HOS through 1984 and left the company in 1985.: 0.21052631578947367,\n",
              " In March 1986, she founded Hamilton Technologies, Inc. in Cambridge, Massachusetts.: 0.13157894736842105,\n",
              " The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.: 2.894736842105263,\n",
              " Anthony Oettinger, Barry Boehm, and Hamilton have been credited with naming the discipline of \"software engineering\".: 1.526315789473684,\n",
              " Hamilton details how she came to make up the term \"software engineering\":\n",
              " When I first came up with the term, no one had heard of it before, at least in our world.: 1.9210526315789473,\n",
              " It was an ongoing joke for a long time.: 0.3157894736842105,\n",
              " They liked to kid me about my radical ideas.: 0.13157894736842105,\n",
              " It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware.: 1.9473684210526314,\n",
              " Not because of his acceptance of the new 'term' per se, but because we had earned his and the acceptance of the others in the room as being in an engineering field in its own right.: 0.8421052631578947,\n",
              " When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science.: 3.421052631578947,\n",
              " Hamilton was concerned with legitimizing software development as an engineering discipline.: 1.7105263157894735,\n",
              " Over time the term \"software engineering\" gained the same respect as any other technical discipline.: 1.8947368421052633,\n",
              " The IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering.: 2.473684210526316,\n",
              " Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system.: 1.9210526315789473,\n",
              " Writing in Wired, Robert McMillan noted: \"At MIT she assisted in the creation of the core principles in computer programming as she worked with her colleagues in writing code for the world's first portable computer\".: 1.4210526315789473,\n",
              " Hamilton's innovations go beyond the feats of playing an important role in getting humans to the Moon.: 0.2631578947368421,\n",
              " According to Wired's Karen Tegan Padir: \"She, along with that other early programming pioneer, COBOL inventor Grace Hopper, also deserve tremendous credit for helping to open the door for more women to enter and succeed in STEM fields like software.: 1.6052631578947367,\n",
              " \"\n",
              " In 2019, to celebrate 50 years to the Apollo landing, Google decided to make a tribute to Hamilton.: 0.3421052631578947,\n",
              " The mirrors at the Ivanpah plant were configured to create a picture of Hamilton and the Apollo 11 by moonlight.: 0.2631578947368421,\n",
              " Hamilton has a sister Kathryn.: 0.02631578947368421,\n",
              " She met her first husband, James Cox Hamilton, in the mid-1950s while attending college.: 0.13157894736842105,\n",
              " They were married on June 15, 1958, the summer after she graduated from Earlham.: 0.23684210526315788,\n",
              " She briefly taught high school mathematics and French at a public school in Boston, Indiana.: 0.3684210526315789,\n",
              " The couple then moved to Boston, Massachusetts, where they had a daughter, Lauren, born on November 10, 1959.: 0.2894736842105263,\n",
              " They divorced in 1967 and Margaret married Dan Lickly two years later.: 0.2894736842105263}"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# get 30% of the maximum score sentences\n",
        "select_length=int(len(sentence_tokens)*0.3)\n",
        "select_length\n",
        "summary=nlargest(select_length, sentence_scores,key=sentence_scores.get)\n",
        "\n",
        "summary"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fhqJ0eLpqiKw",
        "outputId": "785ac2bb-b1ef-4e70-945c-3eb23354a3e4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Her areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques.,\n",
              " The asynchronous executive designed by J. Halcombe Laning was used by Hamilton's team to develop asynchronous flight software:\n",
              " Because of the flight software's system-software's error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts' normal mission displays with priority displays of critical alarms in case of an emergency.,\n",
              " Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton's work \"the foundation for ultra-reliable software design\".\n",
              " Hamilton later wrote of the incident:\n",
              " The computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing.,\n",
              " When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science.,\n",
              " She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program.,\n",
              " The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.,\n",
              " The on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.,\n",
              " The IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering.,\n",
              " She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist.,\n",
              " She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.,\n",
              " By 1965, she was in charge of all Command Module software, which was all the software for navigation and lunar landing guidance.,\n",
              " This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed.,\n",
              " At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience.,\n",
              " In one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon.,\n",
              " From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft.,\n",
              " This depended on our assigning a unique priority to every process in the software in order to ensure that all of its events would take place in the correct order and at the right time relative to everything else that was going on.,\n",
              " In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.,\n",
              " It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware.,\n",
              " Hamilton details how she came to make up the term \"software engineering\":\n",
              " When I first came up with the term, no one had heard of it before, at least in our world.,\n",
              " Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system.,\n",
              " On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA's Apollo Moon missions.,\n",
              " Over time the term \"software engineering\" gained the same respect as any other technical discipline.,\n",
              " Hamilton's team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab.,\n",
              " The software's action, in this case, was to eliminate lower priority tasks and re-establish the more important ones ...,\n",
              " Another part of her team designed and developed the systems software.,\n",
              " Hamilton was concerned with legitimizing software development as an engineering discipline.,\n",
              " She is one of the people credited with coining the term \"software engineering\".]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# get the final summary\n",
        "final_summary=[word for word in summary]\n",
        "summary=''.join(final_summary)\n",
        "summary"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "xmtoCIqyqx80",
        "outputId": "168aa823-9890-4ac3-c9e0-fd43d3790373"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Her areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques.The asynchronous executive designed by J. Halcombe Laning was used by Hamilton\\'s team to develop asynchronous flight software:\\nBecause of the flight software\\'s system-software\\'s error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts\\' normal mission displays with priority displays of critical alarms in case of an emergency.Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton\\'s work \"the foundation for ultra-reliable software design\".\\nHamilton later wrote of the incident:\\nThe computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing.When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science.She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA\\'s Apollo program.The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.\\nThe on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.\\nThe IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering.She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist.She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.\\nBy 1965, she was in charge of all Command Module software, which was all the software for navigation and lunar landing guidance.This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed.At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience.In one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon.From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft.This depended on our assigning a unique priority to every process in the software in order to ensure that all of its events would take place in the correct order and at the right time relative to everything else that was going on.In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware.Hamilton details how she came to make up the term \"software engineering\":\\nWhen I first came up with the term, no one had heard of it before, at least in our world.Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system.On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA\\'s Apollo Moon missions.\\nOver time the term \"software engineering\" gained the same respect as any other technical discipline.Hamilton\\'s team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab.The software\\'s action, in this case, was to eliminate lower priority tasks and re-establish the more important ones\\xa0...Another part of her team designed and developed the systems software.Hamilton was concerned with legitimizing software development as an engineering discipline.She is one of the people credited with coining the term \"software engineering\".\\n'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Gensim\n",
        "\n",
        "Next, we decide to try the Gensim library which is a variation of the TextRank algorithm. The TextRank algorithm is a graph-based ranking algorithm which measures the relationship between the words and finds the most relevant sentences and keywords in a text.TextRank is an extractive and unsupervised model. It works as follows : first we split the text in input in sentences which are then represented in vectors. The similirities between those vectors are calculated and stored in a matrix called the similarity matrix. This matrix is then converted to a graph with the sentences being the vertices of the graph and the similarity scores the edges. Then the best ranked sentences are selected for the summary."
      ],
      "metadata": {
        "id": "Ef5NHOuciHW9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from gensim.summarization.summarizer import summarize\n",
        "from gensim.summarization import keywords"
      ],
      "metadata": {
        "id": "4jzYoD11D8oI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this example, we use the ratio parameter which specifies the portion of the sentences in the original text we want to be included. In our case, 20%"
      ],
      "metadata": {
        "id": "r3xs4C5TqlhO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "summ_pre = summarize(wikicontent, ratio = 0.2)\n",
        "print(\"Percent summary\")\n",
        "print(summ_pre)"
      ],
      "metadata": {
        "id": "XFBm8JUmEAqW",
        "outputId": "f1c57105-79c8-42b1-9196-407d8947dcf5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Percent summary\n",
            "Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner.\n",
            "She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program.\n",
            "She later founded two software companies—Higher Order Software in 1976 and Hamilton Technologies in 1986, both in Cambridge, Massachusetts.\n",
            "She is one of the people credited with coining the term \"software engineering\".On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA's Apollo Moon missions.\n",
            "She developed software for predicting weather, programming on the LGP-30 and the PDP-1 computers at Marvin Minsky's Project MAC.\n",
            "At the time, computer science and software engineering were not yet established disciplines; instead, programmers learned on the job with hands-on experience.\n",
            "From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft.\n",
            "It was her efforts on this project that made her a candidate for the position at NASA as the lead developer for Apollo flight software.\n",
            "Hamilton then joined the Charles Stark Draper Laboratory at MIT, which worked on the Apollo Space Mission.\n",
            "Hamilton was initially hired as a programmer for this process but moved on into systems design.\n",
            "Hamilton's team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab.\n",
            "Another part of her team designed and developed the systems software.\n",
            "This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed.\n",
            "She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist.\n",
            "Hamilton also served as Director of the Software Engineering Division.Her areas of expertise include: systems design and software development, enterprise and process modeling, development paradigm, formal systems modeling languages, system-oriented objects for systems modeling and development, automated life-cycle environments, methods for maximizing software reliability and reuse, domain analysis, correctness by built-in language properties, open-architecture techniques for robust systems, full life-cycle automation, quality assurance, seamless integration, error detection and recovery techniques, human-machine interface systems, operating systems, end-to-end testing techniques, and life-cycle management techniques.\n",
            "The on-board flight software captured these alarms with the \"never supposed to happen displays\" interrupting the astronauts with priority alarm displays.\n",
            "Halcombe Laning was used by Hamilton's team to develop asynchronous flight software:\n",
            "Because of the flight software's system-software's error detection and recovery techniques that included its system-wide \"kill and recompute\" from a \"safe place\" restart approach to its snapshot and rollback techniques, the Display Interface Routines (AKA the priority displays) together with its man-in-the-loop capabilities were able to be created in order to have the capability to interrupt the astronauts' normal mission displays with priority displays of critical alarms in case of an emergency.\n",
            "Jack Garman, a NASA computer engineer in mission control, recognized the meaning of the errors that were presented to the astronauts by the priority displays and shouted, \"Go, go!\" and they continued.\n",
            "Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton's work \"the foundation for ultra-reliable software design\".Hamilton later wrote of the incident:\n",
            "In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.\n",
            "Others have used HOS to formalize the semantics of linguistic quantifiers, and to formalize the design of reliable real-time embedded systems.Hamilton was the CEO of HOS through 1984 and left the company in 1985.\n",
            "The company was developed around the Universal Systems Language (USL) and its associated automated environment, the 001 Tool Suite, based on her paradigm of development before the fact for systems design and software development.\n",
            "Anthony Oettinger, Barry Boehm, and Hamilton have been credited with naming the discipline of \"software engineering\".\n",
            "Hamilton details how she came to make up the term \"software engineering\":\n",
            "When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science.\n",
            "Hamilton was concerned with legitimizing software development as an engineering discipline.\n",
            "Over time the term \"software engineering\" gained the same respect as any other technical discipline.\n",
            "Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system.\n",
            "According to Wired's Karen Tegan Padir: \"She, along with that other early programming pioneer, COBOL inventor Grace Hopper, also deserve tremendous credit for helping to open the door for more women to enter and succeed in STEM fields like software.\"In 2019, to celebrate 50 years to the Apollo landing, Google decided to make a tribute to Hamilton.\n",
            "On April 28, 2017, she received the Computer History Museum Fellow Award, which honors exceptional men and women whose computing ideas have changed the world.\n",
            "\"001: A Full Life Cycle Systems Engineering and Software Development Environment\".\n",
            "Margaret Hamilton ’58 – Presidential Medal of Freedom Recipient Archived July 30, 2019, at the Wayback Machine: Earlham College profile\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "summ_words = summarize(wikicontent, word_count=200)\n",
        "print(summ_words)"
      ],
      "metadata": {
        "id": "RpAPJnMXIb0g",
        "outputId": "efd520f8-791f-45f1-f1e8-c21f9ef3ebe0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program.\n",
            "She is one of the people credited with coining the term \"software engineering\".On November 22, 2016, Hamilton received the Presidential Medal of Freedom from president Barack Obama for her work leading to the development of on-board flight software for NASA's Apollo Moon missions.\n",
            "From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft.\n",
            "Hamilton's team was responsible for developing in-flight software, which included algorithms designed by various senior scientists for the Apollo command module, Apollo lunar module, and the subsequent Skylab.\n",
            "This included error detection and recovery software such as restarts and the Display Interface Routines (also known as the Priority Displays), which Hamilton designed and developed.\n",
            "In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pip install sumy\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PidevFwkSzfl",
        "outputId": "51e4742a-de5b-4b27-8554-214d57705f0b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting sumy\n",
            "  Downloading sumy-0.11.0-py2.py3-none-any.whl (97 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.3/97.3 KB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting breadability>=0.1.20\n",
            "  Downloading breadability-0.1.20.tar.gz (32 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: requests>=2.7.0 in /usr/local/lib/python3.8/dist-packages (from sumy) (2.25.1)\n",
            "Collecting docopt<0.7,>=0.6.1\n",
            "  Downloading docopt-0.6.2.tar.gz (25 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: nltk>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from sumy) (3.7)\n",
            "Collecting pycountry>=18.2.23\n",
            "  Downloading pycountry-22.3.5.tar.gz (10.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.1/10.1 MB\u001b[0m \u001b[31m40.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: chardet in /usr/local/lib/python3.8/dist-packages (from breadability>=0.1.20->sumy) (4.0.0)\n",
            "Requirement already satisfied: lxml>=2.0 in /usr/local/lib/python3.8/dist-packages (from breadability>=0.1.20->sumy) (4.9.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk>=3.0.2->sumy) (1.2.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from nltk>=3.0.2->sumy) (4.64.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk>=3.0.2->sumy) (7.1.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.8/dist-packages (from nltk>=3.0.2->sumy) (2022.6.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from pycountry>=18.2.23->sumy) (57.4.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests>=2.7.0->sumy) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.7.0->sumy) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests>=2.7.0->sumy) (1.24.3)\n",
            "Building wheels for collected packages: breadability, docopt, pycountry\n",
            "  Building wheel for breadability (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for breadability: filename=breadability-0.1.20-py2.py3-none-any.whl size=21714 sha256=7213f3810807c7721ccf03f8bce5028470e77b0d8c9b3be3f9f75a5bbd4efdc7\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/0d/0c/2062d8c1758b4b1a2e42b4a63e6660d9ec2ba9463cfee9eeab\n",
            "  Building wheel for docopt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=3276c8eef167eb6d8c1a2785a414e0c235857123b41cb872a167100878af22d7\n",
            "  Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c\n",
            "  Building wheel for pycountry (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pycountry: filename=pycountry-22.3.5-py2.py3-none-any.whl size=10681845 sha256=c7c67481f86d7ab77ee3131ddf1275ee0f5b2b36c2ee6fab9c73627658152385\n",
            "  Stored in directory: /root/.cache/pip/wheels/e2/aa/0f/c224e473b464387170b83ca7c66947b4a7e33e8d903a679748\n",
            "Successfully built breadability docopt pycountry\n",
            "Installing collected packages: docopt, pycountry, breadability, sumy\n",
            "Successfully installed breadability-0.1.20 docopt-0.6.2 pycountry-22.3.5 sumy-0.11.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import sumy\n",
        "from sumy.parsers.plaintext import PlaintextParser\n",
        "from sumy.nlp.tokenizers import Tokenizer\n",
        "from sumy.summarizers.lex_rank import LexRankSummarizer"
      ],
      "metadata": {
        "id": "QAtTfcB7UkEC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2joJdimRVcTM",
        "outputId": "f72f7bf6-02ed-4ecd-aa9d-b4e0fab10959"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "parser=PlaintextParser.from_string(wikicontent, Tokenizer(\"english\"))"
      ],
      "metadata": {
        "id": "knliWMlHVA5K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "summarizer = LexRankSummarizer()\n",
        "summary = summarizer(parser.document, 10)\n",
        "for sentence in summary:\n",
        "  print(sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p7_4VV3VWpxC",
        "outputId": "09f17662-37d0-4f78-ccfc-ea109e20be55"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program.\n",
            "She studied mathematics at the University of Michigan in 1955 before transferring to Earlham College, where her mother was a student; she earned a BA in mathematics with a minor in philosophy in 1958.\n",
            "I was the first one to get it to work.\n",
            "It was her efforts on this project that made her a candidate for the position at NASA as the lead developer for Apollo flight software.\n",
            "Hamilton was initially hired as a programmer for this process but moved on into systems design.\n",
            "=== Apollo program === In one of the critical moments of the Apollo 11 mission, the Apollo Guidance Computer, together with the on-board flight software, averted an abort of the landing on the Moon.\n",
            "It then sent out an alarm, which meant to the astronaut, 'I'm overloaded with more tasks than I should be doing at this time and I'm going to keep only the more important tasks'; i.e., the ones needed for landing ... Actually, the computer was programmed to do more than recognize error conditions.\n",
            "=== Businesses === In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program.\n",
            "They created a product called USE.IT, based on the HOS methodology they developed at MIT.\n",
            "When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Bert\n",
        "\n",
        "For now we only used extractive text summarization which identified the significant sentences of our text and added them to the summary. In this case the summary contains the exact same sentences from the original text.\n",
        "\n",
        "In the abstractive method, the model identifies the important sections of the text as well but also the context. In this cas the summary is generated by the model.\n",
        "\n",
        "BERT (Bidirectional Encoder Representations from Transformers) is a transformer based machine learning model developped by Google and published in 2018. It uses transformers with an attention mechanism that learns the relations between the words of a text. A transformer is based on an a pile of encoders and decoders. The encoder processes the input by computing the score of each word compared to other words. This is then sent to a decoder that predicts the sentence. As opposed to directional models that read the text sequentially (left to right for example), the tranformer encoder reads all the text at once. This allows the model to learn about the context of a word based not only on the sentence in which it is but all the text."
      ],
      "metadata": {
        "id": "W6ua11qUrgzr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install bert-extractive-summarizer"
      ],
      "metadata": {
        "id": "F4ObecuKYicK",
        "outputId": "87eeb989-085b-40a4-d9c6-e6d4a24372eb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting bert-extractive-summarizer\n",
            "  Downloading bert_extractive_summarizer-0.10.1-py3-none-any.whl (25 kB)\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.25.1-py3-none-any.whl (5.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy in /usr/local/lib/python3.8/dist-packages (from bert-extractive-summarizer) (3.4.4)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from bert-extractive-summarizer) (1.0.2)\n",
            "Requirement already satisfied: numpy>=1.14.6 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->bert-extractive-summarizer) (1.21.6)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->bert-extractive-summarizer) (1.7.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->bert-extractive-summarizer) (3.1.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->bert-extractive-summarizer) (1.2.0)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (2.0.8)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (21.3)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (3.0.8)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (1.10.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (57.4.0)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (2.4.5)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (0.10.1)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (0.10.1)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (3.3.0)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (2.11.3)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (3.0.10)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (4.64.1)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (1.0.4)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (8.1.6)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (1.0.9)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (0.7.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (2.0.7)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (6.3.0)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.8/dist-packages (from spacy->bert-extractive-summarizer) (2.25.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers->bert-extractive-summarizer) (6.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers->bert-extractive-summarizer) (3.8.2)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m92.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting huggingface-hub<1.0,>=0.10.0\n",
            "  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m182.4/182.4 KB\u001b[0m \u001b[31m21.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers->bert-extractive-summarizer) (2022.6.2)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers->bert-extractive-summarizer) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->spacy->bert-extractive-summarizer) (3.0.9)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (2022.12.7)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy->bert-extractive-summarizer) (4.0.0)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy->bert-extractive-summarizer) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy->bert-extractive-summarizer) (0.0.3)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.8/dist-packages (from typer<0.8.0,>=0.3.0->spacy->bert-extractive-summarizer) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy->bert-extractive-summarizer) (2.0.1)\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers, bert-extractive-summarizer\n",
            "Successfully installed bert-extractive-summarizer-0.10.1 huggingface-hub-0.11.1 tokenizers-0.13.2 transformers-4.25.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from summarizer import Summarizer\n",
        "model = Summarizer()"
      ],
      "metadata": {
        "id": "IndZcSF_YpgW",
        "outputId": "1519c749-dc91-4264-b050-f8ddb3f0d0ca",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217,
          "referenced_widgets": [
            "0e6f4d45c5424fa29faf88ce4b633919",
            "fe80c0fdb8154e9f8cd0e6c04542130f",
            "f0627a81cc8746098951c75e2a99c6ac",
            "b936a57982d7475b909a8d5bfb47c132",
            "482c72ff6d2d443bbd3f07f33aa53e73",
            "3e4ad9e8777648f1bf6d54b5b53bc5ff",
            "236eb4c47198471787432449cffdeebc",
            "e71860f04c9f413091cc9dffd27eca9a",
            "3434d43a13544415abb99adf21c55521",
            "e297b5139f944a0cbf90372cbf987e37",
            "53008621bd684e0e868c1495e2cee027",
            "59b0a5d4a36d40079bf9709c3cf3294a",
            "8f0f2a2260364b5c859869dd01fc7d11",
            "ec502d4c2e974e2d98fd0cf6acb7135f",
            "75f5d675c3654a33ad2f2dcbe5141022",
            "2c67ca59297f481197bc7439102b8975",
            "e432ad0920304c7c9ce47e47841cdbc4",
            "70d860cdf00c4fbd8ce0570dcc435d75",
            "d099563b01f64ca7bcb9ada9a3bfe919",
            "49ff316f5632459ea6cca519c70aa5f7",
            "024dd09abd4d4c65b10e75c37b214241",
            "eb9003e88279404cbc94fc22189758a0",
            "ceb972b115e3409897d3cb07dc239767",
            "bd560499fdea491b8621eb302295619c",
            "d3b7084a4caa4997bbe202d51c468df2",
            "fccde74cb2444f16b9564e2936e1b3da",
            "1f14e14eb41c4e409b671e2d10bd51bf",
            "7f3a8cd7256d414598338dbf9b277604",
            "82c06d5d33fd408cbb373682b6edb799",
            "e1c257a6f09047acb83bdcdb7f499be4",
            "14819e1dc63846ea92f77b75a712eeb5",
            "6aab4f29557447b7bb52ef19daa74124",
            "0c0342ca388e45a8bad5a97eeedd35d0",
            "df31d0890aa748c48237b374e5eecbcf",
            "9de98b99c91046a09bfe006932aa9eae",
            "4f8b0c276e1e41e2be2f044b303a7ad0",
            "c94d065654b548568d64ec5d27b9077d",
            "1d1d985c647b421c855bd555d4edef52",
            "3cbd7cea7ac54665847ecf3bed64af56",
            "bd81acf5684f48c89cddfa40cd178930",
            "f4b3d126f1114a45b663a9c8aa0fdfc7",
            "48699a6d213b402a9c770369f3ffcc8f",
            "3e846a309fd0436baa14dcd87b1c5efd",
            "07ac5e1727384642a102cc22438adcfb"
          ]
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/571 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0e6f4d45c5424fa29faf88ce4b633919"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.34G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "59b0a5d4a36d40079bf9709c3cf3294a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-large-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ceb972b115e3409897d3cb07dc239767"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "df31d0890aa748c48237b374e5eecbcf"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result = model(wikicontent, ratio=0.2)"
      ],
      "metadata": {
        "id": "9o66l_LYY2wF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(result)"
      ],
      "metadata": {
        "id": "yqQ2oOGEZlBj",
        "outputId": "80efb08c-00ba-4aa7-b46c-4d8cd869a814",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner. She was director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo program. The family later moved to Michigan, where Margaret graduated from Hancock High School in 1954. She says her poet father and headmaster grandfather inspired her to include a minor in philosophy in her studies. Hamilton said:\n",
            "\n",
            "What they used to do when you came into this organization as a beginner, was to assign you this program which nobody was able to ever figure out or get to run. When I was the beginner they gave it to me as well. Another part of her team designed and developed the systems software. She worked to gain hands-on experience during a time when computer science courses were uncommon and software engineering courses did not exist. Hamilton had thought long and hard about this. By some accounts, the astronauts had inadvertently left the rendezvous radar switch on, causing these alarms to be triggered (the claim that the radar was left on inadvertently by the astronauts is disputed by Robert Wills with the National Museum of Computing). Paul Curto, senior technologist who nominated Hamilton for a NASA Space Act Award, called Hamilton's work \"the foundation for ultra-reliable software design\". === Businesses ===\n",
            "In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program. When Hamilton started using the term \"software engineering\" during the early Apollo missions, software development was not taken seriously compared to other engineering, nor was it regarded as a science. The IEEE Software September/October 2018 issue celebrates the 50th anniversary of software engineering. Hamilton talks about \"Errors\" and how they influenced her work related to software engineering and how her language, USL, could be used to prevent the majority of \"Errors\" in a system. On April 28, 2017, she received the Computer History Museum Fellow Award, which honors exceptional men and women whose computing ideas have changed the world. In 2018, she was awarded an honorary doctorate degree by the Polytechnic University of Catalonia. In 2019, she was awarded The Washington Award. The relationship between design and verification\". Universal Systems Language for Preventative Systems Engineering\", Proc. A Formal Universal Systems Semantics for SysML\". == Personal life ==\n",
            "Hamilton has a sister Kathryn. She briefly taught high school mathematics and French at a public school in Boston, Indiana.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "result1 = model(wikicontent, num_sentences=5)\n",
        "print(result1)"
      ],
      "metadata": {
        "id": "OLg1ci16Zp32",
        "outputId": "7a22bd7d-1ab8-4b6c-8925-b3899c40e5e1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Margaret Heafield Hamilton (born August 17, 1936) is an American computer scientist, systems engineer, and business owner. === SAGE Project ===\n",
            "From 1961 to 1963, Hamilton worked on the Semi-Automatic Ground Environment (SAGE) Project at the MIT Lincoln Lab, where she was one of the programmers who wrote software for the prototype AN/FSQ-7 computer (the XD-1), used by the U.S. Air Force to search for possibly unfriendly aircraft. === Businesses ===\n",
            "In 1976, Hamilton co-founded with Saydean Zeldin a company called Higher Order Software (HOS) to further develop ideas about error prevention and fault tolerance emerging from their experience at MIT working on the Apollo program. In 2019, she was awarded The Washington Award. The relationship between design and verification\".\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Conclusion\n",
        "\n",
        "It can be concluded that BERT, Spacy, and Gensim are all effective tools for summarizing text.  \n",
        "\n",
        "Spacy is a fast and efficient natural language processing library that includes pre-trained models for summarization and other tasks. \n",
        "\n",
        "Gensim is a powerful library for creating and using vector space models, including the Latent Semantic Analysis (LSA) model, which can be used for summarization by identifying the most important words and phrases in a text. \n",
        "\n",
        "As we can see, we achieve much better results with BERT. BERT rewrites the sentences and understands the context in which the words are used which makes the summary much more readable.\n",
        "The first two summaries were simply a sequence of important sentences from the original text. "
      ],
      "metadata": {
        "id": "aEIY3hwOyuZp"
      }
    }
  ]
}