_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EasyTwitterAPI \n",
    "\n",
    "This jupyer notebook contains a summary of the main  `EasyTwitterAPI` functionality that will help you to scrapee and store data from Twitter:\n",
    "#### Profile Information\n",
    "- General information: number of posts, followees, screen name...\n",
    "- Followees (a.k.a., Friends) of a user\n",
    "- Followers of a user\n",
    "\n",
    "#### List information\n",
    "- Lists for which a user is a member/creator/subscriber\n",
    "- Members of a List\n",
    "\n",
    "#### Activity\n",
    "- Timeline (i.e., tweets, answers, retweets, qtweets) of a user\n",
    "- Favourited tweets of a user\n",
    "- Tweets by id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%\n",
    "from EasyTwitterAPI.easy_twitter_api import EasyTwitterAPI\n",
    "\n",
    "import json\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use the `EasyTwitterAPI` package, you need i) MongodDB credentials and ii) Twitter API credentials.\n",
    "\n",
    "The **MongoDB credentials** are stored in a file called `local/host_mongodb.txt`, which should contain the  connection string for your MongoDB database, it should look something like\n",
    "\n",
    "    mongodb+srv://<USERNAME>:<PASSWORD>@cluster[...]/myFirstDatabase?retryWrites=true&w=majority\n",
    "Alternatively, you can install MongoDB locally and then you only need to set `host='localhost'`.\n",
    "\n",
    "The **Twitter API credentials** are stored in a file called `local/credentials_api.json`. It should be a json file with the following four fields\n",
    "\n",
    "    {\n",
    "    \"ACCESS_KEY\": \"XXXXXX\",\n",
    "    \"ACCESS_SECRET\": \"XXXXXX\",\n",
    "    \"CONSUMER_KEY\": \"XXXXXX\",\n",
    "    \"CONSUMER_SECRET\": \"XXXXXX\",\n",
    "    \"BEARER_TOKEN\":\"XXXXXX\"\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"local/host_mongodb.txt\", \"r\") as f:\n",
    "    host = f.readline().strip()\n",
    "scraper = EasyTwitterAPI(cred_file='local/credentials_api.json',  \n",
    "                         db_name='twitter_test',\n",
    "                         sleep_secs=330,\n",
    "                         host=host\n",
    "                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, coll_name in enumerate(scraper.db.collection_names()):\n",
    "    print(f\"{i} {coll_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraper.db.create_indexes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To scrape the profile information of a user:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scraper.activate_cache(True)\n",
    "user = scraper.get_user(screen_name='Twitter')\n",
    "user = scraper.get_user(screen_name='jack')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the profile information from the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = scraper.db.load_users(filter_={'screen_name': {'$in':['Twitter', 'jack']}}, find_one=False, return_as='df')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To scrape the profile information of multiple users at the same time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = scraper.get_many_users(screen_name=['Twitter', 'jack', 'TwitterAPI'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Followees\n",
    "To scrape the followees of a user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "followees = scraper.get_followees(screen_name='TwitterAPI')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Followers\n",
    "To scrape the followers of a user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "followers = scraper.get_followers(screen_name='jack', max_num=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lists of user\n",
    "We can collect three types of lists for a given user:\n",
    " - membership (m)\n",
    " - owned (o)\n",
    " - subscriptions (s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "since = datetime.strptime('2021-09-01', '%Y-%m-%d')\n",
    "lists = scraper.update_lists_of_user(list_type='m', \n",
    "                             min_dt=datetime.now(),\n",
    "                             screen_name='jack',\n",
    "                                    max_num=120)\n",
    "\n",
    "print('\\nThis are the first 4 Lists:')\n",
    "for list_id in lists[:4]:\n",
    "    print(list_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To scraper only the ids of the Lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraper.activate_cache(True)\n",
    "lists_ids = scraper.get_lists_ids_of_user(list_type='m', screen_name='jack')\n",
    "\n",
    "print('\\nThis are the first 4 Lists:')\n",
    "for list_id in lists_ids[:4]:\n",
    "    print(list_id)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To scraper the full information of the Lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "df_lists = scraper.get_lists_of_user_full(list_type='m', screen_name='jack', max_num=289248, force=True)\n",
    "\n",
    "\n",
    "df_lists.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_ = scraper.get_list(list_id_str='1283064489957445633')\n",
    "for key, value in list_.items():\n",
    "    print(f\"{key} : {value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get members of Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "members_list = scraper.get_members_of_list(list_id_str='1283064489957445633', max_num=1000000)\n",
    "print('\\nThis are the first 4 members:')\n",
    "for member_id in members_list[:4]:\n",
    "    print(member_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Timeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraper.activate_cache(True)\n",
    "df = scraper.get_user_activity_limited(screen_name='jack', max_num=2000, update_many=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('type').count()[['id_str']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = scraper.get_user_activity_limited(screen_name='jack', max_num=5151517)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Favourites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = scraper.get_user_favorites(screen_name='jack', max_num=5151517)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_str_list = list(df['id_str'].values)[:612]\n",
    "print(f\"Number of tweets: {len(id_str_list)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_i =  scraper.get_tweets(id_str_list=id_str_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.type=='answer'].head().columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to collect the activity for t the followees of a user?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "followees = scraper.get_followees(screen_name='jack')\n",
    "\n",
    "for foll_id_str in followees[:20]:\n",
    "    df = scraper.get_user_activity_limited(user_id=foll_id_str, max_num=5151517)\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}