{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "colab": { "name": "CMU_BERT_Base_Misinformation_Classification.ipynb", "provenance": [], "collapsed_sections": [ "_lHLplImxL7B", "RkgaNWv4YOjP", "tsxIxgfSwwS7" ], "toc_visible": true }, "accelerator": "GPU" }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "E4majde3xL6_" }, "source": [ "# COVID-19 Tweet Misinformation Detection using BERT\n", "\n", "Contributers:\n", "Sumit Kumar
\n", "Raj Ratn Pranesh
\n", "Prof Kathleen M. Carley" ] }, { "cell_type": "markdown", "metadata": { "id": "_lHLplImxL7B" }, "source": [ "# DistilBERT" ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "6doyy7UZxL7C", "colab": { "base_uri": "https://localhost:8080/", "height": 340 }, "outputId": "7d115954-77d6-4904-9837-717256e023db" }, "source": [ "!pip install transformers==2.3.0" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Requirement already satisfied: transformers==2.3.0 in /usr/local/lib/python3.6/dist-packages (2.3.0)\n", "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (1.18.5)\n", "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (2019.12.20)\n", "Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (0.0.43)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (2.23.0)\n", "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (4.41.1)\n", "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (1.16.0)\n", "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (0.1.91)\n", "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (7.1.2)\n", "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (1.15.0)\n", "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (0.16.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (2020.6.20)\n", "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (2.10)\n", "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (3.0.4)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (1.24.3)\n", "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.3.0) (0.3.3)\n", "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.3.0) (0.10.0)\n", "Requirement already satisfied: botocore<1.20.0,>=1.19.0 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.3.0) (1.19.0)\n", "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.20.0,>=1.19.0->boto3->transformers==2.3.0) (2.8.1)\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "ukCVRdgnxL7N" }, "source": [ "We then load and inspect the dataset we had previously prepared. In order to train faster, we use the sampled version, which contains 10% of the original prepared dataset." ] }, { "cell_type": "code", "metadata": { "id": "CZ4SvMMixL7O" }, "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('/content/result_31.csv', encoding='latin-1')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "2Vvibv3dxL7V", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "29b79a65-8e63-4f3b-e478-1330b5d373cd" }, "source": [ "df.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1970, 4)" ] }, "metadata": { "tags": [] }, "execution_count": 4 } ] }, { "cell_type": "code", "metadata": { "id": "9gRHmTo0xL7b", "colab": { "base_uri": "https://localhost:8080/", "height": 419 }, "outputId": "60224708-1956-47b4-c9c6-e59b6580f3fc" }, "source": [ "df" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labels
01.005030e+05threeSnake oil Salesmen are not gone. They have jus...3
11.009760e+05threeFact check: Gargling water with salt or vinega...3
21.011710e+05zero.. this is great advice. Also, stop fighting f...0
31.017920e+05three14 people in Iran died from alcohol poisoning,...3
41.037400e+05oneTo be fair, China is not the only country alle...1
...............
19651.230000e+18one@TheSecondRevol1 @VincentCrypt46 I have been s...1
19661.230000e+18oneIf anyone is aware of the 5G theory related to...1
19671.230000e+18one@ClydeLewis @ClydeLewis have you looked into 5...1
19681.230000e+18oneHow many have the “Virus” where there’s...1
19691.230000e+18oneSo why am I still so thoroughly stunned by the...1
\n", "

1970 rows × 4 columns

\n", "
" ], "text/plain": [ " ID ... Product_Labels\n", "0 1.005030e+05 ... 3\n", "1 1.009760e+05 ... 3\n", "2 1.011710e+05 ... 0\n", "3 1.017920e+05 ... 3\n", "4 1.037400e+05 ... 1\n", "... ... ... ...\n", "1965 1.230000e+18 ... 1\n", "1966 1.230000e+18 ... 1\n", "1967 1.230000e+18 ... 1\n", "1968 1.230000e+18 ... 1\n", "1969 1.230000e+18 ... 1\n", "\n", "[1970 rows x 4 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "markdown", "metadata": { "id": "8CA6dEA1xL7i" }, "source": [ "Clean(tweet)" ] }, { "cell_type": "code", "metadata": { "cellView": "both", "id": "LzUeXYHhxL7j" }, "source": [ "\n", "import re\n", "import string\n", "def clean(tweet):\n", " \n", " # Special characters\n", " tweet = re.sub(r\"\\x89Û_\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÒ\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÓ\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÏWhen\", \"When\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÏ\", \"\", tweet)\n", " tweet = re.sub(r\"China\\x89Ûªs\", \"China's\", tweet)\n", " tweet = re.sub(r\"let\\x89Ûªs\", \"let's\", tweet)\n", " tweet = re.sub(r\"\\x89Û÷\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Ûª\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û\\x9d\", \"\", tweet)\n", " tweet = re.sub(r\"å_\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û¢\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û¢åÊ\", \"\", tweet)\n", " tweet = re.sub(r\"fromåÊwounds\", \"from wounds\", tweet)\n", " tweet = re.sub(r\"åÊ\", \"\", tweet)\n", " tweet = re.sub(r\"åÈ\", \"\", tweet)\n", " tweet = re.sub(r\"JapÌ_n\", \"Japan\", tweet) \n", " tweet = re.sub(r\"Ì©\", \"e\", tweet)\n", " tweet = re.sub(r\"å¨\", \"\", tweet)\n", " tweet = re.sub(r\"Surṳ\", \"Suruc\", tweet)\n", " tweet = re.sub(r\"åÇ\", \"\", tweet)\n", " tweet = re.sub(r\"å£3million\", \"3 million\", tweet)\n", " tweet = re.sub(r\"åÀ\", \"\", tweet)\n", " \n", " # Contractions\n", " tweet = re.sub(r\"he's\", \"he is\", tweet)\n", " tweet = re.sub(r\"there's\", \"there is\", tweet)\n", " tweet = re.sub(r\"We're\", \"We are\", tweet)\n", " tweet = re.sub(r\"That's\", \"That is\", tweet)\n", " tweet = re.sub(r\"won't\", \"will not\", tweet)\n", " tweet = re.sub(r\"they're\", \"they are\", tweet)\n", " tweet = re.sub(r\"Can't\", \"Cannot\", tweet)\n", " tweet = re.sub(r\"wasn't\", \"was not\", tweet)\n", " tweet = re.sub(r\"don\\x89Ûªt\", \"do not\", tweet)\n", " tweet = re.sub(r\"aren't\", \"are not\", tweet)\n", " tweet = re.sub(r\"isn't\", \"is not\", tweet)\n", " tweet = re.sub(r\"What's\", \"What is\", tweet)\n", " tweet = re.sub(r\"haven't\", \"have not\", tweet)\n", " tweet = re.sub(r\"hasn't\", \"has not\", tweet)\n", " tweet = re.sub(r\"There's\", \"There is\", tweet)\n", " tweet = re.sub(r\"He's\", \"He is\", tweet)\n", " tweet = re.sub(r\"It's\", \"It is\", tweet)\n", " tweet = re.sub(r\"You're\", \"You are\", tweet)\n", " tweet = re.sub(r\"I'M\", \"I am\", tweet)\n", " tweet = re.sub(r\"shouldn't\", \"should not\", tweet)\n", " tweet = re.sub(r\"wouldn't\", \"would not\", tweet)\n", " tweet = re.sub(r\"i'm\", \"I am\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªm\", \"I am\", tweet)\n", " tweet = re.sub(r\"I'm\", \"I am\", tweet)\n", " tweet = re.sub(r\"Isn't\", \"is not\", tweet)\n", " tweet = re.sub(r\"Here's\", \"Here is\", tweet)\n", " tweet = re.sub(r\"you've\", \"you have\", tweet)\n", " tweet = re.sub(r\"you\\x89Ûªve\", \"you have\", tweet)\n", " tweet = re.sub(r\"we're\", \"we are\", tweet)\n", " tweet = re.sub(r\"what's\", \"what is\", tweet)\n", " tweet = re.sub(r\"couldn't\", \"could not\", tweet)\n", " tweet = re.sub(r\"we've\", \"we have\", tweet)\n", " tweet = re.sub(r\"it\\x89Ûªs\", \"it is\", tweet)\n", " tweet = re.sub(r\"doesn\\x89Ûªt\", \"does not\", tweet)\n", " tweet = re.sub(r\"It\\x89Ûªs\", \"It is\", tweet)\n", " tweet = re.sub(r\"Here\\x89Ûªs\", \"Here is\", tweet)\n", " tweet = re.sub(r\"who's\", \"who is\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªve\", \"I have\", tweet)\n", " tweet = re.sub(r\"y'all\", \"you all\", tweet)\n", " tweet = re.sub(r\"can\\x89Ûªt\", \"cannot\", tweet)\n", " tweet = re.sub(r\"would've\", \"would have\", tweet)\n", " tweet = re.sub(r\"it'll\", \"it will\", tweet)\n", " tweet = re.sub(r\"we'll\", \"we will\", tweet)\n", " tweet = re.sub(r\"wouldn\\x89Ûªt\", \"would not\", tweet)\n", " tweet = re.sub(r\"We've\", \"We have\", tweet)\n", " tweet = re.sub(r\"he'll\", \"he will\", tweet)\n", " tweet = re.sub(r\"Y'all\", \"You all\", tweet)\n", " tweet = re.sub(r\"Weren't\", \"Were not\", tweet)\n", " tweet = re.sub(r\"Didn't\", \"Did not\", tweet)\n", " tweet = re.sub(r\"they'll\", \"they will\", tweet)\n", " tweet = re.sub(r\"they'd\", \"they would\", tweet)\n", " tweet = re.sub(r\"DON'T\", \"DO NOT\", tweet)\n", " tweet = re.sub(r\"That\\x89Ûªs\", \"That is\", tweet)\n", " tweet = re.sub(r\"they've\", \"they have\", tweet)\n", " tweet = re.sub(r\"i'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"should've\", \"should have\", tweet)\n", " tweet = re.sub(r\"You\\x89Ûªre\", \"You are\", tweet)\n", " tweet = re.sub(r\"where's\", \"where is\", tweet)\n", " tweet = re.sub(r\"Don\\x89Ûªt\", \"Do not\", tweet)\n", " tweet = re.sub(r\"we'd\", \"we would\", tweet)\n", " tweet = re.sub(r\"i'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"weren't\", \"were not\", tweet)\n", " tweet = re.sub(r\"They're\", \"They are\", tweet)\n", " tweet = re.sub(r\"Can\\x89Ûªt\", \"Cannot\", tweet)\n", " tweet = re.sub(r\"you\\x89Ûªll\", \"you will\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªd\", \"I would\", tweet)\n", " tweet = re.sub(r\"let's\", \"let us\", tweet)\n", " tweet = re.sub(r\"it's\", \"it is\", tweet)\n", " tweet = re.sub(r\"can't\", \"cannot\", tweet)\n", " tweet = re.sub(r\"don't\", \"do not\", tweet)\n", " tweet = re.sub(r\"you're\", \"you are\", tweet)\n", " tweet = re.sub(r\"i've\", \"I have\", tweet)\n", " tweet = re.sub(r\"that's\", \"that is\", tweet)\n", " tweet = re.sub(r\"i'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"doesn't\", \"does not\", tweet)\n", " tweet = re.sub(r\"i'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"didn't\", \"did not\", tweet)\n", " tweet = re.sub(r\"ain't\", \"am not\", tweet)\n", " tweet = re.sub(r\"you'll\", \"you will\", tweet)\n", " tweet = re.sub(r\"I've\", \"I have\", tweet)\n", " tweet = re.sub(r\"Don't\", \"do not\", tweet)\n", " tweet = re.sub(r\"I'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"I'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"Let's\", \"Let us\", tweet)\n", " tweet = re.sub(r\"you'd\", \"You would\", tweet)\n", " tweet = re.sub(r\"It's\", \"It is\", tweet)\n", " tweet = re.sub(r\"Ain't\", \"am not\", tweet)\n", " tweet = re.sub(r\"Haven't\", \"Have not\", tweet)\n", " tweet = re.sub(r\"Could've\", \"Could have\", tweet)\n", " tweet = re.sub(r\"youve\", \"you have\", tweet) \n", " tweet = re.sub(r\"donå«t\", \"do not\", tweet) \n", " \n", " # Character entity references\n", " tweet = re.sub(r\">\", \">\", tweet)\n", " tweet = re.sub(r\"<\", \"<\", tweet)\n", " tweet = re.sub(r\"&\", \"&\", tweet)\n", " \n", " # Typos, slang and informal abbreviations\n", " tweet = re.sub(r\"w/e\", \"whatever\", tweet)\n", " tweet = re.sub(r\"w/\", \"with\", tweet)\n", " tweet = re.sub(r\"USAgov\", \"USA government\", tweet)\n", " tweet = re.sub(r\"recentlu\", \"recently\", tweet)\n", " tweet = re.sub(r\"Ph0tos\", \"Photos\", tweet)\n", " tweet = re.sub(r\"amirite\", \"am I right\", tweet)\n", " tweet = re.sub(r\"exp0sed\", \"exposed\", tweet)\n", " tweet = re.sub(r\"<3\", \"love\", tweet)\n", " tweet = re.sub(r\"amageddon\", \"armageddon\", tweet)\n", " tweet = re.sub(r\"Trfc\", \"Traffic\", tweet)\n", " tweet = re.sub(r\"8/5/2015\", \"2015-08-05\", tweet)\n", " tweet = re.sub(r\"WindStorm\", \"Wind Storm\", tweet)\n", " tweet = re.sub(r\"8/6/2015\", \"2015-08-06\", tweet)\n", " tweet = re.sub(r\"10:38PM\", \"10:38 PM\", tweet)\n", " tweet = re.sub(r\"10:30pm\", \"10:30 PM\", tweet)\n", " tweet = re.sub(r\"16yr\", \"16 year\", tweet)\n", " tweet = re.sub(r\"lmao\", \"laughing my ass off\", tweet) \n", " tweet = re.sub(r\"TRAUMATISED\", \"traumatized\", tweet)\n", " \n", " # re.sub(r\"humanconsumption\", \"human consumption\", tweet)\n", " # tweet = re.sub(r\"Typhoon-Devastated\", \"Typhoon Devastated\", tweet)\n", " # tweet = re.sub(r\"Meat-Loving\", \"Meat Loving\", tweet)\n", " # tweet = re.sub(r\"facialabuse\", \"facial abuse\", tweet)\n", " # tweet = re.sub(r\"LakeCounty\", \"Lake County\", tweet)\n", " # tweet = re.sub(r\"BeingAuthor\", \"Being Author\", tweet)\n", " # tweet = re.sub(r\"withheavenly\", \"with heavenly\", tweet)\n", " # tweet = re.sub(r\"thankU\", \"thank you\", tweet)\n", " # tweet = re.sub(r\"iTunesMusic\", \"iTunes Music\", tweet)\n", " # tweet = re.sub(r\"OffensiveContent\", \"Offensive Content\", tweet)\n", " # tweet = re.sub(r\"WorstSummerJob\", \"Worst Summer Job\", tweet)\n", " # tweet = re.sub(r\"HarryBeCareful\", \"Harry Be Careful\", tweet)\n", " # tweet = re.sub(r\"NASASolarSystem\", \"NASA Solar System\", tweet)\n", " # tweet = re.sub(r\"animalrescue\", \"animal rescue\", tweet)\n", " # tweet = re.sub(r\"KurtSchlichter\", \"Kurt Schlichter\", tweet)\n", " # tweet = re.sub(r\"aRmageddon\", \"armageddon\", tweet)\n", " # tweet = re.sub(r\"Throwingknifes\", \"Throwing knives\", tweet)\n", " # tweet = re.sub(r\"GodsLove\", \"God's Love\", tweet)\n", " # tweet = re.sub(r\"bookboost\", \"book boost\", tweet)\n", " # tweet = re.sub(r\"ibooklove\", \"I book love\", tweet)\n", " # tweet = re.sub(r\"NestleIndia\", \"Nestle India\", tweet)\n", " # tweet = re.sub(r\"realDonaldTrump\", \"Donald Trump\", tweet)\n", " # tweet = re.sub(r\"DavidVonderhaar\", \"David Vonderhaar\", tweet)\n", " # tweet = re.sub(r\"CecilTheLion\", \"Cecil The Lion\", tweet)\n", " # tweet = re.sub(r\"weathernetwork\", \"weather network\", tweet)\n", " # tweet = re.sub(r\"withBioterrorism # Hashtags and usernames\n", " # tweet = re.sub(r\"IranDeal\", \"Iran Deal\", tweet)\n", " # tweet = re.sub(r\"ArianaGrande\", \"Ariana Grande\", tweet)\n", " # tweet = re.sub(r\"camilacabello97\", \"camila cabello\", tweet) \n", " # tweet = re.sub(r\"RondaRousey\", \"Ronda Rousey\", tweet) \n", " # tweet = re.sub(r\"MTVHottest\", \"MTV Hottest\", tweet)\n", " # tweet = re.sub(r\"TrapMusic\", \"Trap Music\", tweet)\n", " # tweet = re.sub(r\"ProphetMuhammad\", \"Prophet Muhammad\", tweet)\n", " # tweet = re.sub(r\"PantherAttack\", \"Panther Attack\", tweet)\n", " # tweet = re.sub(r\"StrategicPatience\", \"Strategic Patience\", tweet)\n", " # tweet = re.sub(r\"socialnews\", \"social news\", tweet)\n", " # tweet = re.sub(r\"NASAHurricane\", \"NASA Hurricane\", tweet)\n", " # tweet = re.sub(r\"onlinecommunities\", \"online communities\", tweet)\n", " # tweet = &use\", \"with Bioterrorism & use\", tweet)\n", " # tweet = re.sub(r\"Hostage&2\", \"Hostage & 2\", tweet)\n", " # tweet = re.sub(r\"GOPDebate\", \"GOP Debate\", tweet)\n", " # tweet = re.sub(r\"RickPerry\", \"Rick Perry\", tweet)\n", " # tweet = re.sub(r\"frontpage\", \"front page\", tweet)\n", " # tweet = re.sub(r\"NewsInTweets\", \"News In Tweets\", tweet)\n", " # tweet = re.sub(r\"ViralSpell\", \"Viral Spell\", tweet)\n", " # tweet = re.sub(r\"til_now\", \"until now\", tweet)\n", " # tweet = re.sub(r\"volcanoinRussia\", \"volcano in Russia\", tweet)\n", " # tweet = re.sub(r\"ZippedNews\", \"Zipped News\", tweet)\n", " # tweet = re.sub(r\"MicheleBachman\", \"Michele Bachman\", tweet)\n", " # tweet = re.sub(r\"53inch\", \"53 inch\", tweet)\n", " # tweet = re.sub(r\"KerrickTrial\", \"Kerrick Trial\", tweet)\n", " # tweet = re.sub(r\"abstorm\", \"Alberta Storm\", tweet)\n", " # tweet = re.sub(r\"Beyhive\", \"Beyonce hive\", tweet)\n", " # tweet = re.sub(r\"IDFire\", \"Idaho Fire\", tweet)\n", " # tweet = re.sub(r\"DETECTADO\", \"Detected\", tweet)\n", " # tweet = re.sub(r\"RockyFire\", \"Rocky Fire\", tweet)\n", " # tweet = re.sub(r\"Listen/Buy\", \"Listen / Buy\", tweet)\n", " # tweet = re.sub(r\"NickCannon\", \"Nick Cannon\", tweet)\n", " # tweet = re.sub(r\"FaroeIslands\", \"Faroe Islands\", tweet)\n", " # tweet = re.sub(r\"yycstorm\", \"Calgary Storm\", tweet)\n", " # tweet = re.sub(r\"IDPs:\", \"Internally Displaced People :\", tweet)\n", " # tweet = re.sub(r\"ArtistsUnited\", \"Artists United\", tweet)\n", " # tweet = re.sub(r\"ClaytonBryant\", \"Clayton Bryant\", tweet)\n", " # tweet = re.sub(r\"jimmyfallon\", \"jimmy fallon\", tweet)\n", " # tweet = re.sub(r\"justinbieber\", \"justin bieber\", tweet) \n", " # tweet = re.sub(r\"UTC2015\", \"UTC 2015\", tweet)\n", " # tweet = re.sub(r\"Time2015\", \"Time 2015\", tweet)\n", " # tweet = re.sub(r\"djicemoon\", \"dj icemoon\", tweet)\n", " # tweet = re.sub(r\"LivingSafely\", \"Living Safely\", tweet)\n", " # tweet = re.sub(r\"FIFA16\", \"Fifa 2016\", tweet)\n", " # tweet = re.sub(r\"thisiswhywecanthavenicethings\", \"this is why we cannot have nice things\", tweet)\n", " # tweet = re.sub(r\"bbcnews\", \"bbc news\", tweet)\n", " # tweet = re.sub(r\"UndergroundRailraod\", \"Underground Railraod\", tweet)\n", " # tweet = re.sub(r\"c4news\", \"c4 news\", tweet)\n", " # tweet = re.sub(r\"OBLITERATION\", \"obliteration\", tweet)\n", " # tweet = re.sub(r\"MUDSLIDE\", \"mudslide\", tweet)\n", " # tweet = re.sub(r\"NoSurrender\", \"No Surrender\", tweet)\n", " # tweet = re.sub(r\"NotExplained\", \"Not Explained\", tweet)\n", " # tweet = re.sub(r\"greatbritishbakeoff\", \"great british bake off\", tweet)\n", " # tweet = re.sub(r\"LondonFire\", \"London Fire\", tweet)\n", " # tweet = re.sub(r\"KOTAWeather\", \"KOTA Weather\", tweet)\n", " # tweet = re.sub(r\"LuchaUnderground\", \"Lucha Underground\", tweet)\n", " # tweet = re.sub(r\"KOIN6News\", \"KOIN 6 News\", tweet)\n", " # tweet = re.sub(r\"LiveOnK2\", \"Live On K2\", tweet)\n", " # tweet = re.sub(r\"9NewsGoldCoast\", \"9 News Gold Coast\", tweet)\n", " # tweet = re.sub(r\"nikeplus\", \"nike plus\", tweet)\n", " # tweet = re.sub(r\"david_cameron\", \"David Cameron\", tweet)\n", " # tweet = re.sub(r\"peterjukes\", \"Peter Jukes\", tweet)\n", " # tweet = re.sub(r\"JamesMelville\", \"James Melville\", tweet)\n", " # tweet = re.sub(r\"megynkelly\", \"Megyn Kelly\", tweet)\n", " # tweet = re.sub(r\"cnewslive\", \"C News Live\", tweet)\n", " # tweet = re.sub(r\"JamaicaObserver\", \"Jamaica Observer\", tweet)\n", " # tweet = re.sub(r\"TweetLikeItsSeptember11th2001\", \"Tweet like it is september 11th 2001\", tweet)\n", " # tweet = re.sub(r\"cbplawyers\", \"cbp lawyers\", tweet)\n", " # tweet = re.sub(r\"fewmoretweets\", \"few more tweets\", tweet)\n", " # tweet = re.sub(r\"BlackLivesMatter\", \"Black Lives Matter\", tweet)\n", " # tweet = re.sub(r\"cjoyner\", \"Chris Joyner\", tweet)\n", " # tweet = re.sub(r\"ENGvAUS\", \"England vs Australia\", tweet)\n", " # tweet = re.sub(r\"ScottWalker\", \"Scott Walker\", tweet)\n", " # tweet = re.sub(r\"MikeParrActor\", \"Michael Parr\", tweet)\n", " # tweet = re.sub(r\"4PlayThursdays\", \"Foreplay Thursdays\", tweet)\n", " # tweet = re.sub(r\"TGF2015\", \"Tontitown Grape Festival\", tweet)\n", " # tweet = re.sub(r\"realmandyrain\", \"Mandy Rain\", tweet)\n", " # tweet = re.sub(r\"GraysonDolan\", \"Grayson Dolan\", tweet)\n", " # tweet = re.sub(r\"ApolloBrown\", \"Apollo Brown\", tweet)\n", " # tweet = re.sub(r\"saddlebrooke\", \"Saddlebrooke\", tweet)\n", " # tweet = re.sub(r\"TontitownGrape\", \"Tontitown Grape\", tweet)\n", " # tweet = re.sub(r\"AbbsWinston\", \"Abbs Winston\", tweet)\n", " # tweet = re.sub(r\"ShaunKing\", \"Shaun King\", tweet)\n", " # tweet = re.sub(r\"MeekMill\", \"Meek Mill\", tweet)\n", " # tweet = re.sub(r\"TornadoGiveaway\", \"Tornado Giveaway\", tweet)\n", " # tweet = re.sub(r\"GRupdates\", \"GR updates\", tweet)\n", " # tweet = re.sub(r\"SouthDowns\", \"South Downs\", tweet)\n", " # tweet = re.sub(r\"braininjury\", \"brain injury\", tweet)\n", " # tweet = re.sub(r\"auspol\", \"Australian politics\", tweet)\n", " # tweet = re.sub(r\"PlannedParenthood\", \"Planned Parenthood\", tweet)\n", " # tweet = re.sub(r\"calgaryweather\", \"Calgary Weather\", tweet)\n", " # tweet = re.sub(r\"weallheartonedirection\", \"we all heart one direction\", tweet)\n", " # tweet = re.sub(r\"edsheeran\", \"Ed Sheeran\", tweet)\n", " # tweet = re.sub(r\"TrueHeroes\", \"True Heroes\", tweet)\n", " # tweet = re.sub(r\"S3XLEAK\", \"sex leak\", tweet)\n", " # tweet = re.sub(r\"ComplexMag\", \"Complex Magazine\", tweet)\n", " # tweet = re.sub(r\"TheAdvocateMag\", \"The Advocate Magazine\", tweet)\n", " # tweet = re.sub(r\"CityofCalgary\", \"City of Calgary\", tweet)\n", " # tweet = re.sub(r\"EbolaOutbreak\", \"Ebola Outbreak\", tweet)\n", " # tweet = re.sub(r\"SummerFate\", \"Summer Fate\", tweet)\n", " # tweet = re.sub(r\"RAmag\", \"Royal Academy Magazine\", tweet)\n", " # tweet = re.sub(r\"offers2go\", \"offers to go\", tweet)\n", " # tweet = re.sub(r\"foodscare\", \"food scare\", tweet)\n", " # tweet = re.sub(r\"MNPDNashville\", \"Metropolitan Nashville Police Department\", tweet)\n", " # tweet = re.sub(r\"TfLBusAlerts\", \"TfL Bus Alerts\", tweet)\n", " # tweet = re.sub(r\"GamerGate\", \"Gamer Gate\", tweet)\n", " # tweet = re.sub(r\"IHHen\", \"Humanitarian Relief\", tweet)\n", " # tweet = re.sub(r\"spinningbot\", \"spinning bot\", tweet)\n", " # tweet = re.sub(r\"ModiMinistry\", \"Modi Ministry\", tweet)\n", " # tweet = re.sub(r\"TAXIWAYS\", \"taxi ways\", tweet)\n", " # tweet = re.sub(r\"Calum5SOS\", \"Calum Hood\", tweet)\n", " # tweet = re.sub(r\"po_st\", \"po.st\", tweet)\n", " # tweet = re.sub(r\"scoopit\", \"scoop.it\", tweet)\n", " # tweet = re.sub(r\"UltimaLucha\", \"Ultima Lucha\", tweet)\n", " # tweet = re.sub(r\"JonathanFerrell\", \"Jonathan Ferrell\", tweet)\n", " # tweet = re.sub(r\"aria_ahrary\", \"Aria Ahrary\", tweet)\n", " # tweet = re.sub(r\"rapidcity\", \"Rapid City\", tweet)\n", " # tweet = re.sub(r\"OutBid\", \"outbid\", tweet)\n", " # tweet = re.sub(r\"lavenderpoetrycafe\", \"lavender poetry cafe\", tweet)\n", " # tweet = re.sub(r\"EudryLantiqua\", \"Eudry Lantiqua\", tweet)\n", " # tweet = re.sub(r\"15PM\", \"15 PM\", tweet)\n", " # tweet = re.sub(r\"OriginalFunko\", \"Funko\", tweet)\n", " # tweet = re.sub(r\"rightwaystan\", \"Richard Tan\", tweet)\n", " # tweet = re.sub(r\"CindyNoonan\", \"Cindy Noonan\", tweet)\n", " # tweet = re.sub(r\"RT_America\", \"RT America\", tweet)\n", " # tweet = re.sub(r\"narendramodi\", \"Narendra Modi\", tweet)\n", " # tweet = re.sub(r\"BakeOffFriends\", \"Bake Off Friends\", tweet)\n", " # tweet = re.sub(r\"TeamHendrick\", \"Hendrick Motorsports\", tweet)\n", " # tweet = re.sub(r\"alexbelloli\", \"Alex Belloli\", tweet)\n", " # tweet = re.sub(r\"itsjustinstuart\", \"Justin Stuart\", tweet)\n", " # tweet = re.sub(r\"gunsense\", \"gun sense\", tweet)\n", " # tweet = re.sub(r\"DebateQuestionsWeWantToHear\", \"debate questions we want to hear\", tweet)\n", " # tweet = re.sub(r\"RoyalCarribean\", \"Royal Carribean\", tweet)\n", " # tweet = re.sub(r\"samanthaturne19\", \"Samantha Turner\", tweet)\n", " # tweet = re.sub(r\"JonVoyage\", \"Jon Stewart\", tweet)\n", " # tweet = re.sub(r\"renew911health\", \"renew 911 health\", tweet)\n", " # tweet = re.sub(r\"SuryaRay\", \"Surya Ray\", tweet)\n", " # tweet = re.sub(r\"pattonoswalt\", \"Patton Oswalt\", tweet)\n", " # tweet = re.sub(r\"minhazmerchant\", \"Minhaz Merchant\", tweet)\n", " # tweet = re.sub(r\"TLVFaces\", \"Israel Diaspora Coalition\", tweet)\n", " # tweet = re.sub(r\"pmarca\", \"Marc Andreessen\", tweet)\n", " # tweet = re.sub(r\"pdx911\", \"Portland Police\", tweet)\n", " # tweet = re.sub(r\"jamaicaplain\", \"Jamaica Plain\", tweet)\n", " # tweet = re.sub(r\"Japton\", \"Arkansas\", tweet)\n", " # tweet = re.sub(r\"RouteComplex\", \"Route Complex\", tweet)\n", " # tweet = re.sub(r\"INSubcontinent\", \"Indian Subcontinent\", tweet)\n", " # tweet = re.sub(r\"NJTurnpike\", \"New Jersey Turnpike\", tweet)\n", " # tweet = re.sub(r\"Politifiact\", \"PolitiFact\", tweet)\n", " # tweet = re.sub(r\"Hiroshima70\", \"Hiroshima\", tweet)\n", " # tweet = re.sub(r\"GMMBC\", \"Greater Mt Moriah Baptist Church\", tweet)\n", " # tweet = re.sub(r\"versethe\", \"verse the\", tweet)\n", " # tweet = re.sub(r\"TubeStrike\", \"Tube Strike\", tweet)\n", " # tweet = re.sub(r\"MissionHills\", \"Mission Hills\", tweet)\n", " # tweet = re.sub(r\"ProtectDenaliWolves\", \"Protect Denali Wolves\", tweet)\n", " # tweet = re.sub(r\"NANKANA\", \"Nankana\", tweet)\n", " # tweet = re.sub(r\"SAHIB\", \"Sahib\", tweet)\n", " # tweet = re.sub(r\"PAKPATTAN\", \"Pakpattan\", tweet)\n", " # tweet = re.sub(r\"Newz_Sacramento\", \"News Sacramento\", tweet)\n", " # tweet = re.sub(r\"gofundme\", \"go fund me\", tweet)\n", " # tweet = re.sub(r\"pmharper\", \"Stephen Harper\", tweet)\n", " # tweet = re.sub(r\"IvanBerroa\", \"Ivan Berroa\", tweet)\n", " # tweet = re.sub(r\"LosDelSonido\", \"Los Del Sonido\", tweet)\n", " # tweet = re.sub(r\"bancodeseries\", \"banco de series\", tweet)\n", " # tweet = re.sub(r\"timkaine\", \"Tim Kaine\", tweet)\n", " # tweet = re.sub(r\"IdentityTheft\", \"Identity Theft\", tweet)\n", " # tweet = re.sub(r\"AllLivesMatter\", \"All Lives Matter\", tweet)\n", " # tweet = re.sub(r\"mishacollins\", \"Misha Collins\", tweet)\n", " # tweet = re.sub(r\"BillNeelyNBC\", \"Bill Neely\", tweet)\n", " # tweet = re.sub(r\"BeClearOnCancer\", \"be clear on cancer\", tweet)\n", " # tweet = re.sub(r\"Kowing\", \"Knowing\", tweet)\n", " # tweet = re.sub(r\"ScreamQueens\", \"Scream Queens\", tweet)\n", " # tweet = re.sub(r\"AskCharley\", \"Ask Charley\", tweet)\n", " # tweet = re.sub(r\"BlizzHeroes\", \"Heroes of the Storm\", tweet)\n", " # tweet = re.sub(r\"BradleyBrad47\", \"Bradley Brad\", tweet)\n", " # tweet = re.sub(r\"HannaPH\", \"Typhoon Hanna\", tweet)\n", " # tweet = re.sub(r\"meinlcymbals\", \"MEINL Cymbals\", tweet)\n", " # tweet = re.sub(r\"Ptbo\", \"Peterborough\", tweet)\n", " # tweet = re.sub(r\"cnnbrk\", \"CNN Breaking News\", tweet)\n", " # tweet = re.sub(r\"IndianNews\", \"Indian News\", tweet)\n", " # tweet = re.sub(r\"savebees\", \"save bees\", tweet)\n", " # tweet = re.sub(r\"GreenHarvard\", \"Green Harvard\", tweet)\n", " # tweet = re.sub(r\"StandwithPP\", \"Stand with planned parenthood\", tweet)\n", " # tweet = re.sub(r\"hermancranston\", \"Herman Cranston\", tweet)\n", " # tweet = re.sub(r\"WMUR9\", \"WMUR-TV\", tweet)\n", " # tweet = re.sub(r\"RockBottomRadFM\", \"Rock Bottom Radio\", tweet)\n", " # tweet = re.sub(r\"ameenshaikh3\", \"Ameen Shaikh\", tweet)\n", " # tweet = re.sub(r\"ProSyn\", \"Project Syndicate\", tweet)\n", " # tweet = re.sub(r\"Daesh\", \"ISIS\", tweet)\n", " # tweet = re.sub(r\"s2g\", \"swear to god\", tweet)\n", " # tweet = re.sub(r\"listenlive\", \"listen live\", tweet)\n", " # tweet = re.sub(r\"CDCgov\", \"Centers for Disease Control and Prevention\", tweet)\n", " # tweet = re.sub(r\"FoxNew\", \"Fox News\", tweet)\n", " # tweet = re.sub(r\"CBSBigBrother\", \"Big Brother\", tweet)\n", " # tweet = re.sub(r\"JulieDiCaro\", \"Julie DiCaro\", tweet)\n", " # tweet = re.sub(r\"theadvocatemag\", \"The Advocate Magazine\", tweet)\n", " # tweet = re.sub(r\"RohnertParkDPS\", \"Rohnert Park Police Department\", tweet)\n", " # tweet = re.sub(r\"THISIZBWRIGHT\", \"Bonnie Wright\", tweet)\n", " # tweet = re.sub(r\"Popularmmos\", \"Popular MMOs\", tweet)\n", " # tweet = re.sub(r\"WildHorses\", \"Wild Horses\", tweet)\n", " # tweet = re.sub(r\"FantasticFour\", \"Fantastic Four\", tweet)\n", " # tweet = re.sub(r\"HORNDALE\", \"Horndale\", tweet)\n", " # tweet = re.sub(r\"PINER\", \"Piner\", tweet)\n", " # tweet = re.sub(r\"BathAndNorthEastSomerset\", \"Bath and North East Somerset\", tweet)\n", " # tweet = re.sub(r\"thatswhatfriendsarefor\", \"that is what friends are for\", tweet)\n", " # tweet = re.sub(r\"residualincome\", \"residual income\", tweet)\n", " # tweet = re.sub(r\"YahooNewsDigest\", \"Yahoo News Digest\", tweet)\n", " # tweet = re.sub(r\"MalaysiaAirlines\", \"Malaysia Airlines\", tweet)\n", " # tweet = re.sub(r\"AmazonDeals\", \"Amazon Deals\", tweet)\n", " # tweet = re.sub(r\"MissCharleyWebb\", \"Charley Webb\", tweet)\n", " # tweet = re.sub(r\"shoalstraffic\", \"shoals traffic\", tweet)\n", " # tweet = re.sub(r\"GeorgeFoster72\", \"George Foster\", tweet)\n", " # tweet = re.sub(r\"pop2015\", \"pop 2015\", tweet)\n", " # tweet = re.sub(r\"_PokemonCards_\", \"Pokemon Cards\", tweet)\n", " # tweet = re.sub(r\"DianneG\", \"Dianne Gallagher\", tweet)\n", " # tweet = re.sub(r\"KashmirConflict\", \"Kashmir Conflict\", tweet)\n", " # tweet = re.sub(r\"BritishBakeOff\", \"British Bake Off\", tweet)\n", " # tweet = re.sub(r\"FreeKashmir\", \"Free Kashmir\", tweet)\n", " # tweet = re.sub(r\"mattmosley\", \"Matt Mosley\", tweet)\n", " # tweet = re.sub(r\"BishopFred\", \"Bishop Fred\", tweet)\n", " # tweet = re.sub(r\"EndConflict\", \"End Conflict\", tweet)\n", " # tweet = re.sub(r\"EndOccupation\", \"End Occupation\", tweet)\n", " # tweet = re.sub(r\"UNHEALED\", \"unhealed\", tweet)\n", " # tweet = re.sub(r\"CharlesDagnall\", \"Charles Dagnall\", tweet)\n", " # tweet = re.sub(r\"Latestnews\", \"Latest news\", tweet)\n", " # tweet = re.sub(r\"KindleCountdown\", \"Kindle Countdown\", tweet)\n", " # tweet = re.sub(r\"NoMoreHandouts\", \"No More Handouts\", tweet)\n", " # tweet = re.sub(r\"datingtips\", \"dating tips\", tweet)\n", " # tweet = re.sub(r\"charlesadler\", \"Charles Adler\", tweet)\n", " # tweet = re.sub(r\"twia\", \"Texas Windstorm Insurance Association\", tweet)\n", " # tweet = re.sub(r\"txlege\", \"Texas Legislature\", tweet)\n", " # tweet = re.sub(r\"WindstormInsurer\", \"Windstorm Insurer\", tweet)\n", " # tweet = re.sub(r\"Newss\", \"News\", tweet)\n", " # tweet = re.sub(r\"hempoil\", \"hemp oil\", tweet)\n", " # tweet = re.sub(r\"CommoditiesAre\", \"Commodities are\", tweet)\n", " # tweet = re.sub(r\"tubestrike\", \"tube strike\", tweet)\n", " # tweet = re.sub(r\"JoeNBC\", \"Joe Scarborough\", tweet)\n", " # tweet = re.sub(r\"LiteraryCakes\", \"Literary Cakes\", tweet)\n", " # tweet = re.sub(r\"TI5\", \"The International 5\", tweet)\n", " # tweet = re.sub(r\"thehill\", \"the hill\", tweet)\n", " # tweet = re.sub(r\"3others\", \"3 others\", tweet)\n", " # tweet = re.sub(r\"stighefootball\", \"Sam Tighe\", tweet)\n", " # tweet = re.sub(r\"whatstheimportantvideo\", \"what is the important video\", tweet)\n", " # tweet = re.sub(r\"ClaudioMeloni\", \"Claudio Meloni\", tweet)\n", " # tweet = re.sub(r\"DukeSkywalker\", \"Duke Skywalker\", tweet)\n", " # tweet = re.sub(r\"carsonmwr\", \"Fort Carson\", tweet)\n", " # tweet = re.sub(r\"offdishduty\", \"off dish duty\", tweet)\n", " # tweet = re.sub(r\"andword\", \"and word\", tweet)\n", " # tweet = re.sub(r\"rhodeisland\", \"Rhode Island\", tweet)\n", " # tweet = re.sub(r\"easternoregon\", \"Eastern Oregon\", tweet)\n", " # tweet = re.sub(r\"WAwildfire\", \"Washington Wildfire\", tweet)\n", " # tweet = re.sub(r\"fingerrockfire\", \"Finger Rock Fire\", tweet)\n", " # tweet = re.sub(r\"57am\", \"57 am\", tweet)\n", " # tweet = re.sub(r\"fingerrockfire\", \"Finger Rock Fire\", tweet)\n", " # tweet = re.sub(r\"JacobHoggard\", \"Jacob Hoggard\", tweet)\n", " # tweet = re.sub(r\"newnewnew\", \"new new new\", tweet)\n", " # tweet = re.sub(r\"under50\", \"under 50\", tweet)\n", " # tweet = re.sub(r\"getitbeforeitsgone\", \"get it before it is gone\", tweet)\n", " # tweet = re.sub(r\"freshoutofthebox\", \"fresh out of the box\", tweet)\n", " # tweet = re.sub(r\"amwriting\", \"am writing\", tweet)\n", " # tweet = re.sub(r\"Bokoharm\", \"Boko Haram\", tweet)\n", " # tweet = re.sub(r\"Nowlike\", \"Now like\", tweet)\n", " # tweet = re.sub(r\"seasonfrom\", \"season from\", tweet)\n", " # tweet = re.sub(r\"epicente\", \"epicenter\", tweet)\n", " # tweet = re.sub(r\"epicenterr\", \"epicenter\", tweet)\n", " # tweet = re.sub(r\"sicklife\", \"sick life\", tweet)\n", " # tweet = re.sub(r\"yycweather\", \"Calgary Weather\", tweet)\n", " # tweet = re.sub(r\"calgarysun\", \"Calgary Sun\", tweet)\n", " # tweet = re.sub(r\"approachng\", \"approaching\", tweet)\n", " # tweet = re.sub(r\"evng\", \"evening\", tweet)\n", " # tweet = re.sub(r\"Sumthng\", \"something\", tweet)\n", " # tweet = re.sub(r\"EllenPompeo\", \"Ellen Pompeo\", tweet)\n", " # tweet = re.sub(r\"shondarhimes\", \"Shonda Rhimes\", tweet)\n", " # tweet = re.sub(r\"ABCNetwork\", \"ABC Network\", tweet)\n", " # tweet = re.sub(r\"SushmaSwaraj\", \"Sushma Swaraj\", tweet)\n", " # tweet = re.sub(r\"pray4japan\", \"Pray for Japan\", tweet)\n", " # tweet = re.sub(r\"hope4japan\", \"Hope for Japan\", tweet)\n", " # tweet = re.sub(r\"Illusionimagess\", \"Illusion images\", tweet)\n", " # tweet = re.sub(r\"SummerUnderTheStars\", \"Summer Under The Stars\", tweet)\n", " # tweet = re.sub(r\"ShallWeDance\", \"Shall We Dance\", tweet)\n", " # tweet = re.sub(r\"TCMParty\", \"TCM Party\", tweet)\n", " # tweet = re.sub(r\"marijuananews\", \"marijuana news\", tweet)\n", " # tweet = re.sub(r\"onbeingwithKristaTippett\", \"on being with Krista Tippett\", tweet)\n", " # tweet = re.sub(r\"Beingtweets\", \"Being tweets\", tweet)\n", " # tweet = re.sub(r\"newauthors\", \"new authors\", tweet)\n", " # tweet = re.sub(r\"remedyyyy\", \"remedy\", tweet)\n", " # tweet = re.sub(r\"44PM\", \"44 PM\", tweet)\n", " # tweet = re.sub(r\"HeadlinesApp\", \"Headlines App\", tweet)\n", " # tweet = re.sub(r\"40PM\", \"40 PM\", tweet)\n", " # tweet = re.sub(r\"myswc\", \"Severe Weather Center\", tweet)\n", " # tweet = re.sub(r\"ithats\", \"that is\", tweet)\n", " # tweet = re.sub(r\"icouldsitinthismomentforever\", \"I could sit in this moment forever\", tweet)\n", " # tweet = re.sub(r\"FatLoss\", \"Fat Loss\", tweet)\n", " # tweet = re.sub(r\"02PM\", \"02 PM\", tweet)\n", " # tweet = re.sub(r\"MetroFmTalk\", \"Metro Fm Talk\", tweet)\n", " # tweet = re.sub(r\"Bstrd\", \"bastard\", tweet)\n", " # tweet = re.sub(r\"bldy\", \"bloody\", tweet)\n", " # tweet = re.sub(r\"MetrofmTalk\", \"Metro Fm Talk\", tweet)\n", " # tweet = re.sub(r\"terrorismturn\", \"terrorism turn\", tweet)\n", " # tweet = re.sub(r\"BBCNewsAsia\", \"BBC News Asia\", tweet)\n", " # tweet = re.sub(r\"BehindTheScenes\", \"Behind The Scenes\", tweet)\n", " # tweet = re.sub(r\"GeorgeTakei\", \"George Takei\", tweet)\n", " # tweet = re.sub(r\"WomensWeeklyMag\", \"Womens Weekly Magazine\", tweet)\n", " # tweet = re.sub(r\"SurvivorsGuidetoEarth\", \"Survivors Guide to Earth\", tweet)\n", " # tweet = re.sub(r\"incubusband\", \"incubus band\", tweet)\n", " # tweet = re.sub(r\"Babypicturethis\", \"Baby picture this\", tweet)\n", " # tweet = re.sub(r\"BombEffects\", \"Bomb Effects\", tweet)\n", " # tweet = re.sub(r\"win10\", \"Windows 10\", tweet)\n", " # tweet = re.sub(r\"idkidk\", \"I do not know I do not know\", tweet)\n", " # tweet = re.sub(r\"TheWalkingDead\", \"The Walking Dead\", tweet)\n", " # tweet = re.sub(r\"amyschumer\", \"Amy Schumer\", tweet)\n", " # tweet = re.sub(r\"crewlist\", \"crew list\", tweet)\n", " # tweet = re.sub(r\"Erdogans\", \"Erdogan\", tweet)\n", " # tweet = re.sub(r\"BBCLive\", \"BBC Live\", tweet)\n", " # tweet = re.sub(r\"TonyAbbottMHR\", \"Tony Abbott\", tweet)\n", " # tweet = re.sub(r\"paulmyerscough\", \"Paul Myerscough\", tweet)\n", " # tweet = re.sub(r\"georgegallagher\", \"George Gallagher\", tweet)\n", " # tweet = re.sub(r\"JimmieJohnson\", \"Jimmie Johnson\", tweet)\n", " # tweet = re.sub(r\"pctool\", \"pc tool\", tweet)\n", " # tweet = re.sub(r\"DoingHashtagsRight\", \"Doing Hashtags Right\", tweet)\n", " # tweet = re.sub(r\"ThrowbackThursday\", \"Throwback Thursday\", tweet)\n", " # tweet = re.sub(r\"SnowBackSunday\", \"Snowback Sunday\", tweet)\n", " # tweet = re.sub(r\"LakeEffect\", \"Lake Effect\", tweet)\n", " # tweet = re.sub(r\"RTphotographyUK\", \"Richard Thomas Photography UK\", tweet)\n", " # tweet = re.sub(r\"BigBang_CBS\", \"Big Bang CBS\", tweet)\n", " # tweet = re.sub(r\"writerslife\", \"writers life\", tweet)\n", " # tweet = re.sub(r\"NaturalBirth\", \"Natural Birth\", tweet)\n", " # tweet = re.sub(r\"UnusualWords\", \"Unusual Words\", tweet)\n", " # tweet = re.sub(r\"wizkhalifa\", \"Wiz Khalifa\", tweet)\n", " # tweet = re.sub(r\"acreativedc\", \"a creative DC\", tweet)\n", " # tweet = re.sub(r\"vscodc\", \"vsco DC\", tweet)\n", " # tweet = re.sub(r\"VSCOcam\", \"vsco camera\", tweet)\n", " # tweet = re.sub(r\"TheBEACHDC\", \"The beach DC\", tweet)\n", " # tweet = re.sub(r\"buildingmuseum\", \"building museum\", tweet)\n", " # tweet = re.sub(r\"WorldOil\", \"World Oil\", tweet)\n", " # tweet = re.sub(r\"redwedding\", \"red wedding\", tweet)\n", " # tweet = re.sub(r\"AmazingRaceCanada\", \"Amazing Race Canada\", tweet)\n", " # tweet = re.sub(r\"WakeUpAmerica\", \"Wake Up America\", tweet)\n", " # tweet = re.sub(r\"\\\\Allahuakbar\\\\\", \"Allahu Akbar\", tweet)\n", " # tweet = re.sub(r\"bleased\", \"blessed\", tweet)\n", " # tweet = re.sub(r\"nigeriantribune\", \"Nigerian Tribune\", tweet)\n", " # tweet = re.sub(r\"HIDEO_KOJIMA_EN\", \"Hideo Kojima\", tweet)\n", " # tweet = re.sub(r\"FusionFestival\", \"Fusion Festival\", tweet)\n", " # tweet = re.sub(r\"50Mixed\", \"50 Mixed\", tweet)\n", " # tweet = re.sub(r\"NoAgenda\", \"No Agenda\", tweet)\n", " # tweet = re.sub(r\"WhiteGenocide\", \"White Genocide\", tweet)\n", " # tweet = re.sub(r\"dirtylying\", \"dirty lying\", tweet)\n", " # tweet = re.sub(r\"SyrianRefugees\", \"Syrian Refugees\", tweet)\n", " # tweet = re.sub(r\"changetheworld\", \"change the world\", tweet)\n", " # tweet = re.sub(r\"Ebolacase\", \"Ebola case\", tweet)\n", " # tweet = re.sub(r\"mcgtech\", \"mcg technologies\", tweet)\n", " # tweet = re.sub(r\"withweapons\", \"with weapons\", tweet)\n", " # tweet = re.sub(r\"advancedwarfare\", \"advanced warfare\", tweet)\n", " # tweet = re.sub(r\"letsFootball\", \"let us Football\", tweet)\n", " # tweet = re.sub(r\"LateNiteMix\", \"late night mix\", tweet)\n", " # tweet = re.sub(r\"PhilCollinsFeed\", \"Phil Collins\", tweet)\n", " # tweet = re.sub(r\"RudyHavenstein\", \"Rudy Havenstein\", tweet)\n", " # tweet = re.sub(r\"22PM\", \"22 PM\", tweet)\n", " # tweet = re.sub(r\"54am\", \"54 AM\", tweet)\n", " # tweet = re.sub(r\"38am\", \"38 AM\", tweet)\n", " # tweet = re.sub(r\"OldFolkExplainStuff\", \"Old Folk Explain Stuff\", tweet)\n", " # tweet = re.sub(r\"BlacklivesMatter\", \"Black Lives Matter\", tweet)\n", " # tweet = re.sub(r\"InsaneLimits\", \"Insane Limits\", tweet)\n", " # tweet = re.sub(r\"youcantsitwithus\", \"you cannot sit with us\", tweet)\n", " # tweet = re.sub(r\"2k15\", \"2015\", tweet)\n", " # tweet = re.sub(r\"TheIran\", \"Iran\", tweet)\n", " # tweet = re.sub(r\"JimmyFallon\", \"Jimmy Fallon\", tweet)\n", " # tweet = re.sub(r\"AlbertBrooks\", \"Albert Brooks\", tweet)\n", " # tweet = re.sub(r\"defense_news\", \"defense news\", tweet)\n", " # tweet = re.sub(r\"nuclearrcSA\", \"Nuclear Risk Control Self Assessment\", tweet)\n", " # tweet = re.sub(r\"Auspol\", \"Australia Politics\", tweet)\n", " # tweet = re.sub(r\"NuclearPower\", \"Nuclear Power\", tweet)\n", " # tweet = re.sub(r\"WhiteTerrorism\", \"White Terrorism\", tweet)\n", " # tweet = re.sub(r\"truthfrequencyradio\", \"Truth Frequency Radio\", tweet)\n", " # tweet = re.sub(r\"ErasureIsNotEquality\", \"Erasure is not equality\", tweet)\n", " # tweet = re.sub(r\"ProBonoNews\", \"Pro Bono News\", tweet)\n", " # tweet = re.sub(r\"JakartaPost\", \"Jakarta Post\", tweet)\n", " # tweet = re.sub(r\"toopainful\", \"too painful\", tweet)\n", " # tweet = re.sub(r\"melindahaunton\", \"Melinda Haunton\", tweet)\n", " # tweet = re.sub(r\"NoNukes\", \"No Nukes\", tweet)\n", " # tweet = re.sub(r\"curryspcworld\", \"Currys PC World\", tweet)\n", " # tweet = re.sub(r\"ineedcake\", \"I need cake\", tweet)\n", " # tweet = re.sub(r\"blackforestgateau\", \"black forest gateau\", tweet)\n", " # tweet = re.sub(r\"BBCOne\", \"BBC One\", tweet)\n", " # tweet = re.sub(r\"AlexxPage\", \"Alex Page\", tweet)\n", " # tweet = re.sub(r\"jonathanserrie\", \"Jonathan Serrie\", tweet)\n", " # tweet = re.sub(r\"SocialJerkBlog\", \"Social Jerk Blog\", tweet)\n", " # tweet = re.sub(r\"ChelseaVPeretti\", \"Chelsea Peretti\", tweet)\n", " # tweet = re.sub(r\"irongiant\", \"iron giant\", tweet)\n", " # tweet = re.sub(r\"RonFunches\", \"Ron Funches\", tweet)\n", " # tweet = re.sub(r\"TimCook\", \"Tim Cook\", tweet)\n", " # tweet = re.sub(r\"sebastianstanisaliveandwell\", \"Sebastian Stan is alive and well\", tweet)\n", " # tweet = re.sub(r\"Madsummer\", \"Mad summer\", tweet)\n", " # tweet = re.sub(r\"NowYouKnow\", \"Now you know\", tweet)\n", " # tweet = re.sub(r\"concertphotography\", \"concert photography\", tweet)\n", " # tweet = re.sub(r\"TomLandry\", \"Tom Landry\", tweet)\n", " # tweet = re.sub(r\"showgirldayoff\", \"show girl day off\", tweet)\n", " # tweet = re.sub(r\"Yougslavia\", \"Yugoslavia\", tweet)\n", " # tweet = re.sub(r\"QuantumDataInformatics\", \"Quantum Data Informatics\", tweet)\n", " # tweet = re.sub(r\"FromTheDesk\", \"From The Desk\", tweet)\n", " # tweet = re.sub(r\"TheaterTrial\", \"Theater Trial\", tweet)\n", " # tweet = re.sub(r\"CatoInstitute\", \"Cato Institute\", tweet)\n", " # tweet = re.sub(r\"EmekaGift\", \"Emeka Gift\", tweet)\n", " # tweet = re.sub(r\"LetsBe_Rational\", \"Let us be rational\", tweet)\n", " # tweet = re.sub(r\"Cynicalreality\", \"Cynical reality\", tweet)\n", " # tweet = re.sub(r\"FredOlsenCruise\", \"Fred Olsen Cruise\", tweet)\n", " # tweet = re.sub(r\"NotSorry\", \"not sorry\", tweet)\n", " # tweet = re.sub(r\"UseYourWords\", \"use your words\", tweet)\n", " # tweet = re.sub(r\"WordoftheDay\", \"word of the day\", tweet)\n", " # tweet = re.sub(r\"Dictionarycom\", \"Dictionary.com\", tweet)\n", " # tweet = re.sub(r\"TheBrooklynLife\", \"The Brooklyn Life\", tweet)\n", " # tweet = re.sub(r\"jokethey\", \"joke they\", tweet)\n", " # tweet = re.sub(r\"nflweek1picks\", \"NFL week 1 picks\", tweet)\n", " # tweet = re.sub(r\"uiseful\", \"useful\", tweet)\n", " # tweet = re.sub(r\"JusticeDotOrg\", \"The American Association for Justice\", tweet)\n", " # tweet = re.sub(r\"autoaccidents\", \"auto accidents\", tweet)\n", " # tweet = re.sub(r\"SteveGursten\", \"Steve Gursten\", tweet)\n", " # tweet = re.sub(r\"MichiganAutoLaw\", \"Michigan Auto Law\", tweet)\n", " # tweet = re.sub(r\"birdgang\", \"bird gang\", tweet)\n", " # tweet = re.sub(r\"nflnetwork\", \"NFL Network\", tweet)\n", " # tweet = re.sub(r\"NYDNSports\", \"NY Daily News Sports\", tweet)\n", " # tweet = re.sub(r\"RVacchianoNYDN\", \"Ralph Vacchiano NY Daily News\", tweet)\n", " # tweet = re.sub(r\"EdmontonEsks\", \"Edmonton Eskimos\", tweet)\n", " # tweet = re.sub(r\"david_brelsford\", \"David Brelsford\", tweet)\n", " # tweet = re.sub(r\"TOI_India\", \"The Times of India\", tweet)\n", " # tweet = re.sub(r\"hegot\", \"he got\", tweet)\n", " # tweet = re.sub(r\"SkinsOn9\", \"Skins on 9\", tweet)\n", " # tweet = re.sub(r\"sothathappened\", \"so that happened\", tweet)\n", " # tweet = re.sub(r\"LCOutOfDoors\", \"LC Out Of Doors\", tweet)\n", " # tweet = re.sub(r\"NationFirst\", \"Nation First\", tweet)\n", " # tweet = re.sub(r\"IndiaToday\", \"India Today\", tweet)\n", " # tweet = re.sub(r\"HLPS\", \"helps\", tweet)\n", " # tweet = re.sub(r\"HOSTAGESTHROSW\", \"hostages throw\", tweet)\n", " # tweet = re.sub(r\"SNCTIONS\", \"sanctions\", tweet)\n", " # tweet = re.sub(r\"BidTime\", \"Bid Time\", tweet)\n", " # tweet = re.sub(r\"crunchysensible\", \"crunchy sensible\", tweet)\n", " # tweet = re.sub(r\"RandomActsOfRomance\", \"Random acts of romance\", tweet)\n", " # tweet = re.sub(r\"MomentsAtHill\", \"Moments at hill\", tweet)\n", " # tweet = re.sub(r\"eatshit\", \"eat shit\", tweet)\n", " # tweet = re.sub(r\"liveleakfun\", \"live leak fun\", tweet)\n", " # tweet = re.sub(r\"SahelNews\", \"Sahel News\", tweet)\n", " # tweet = re.sub(r\"abc7newsbayarea\", \"ABC 7 News Bay Area\", tweet)\n", " # tweet = re.sub(r\"facilitiesmanagement\", \"facilities management\", tweet)\n", " # tweet = re.sub(r\"facilitydude\", \"facility dude\", tweet)\n", " # tweet = re.sub(r\"CampLogistics\", \"Camp logistics\", tweet)\n", " # tweet = re.sub(r\"alaskapublic\", \"Alaska public\", tweet)\n", " # tweet = re.sub(r\"MarketResearch\", \"Market Research\", tweet)\n", " # tweet = re.sub(r\"AccuracyEsports\", \"Accuracy Esports\", tweet)\n", " # tweet = re.sub(r\"TheBodyShopAust\", \"The Body Shop Australia\", tweet)\n", " # tweet = re.sub(r\"yychail\", \"Calgary hail\", tweet)\n", " # tweet = re.sub(r\"yyctraffic\", \"Calgary traffic\", tweet)\n", " # tweet = re.sub(r\"eliotschool\", \"eliot school\", tweet)\n", " # tweet = re.sub(r\"TheBrokenCity\", \"The Broken City\", tweet)\n", " # tweet = re.sub(r\"OldsFireDept\", \"Olds Fire Department\", tweet)\n", " # tweet = re.sub(r\"RiverComplex\", \"River Complex\", tweet)\n", " # tweet = re.sub(r\"fieldworksmells\", \"field work smells\", tweet)\n", " # tweet = re.sub(r\"IranElection\", \"Iran Election\", tweet)\n", " # tweet = re.sub(r\"glowng\", \"glowing\", tweet)\n", " # tweet = re.sub(r\"kindlng\", \"kindling\", tweet)\n", " # tweet = re.sub(r\"riggd\", \"rigged\", tweet)\n", " # tweet = re.sub(r\"slownewsday\", \"slow news day\", tweet)\n", " # tweet = re.sub(r\"MyanmarFlood\", \"Myanmar Flood\", tweet)\n", " # tweet = re.sub(r\"abc7chicago\", \"ABC 7 Chicago\", tweet)\n", " # tweet = re.sub(r\"copolitics\", \"Colorado Politics\", tweet)\n", " # tweet = re.sub(r\"AdilGhumro\", \"Adil Ghumro\", tweet)\n", " # tweet = re.sub(r\"netbots\", \"net bots\", tweet)\n", " # tweet = re.sub(r\"byebyeroad\", \"bye bye road\", tweet)\n", " # tweet = re.sub(r\"massiveflooding\", \"massive flooding\", tweet)\n", " # tweet = re.sub(r\"EndofUS\", \"End of United States\", tweet)\n", " # tweet = re.sub(r\"35PM\", \"35 PM\", tweet)\n", " # tweet = re.sub(r\"greektheatrela\", \"Greek Theatre Los Angeles\", tweet)\n", " # tweet = re.sub(r\"76mins\", \"76 minutes\", tweet)\n", " # tweet = re.sub(r\"publicsafetyfirst\", \"public safety first\", tweet)\n", " # tweet = re.sub(r\"livesmatter\", \"lives matter\", tweet)\n", " # tweet = re.sub(r\"myhometown\", \"my hometown\", tweet)\n", " # tweet = re.sub(r\"tankerfire\", \"tanker fire\", tweet)\n", " # tweet = re.sub(r\"MEMORIALDAY\", \"memorial day\", tweet)\n", " # tweet = re.sub(r\"MEMORIAL_DAY\", \"memorial day\", tweet)\n", " # tweet = re.sub(r\"instaxbooty\", \"instagram booty\", tweet)\n", " # tweet = re.sub(r\"Jerusalem_Post\", \"Jerusalem Post\", tweet)\n", " # tweet = re.sub(r\"WayneRooney_INA\", \"Wayne Rooney\", tweet)\n", " # tweet = re.sub(r\"VirtualReality\", \"Virtual Reality\", tweet)\n", " # tweet = re.sub(r\"OculusRift\", \"Oculus Rift\", tweet)\n", " # tweet = re.sub(r\"OwenJones84\", \"Owen Jones\", tweet)\n", " # tweet = re.sub(r\"jeremycorbyn\", \"Jeremy Corbyn\", tweet)\n", " # tweet = re.sub(r\"paulrogers002\", \"Paul Rogers\", tweet)\n", " # tweet = re.sub(r\"mortalkombatx\", \"Mortal Kombat X\", tweet)\n", " # tweet = re.sub(r\"mortalkombat\", \"Mortal Kombat\", tweet)\n", " # tweet = re.sub(r\"FilipeCoelho92\", \"Filipe Coelho\", tweet)\n", " # tweet = re.sub(r\"OnlyQuakeNews\", \"Only Quake News\", tweet)\n", " # tweet = re.sub(r\"kostumes\", \"costumes\", tweet)\n", " # tweet = re.sub(r\"YEEESSSS\", \"yes\", tweet)\n", " # tweet = re.sub(r\"ToshikazuKatayama\", \"Toshikazu Katayama\", tweet)\n", " # tweet = re.sub(r\"IntlDevelopment\", \"Intl Development\", tweet)\n", " # tweet = re.sub(r\"ExtremeWeather\", \"Extreme Weather\", tweet)\n", " # tweet = re.sub(r\"WereNotGruberVoters\", \"We are not gruber voters\", tweet)\n", " # tweet = re.sub(r\"NewsThousands\", \"News Thousands\", tweet)\n", " # tweet = re.sub(r\"EdmundAdamus\", \"Edmund Adamus\", tweet)\n", " # tweet = re.sub(r\"EyewitnessWV\", \"Eye witness WV\", tweet)\n", " # tweet = re.sub(r\"PhiladelphiaMuseu\", \"Philadelphia Museum\", tweet)\n", " # tweet = re.sub(r\"DublinComicCon\", \"Dublin Comic Con\", tweet)\n", " # tweet = re.sub(r\"NicholasBrendon\", \"Nicholas Brendon\", tweet)\n", " # tweet = re.sub(r\"Alltheway80s\", \"All the way 80s\", tweet)\n", " # tweet = re.sub(r\"FromTheField\", \"From the field\", tweet)\n", " # tweet = re.sub(r\"NorthIowa\", \"North Iowa\", tweet)\n", " # tweet = re.sub(r\"WillowFire\", \"Willow Fire\", tweet)\n", " # tweet = re.sub(r\"MadRiverComplex\", \"Mad River Complex\", tweet)\n", " # tweet = re.sub(r\"feelingmanly\", \"feeling manly\", tweet)\n", " # tweet = re.sub(r\"stillnotoverit\", \"still not over it\", tweet)\n", " # tweet = re.sub(r\"FortitudeValley\", \"Fortitude Valley\", tweet)\n", " # tweet = re.sub(r\"CoastpowerlineTramTr\", \"Coast powerline\", tweet)\n", " # tweet = re.sub(r\"ServicesGold\", \"Services Gold\", tweet)\n", " # tweet = re.sub(r\"NewsbrokenEmergency\", \"News broken emergency\", tweet)\n", " # tweet = re.sub(r\"Evaucation\", \"evacuation\", tweet)\n", " # tweet = re.sub(r\"leaveevacuateexitbe\", \"leave evacuate exit be\", tweet)\n", " # tweet = re.sub(r\"P_EOPLE\", \"PEOPLE\", tweet)\n", " # tweet = re.sub(r\"Tubestrike\", \"tube strike\", tweet)\n", " # tweet = re.sub(r\"CLASS_SICK\", \"CLASS SICK\", tweet)\n", " # tweet = re.sub(r\"localplumber\", \"local plumber\", tweet)\n", " # tweet = re.sub(r\"awesomejobsiri\", \"awesome job siri\", tweet)\n", " # tweet = re.sub(r\"PayForItHow\", \"Pay for it how\", tweet)\n", " # tweet = re.sub(r\"ThisIsAfrica\", \"This is Africa\", tweet)\n", " # tweet = re.sub(r\"crimeairnetwork\", \"crime air network\", tweet)\n", " # tweet = re.sub(r\"KimAcheson\", \"Kim Acheson\", tweet)\n", " # tweet = re.sub(r\"cityofcalgary\", \"City of Calgary\", tweet)\n", " # tweet = re.sub(r\"prosyndicate\", \"pro syndicate\", tweet)\n", " # tweet = re.sub(r\"660NEWS\", \"660 NEWS\", tweet)\n", " # tweet = re.sub(r\"BusInsMagazine\", \"Business Insurance Magazine\", tweet)\n", " # tweet = re.sub(r\"wfocus\", \"focus\", tweet)\n", " # tweet = re.sub(r\"ShastaDam\", \"Shasta Dam\", tweet)\n", " # tweet = re.sub(r\"go2MarkFranco\", \"Mark Franco\", tweet)\n", " # tweet = re.sub(r\"StephGHinojosa\", \"Steph Hinojosa\", tweet)\n", " # tweet = re.sub(r\"Nashgrier\", \"Nash Grier\", tweet)\n", " # tweet = re.sub(r\"NashNewVideo\", \"Nash new video\", tweet)\n", " # tweet = re.sub(r\"IWouldntGetElectedBecause\", \"I would not get elected because\", tweet)\n", " # tweet = re.sub(r\"SHGames\", \"Sledgehammer Games\", tweet)\n", " # tweet = re.sub(r\"bedhair\", \"bed hair\", tweet)\n", " # tweet = re.sub(r\"JoelHeyman\", \"Joel Heyman\", tweet)\n", " # tweet = re.sub(r\"viaYouTube\", \"via YouTube\", tweet)\n", " \n", " # Urls\n", " tweet = re.sub(r\"https?:\\/\\/t.co\\/[A-Za-z0-9]+\", \"\", tweet)\n", " \n", " # Words with punctuations and special characters\n", " punctuations = '@#!?+&*[]-%.:/();$=><|{}^' + \"'`\"\n", " for p in punctuations:\n", " tweet = tweet.replace(p, f' {p} ')\n", " \n", " # ... and ..\n", " tweet = tweet.replace('...', ' ... ')\n", " if '...' not in tweet:\n", " tweet = tweet.replace('..', ' ... ') \n", " \n", " # # Acronyms\n", " # tweet = re.sub(r\"MH370\", \"Malaysia Airlines Flight 370\", tweet)\n", " # tweet = re.sub(r\"m̼sica\", \"music\", tweet)\n", " # tweet = re.sub(r\"okwx\", \"Oklahoma City Weather\", tweet)\n", " # tweet = re.sub(r\"arwx\", \"Arkansas Weather\", tweet) \n", " # tweet = re.sub(r\"gawx\", \"Georgia Weather\", tweet) \n", " # tweet = re.sub(r\"scwx\", \"South Carolina Weather\", tweet) \n", " # tweet = re.sub(r\"cawx\", \"California Weather\", tweet)\n", " # tweet = re.sub(r\"tnwx\", \"Tennessee Weather\", tweet)\n", " # tweet = re.sub(r\"azwx\", \"Arizona Weather\", tweet) \n", " # tweet = re.sub(r\"alwx\", \"Alabama Weather\", tweet)\n", " # tweet = re.sub(r\"wordpressdotcom\", \"wordpress\", tweet) \n", " # tweet = re.sub(r\"usNWSgov\", \"United States National Weather Service\", tweet)\n", " # tweet = re.sub(r\"Suruc\", \"Sanliurfa\", tweet) \n", " \n", " # Grouping same words without embeddings\n", " tweet = re.sub(r\"Bestnaijamade\", \"bestnaijamade\", tweet)\n", " tweet = re.sub(r\"SOUDELOR\", \"Soudelor\", tweet)\n", "\n", " tweet = re.sub(r'https?://\\S+|www\\.\\S+',r'',tweet)\n", " tweet = re.sub(r'<.*?>',r'',tweet)\n", " tweet = re.sub(\"[\"\n", " u\"\\U0001F600-\\U0001F64F\" # emoticons\n", " u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n", " u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n", " u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n", " u\"\\U00002702-\\U000027B0\"\n", " u\"\\U000024C2-\\U0001F251\"\n", " \"]+\",r'', tweet, flags=re.UNICODE)\n", " table=str.maketrans('','',string.punctuation)\n", " return tweet.translate(table).lower()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Z-cXDAIPxL7r" }, "source": [ "df['clean_text'] = df['text'].apply(lambda x: clean(x))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "C__jK8aQxL71" }, "source": [ "df['Product_Label']=df['Product_Labels']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "EUPXiUYBxL7_" }, "source": [ "df['Complaint']=df['clean_text']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "IkuvB_dQxL8I" }, "source": [ "df['Product']=df['Primary']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Q8B20BwMxL8O", "colab": { "base_uri": "https://localhost:8080/", "height": 470 }, "outputId": "75abfeb6-e28c-434a-9fc4-48e707d3a30d" }, "source": [ "df" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labelsclean_textProduct_LabelComplaintProduct
01.005030e+05threeSnake oil Salesmen are not gone. They have jus...3snake oil salesmen are not gone they have ju...3snake oil salesmen are not gone they have ju...three
11.009760e+05threeFact check: Gargling water with salt or vinega...3fact check gargling water with salt or vineg...3fact check gargling water with salt or vineg...three
21.011710e+05zero.. this is great advice. Also, stop fighting f...0this is great advice also stop fighting...0this is great advice also stop fighting...zero
31.017920e+05three14 people in Iran died from alcohol poisoning,...314 people in iran died from alcohol poisoning ...314 people in iran died from alcohol poisoning ...three
41.037400e+05oneTo be fair, China is not the only country alle...1to be fair china is not the only country alleg...1to be fair china is not the only country alleg...one
...........................
19651.230000e+18one@TheSecondRevol1 @VincentCrypt46 I have been s...1thesecondrevol1 vincentcrypt46 i have been...1thesecondrevol1 vincentcrypt46 i have been...one
19661.230000e+18oneIf anyone is aware of the 5G theory related to...1if anyone is aware of the 5g theory related to...1if anyone is aware of the 5g theory related to...one
19671.230000e+18one@ClydeLewis @ClydeLewis have you looked into 5...1clydelewis clydelewis have you looked into...1clydelewis clydelewis have you looked into...one
19681.230000e+18oneHow many have the “Virus” where there’s...1how many have the “virus” where there’s...1how many have the “virus” where there’s...one
19691.230000e+18oneSo why am I still so thoroughly stunned by the...1so why am i still so thoroughly stunned by the...1so why am i still so thoroughly stunned by the...one
\n", "

1970 rows × 8 columns

\n", "
" ], "text/plain": [ " ID ... Product\n", "0 1.005030e+05 ... three\n", "1 1.009760e+05 ... three\n", "2 1.011710e+05 ... zero\n", "3 1.017920e+05 ... three\n", "4 1.037400e+05 ... one\n", "... ... ... ...\n", "1965 1.230000e+18 ... one\n", "1966 1.230000e+18 ... one\n", "1967 1.230000e+18 ... one\n", "1968 1.230000e+18 ... one\n", "1969 1.230000e+18 ... one\n", "\n", "[1970 rows x 8 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "code", "metadata": { "id": "MnNrjBl-xL8c", "colab": { "base_uri": "https://localhost:8080/", "height": 359 }, "outputId": "7f99c1db-791e-46b1-e7c7-abf82123ad32" }, "source": [ "df.head(10)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labelsclean_textProduct_LabelComplaintProduct
0100503.0threeSnake oil Salesmen are not gone. They have jus...3snake oil salesmen are not gone they have ju...3snake oil salesmen are not gone they have ju...three
1100976.0threeFact check: Gargling water with salt or vinega...3fact check gargling water with salt or vineg...3fact check gargling water with salt or vineg...three
2101171.0zero.. this is great advice. Also, stop fighting f...0this is great advice also stop fighting...0this is great advice also stop fighting...zero
3101792.0three14 people in Iran died from alcohol poisoning,...314 people in iran died from alcohol poisoning ...314 people in iran died from alcohol poisoning ...three
4103740.0oneTo be fair, China is not the only country alle...1to be fair china is not the only country alleg...1to be fair china is not the only country alleg...one
5108359.0threeCocaine does NOT cure #coronavirus: French gov...3cocaine does not cure coronavirus french g...3cocaine does not cure coronavirus french g...three
6108430.0three2008 Research paper demonstrating various esse...32008 research paper demonstrating various esse...32008 research paper demonstrating various esse...three
7110352.0three@SwerveChris @FakeNewsRehab @RudyGiuliani http...3swervechris fakenewsrehab rudygiuliani ...3swervechris fakenewsrehab rudygiuliani ...three
8110704.0three@wevarts @oliverdarcy @RudyGiuliani @charlieki...3wevarts oliverdarcy rudygiuliani charl...3wevarts oliverdarcy rudygiuliani charl...three
9110956.0zeroWasn't it great when our only health concerns ...0wasn t it great when our only health concerns...0wasn t it great when our only health concerns...zero
\n", "
" ], "text/plain": [ " ID Primary ... Complaint Product\n", "0 100503.0 three ... snake oil salesmen are not gone they have ju... three\n", "1 100976.0 three ... fact check gargling water with salt or vineg... three\n", "2 101171.0 zero ... this is great advice also stop fighting... zero\n", "3 101792.0 three ... 14 people in iran died from alcohol poisoning ... three\n", "4 103740.0 one ... to be fair china is not the only country alleg... one\n", "5 108359.0 three ... cocaine does not cure coronavirus french g... three\n", "6 108430.0 three ... 2008 research paper demonstrating various esse... three\n", "7 110352.0 three ... swervechris fakenewsrehab rudygiuliani ... three\n", "8 110704.0 three ... wevarts oliverdarcy rudygiuliani charl... three\n", "9 110956.0 zero ... wasn t it great when our only health concerns... zero\n", "\n", "[10 rows x 8 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 12 } ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "r5m3OPKwxL8i", "colab": { "base_uri": "https://localhost:8080/", "height": 173 }, "outputId": "ab78ed28-2684-4798-b8a3-37b4d1a1044e" }, "source": [ "label_counts = pd.DataFrame(df['Product'].value_counts())\n", "label_counts" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Product
zero768
two468
one462
three272
\n", "
" ], "text/plain": [ " Product\n", "zero 768\n", "two 468\n", "one 462\n", "three 272" ] }, "metadata": { "tags": [] }, "execution_count": 13 } ] }, { "cell_type": "markdown", "metadata": { "id": "3WNEdmm_xL8u" }, "source": [ "Here we create an array with the label names in the order they were numerically encoded. We use them later when plotting model performance data." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "eQcd-fGyxL8v", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "f8a9c946-f574-4f58-8802-5b3af435b4ac" }, "source": [ "label_values = list(label_counts.index)\n", "order = list(pd.DataFrame(df['Product_Label'].value_counts()).index)\n", "label_values = [l for _,l in sorted(zip(order, label_values))]\n", "\n", "label_values" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['zero', 'one', 'two', 'three']" ] }, "metadata": { "tags": [] }, "execution_count": 14 } ] }, { "cell_type": "markdown", "metadata": { "id": "WEPaiaCqxL81" }, "source": [ "We need to create 2 arrays: one with the textual data, which is our feature data, and one with the numerically encoded labels, representing our target data." ] }, { "cell_type": "code", "metadata": { "id": "GIikmH20xL82" }, "source": [ "texts = df['Complaint'].values\n", "labels = df['Product_Label'].values" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "VBmrAPeCxL87" }, "source": [ "BERT is a ‘heavy-weight’´model. This makes the training a very resource-intensive process, specially when we are fine-tuning for all model layers. To mitigate this, we can control the sequence length of our input text, which is given by the number of tokens in our input text, plus 2 special tokens to mark the beginning and ending of a text sequence." ] }, { "cell_type": "code", "metadata": { "id": "x0l_LFz4xL89", "colab": { "base_uri": "https://localhost:8080/", "height": 51 }, "outputId": "6da044ea-3db2-456d-f4a3-d18593699632" }, "source": [ "text_lengths = [len(texts[i].split()) for i in range(len(texts))]\n", "print(min(text_lengths))\n", "print(max(text_lengths))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "1\n", "90\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "BXJrwvTExL9C", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "464e07ce-f183-4c04-d86b-76c806e40f74" }, "source": [ "sum([1 for i in range(len(text_lengths)) if text_lengths[i] >= 55])" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "7" ] }, "metadata": { "tags": [] }, "execution_count": 17 } ] }, { "cell_type": "markdown", "metadata": { "id": "UZsB1xuXxL9J" }, "source": [ "Here we instantiate a BERT tokenizer and show an example of a tokenized text. " ] }, { "cell_type": "code", "metadata": { "scrolled": false, "id": "Fb1ndOhhxL9K", "colab": { "base_uri": "https://localhost:8080/", "height": 122 }, "outputId": "4310e6f4-6f42-4a5b-c780-f31096d78adc" }, "source": [ "from transformers import DistilBertTokenizer\n", "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased', do_lower_case=True)\n", "\n", "\n", "\n", "print('Original Text: ', texts[0], '\\n')\n", "print('Tokenized Text: ', tokenizer.tokenize(texts[0]), '\\n')\n", "print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(texts[0])))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Original Text: snake oil salesmen are not gone they have just replaced the horse cart with twitter accounts and websites got an fda warning letter for advertising and selling the most powerful essential oil that can treat and defend against covid19 stay safe \n", "\n", "Tokenized Text: ['snake', 'oil', 'sales', '##men', 'are', 'not', 'gone', 'they', 'have', 'just', 'replaced', 'the', 'horse', 'cart', 'with', 'twitter', 'accounts', 'and', 'websites', 'got', 'an', 'fda', 'warning', 'letter', 'for', 'advertising', 'and', 'selling', 'the', 'most', 'powerful', 'essential', 'oil', 'that', 'can', 'treat', 'and', 'defend', 'against', 'co', '##vid', '##19', 'stay', 'safe'] \n", "\n", "Token IDs: [7488, 3514, 4341, 3549, 2024, 2025, 2908, 2027, 2031, 2074, 2999, 1996, 3586, 11122, 2007, 10474, 6115, 1998, 11744, 2288, 2019, 17473, 5432, 3661, 2005, 6475, 1998, 4855, 1996, 2087, 3928, 6827, 3514, 2008, 2064, 7438, 1998, 6985, 2114, 2522, 17258, 16147, 2994, 3647]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "zLRP4WBsxL9Q" }, "source": [ "We then tokenize and encode the entire dataset. In this process, we perform the following:\n", "- tokenize the text as shown above\n", "- encode it to the corresponding numeric values for each token.\n", "- truncate it to the maximum sequence length of 55.\n", "- pad the tokens positions greater than 55.\n", "- include the special token IDs to mark the beginning and end of each sequence." ] }, { "cell_type": "code", "metadata": { "id": "zbbF0BuOxL9R", "colab": { "base_uri": "https://localhost:8080/", "height": 952 }, "outputId": "bb4691f9-3fbb-46b2-c820-9a38b0127ca5" }, "source": [ "text_ids = [tokenizer.encode(text, max_length=55, pad_to_max_length=True) for text in texts]\n", "\n", "text_ids[0]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[101,\n", " 7488,\n", " 3514,\n", " 4341,\n", " 3549,\n", " 2024,\n", " 2025,\n", " 2908,\n", " 2027,\n", " 2031,\n", " 2074,\n", " 2999,\n", " 1996,\n", " 3586,\n", " 11122,\n", " 2007,\n", " 10474,\n", " 6115,\n", " 1998,\n", " 11744,\n", " 2288,\n", " 2019,\n", " 17473,\n", " 5432,\n", " 3661,\n", " 2005,\n", " 6475,\n", " 1998,\n", " 4855,\n", " 1996,\n", " 2087,\n", " 3928,\n", " 6827,\n", " 3514,\n", " 2008,\n", " 2064,\n", " 7438,\n", " 1998,\n", " 6985,\n", " 2114,\n", " 2522,\n", " 17258,\n", " 16147,\n", " 2994,\n", " 3647,\n", " 102,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0]" ] }, "metadata": { "tags": [] }, "execution_count": 18 } ] }, { "cell_type": "code", "metadata": { "id": "Ts3PCOq4xL9Z", "colab": { "base_uri": "https://localhost:8080/", "height": 51 }, "outputId": "778aab46-6a86-448d-a819-aa4d905814c8" }, "source": [ "text_ids_lengths = [len(text_ids[i]) for i in range(len(text_ids))]\n", "print(min(text_ids_lengths))\n", "print(max(text_ids_lengths))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "55\n", "55\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "WvCdlWKSxL9f" }, "source": [ "To fine-tune our model, we need two inputs: one array of token IDs (created above) and one array of a corresponding binary mask, called attention mask in the BERT model specification. Each attention mask has the same length of the corresponding input sequence and has a 0 if the corresponding token is a pad token, or a 1 otherwise. " ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "R7-cT8xgxL9g", "colab": { "base_uri": "https://localhost:8080/", "height": 952 }, "outputId": "77e70d54-f8f2-4ffb-f424-7f4f0e128ecc" }, "source": [ "att_masks = []\n", "for ids in text_ids:\n", " masks = [int(id > 0) for id in ids]\n", " att_masks.append(masks)\n", " \n", "att_masks[0]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0]" ] }, "metadata": { "tags": [] }, "execution_count": 20 } ] }, { "cell_type": "markdown", "metadata": { "id": "qYbr4F0UxL9m" }, "source": [ "Here we split the input and output arrays created before into train, validation, and test sets. We use 80% of the data for training, 10% for training validation, and 10% for final testing." ] }, { "cell_type": "code", "metadata": { "id": "iisem5RsxL9n" }, "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "train_x, test_val_x, train_y, test_val_y = train_test_split(text_ids, labels, random_state=11, test_size=0.2, stratify=labels)\n", "train_m, test_val_m = train_test_split(att_masks, random_state=11, test_size=0.2)\n", "\n", "test_x, val_x, test_y, val_y = train_test_split(test_val_x, test_val_y, random_state=11, test_size=0.5, stratify=test_val_y)\n", "test_m, val_m = train_test_split(test_val_m, random_state=11, test_size=0.5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "XOgx3lnjxL9t" }, "source": [ "We are working with the PyTorch artifacts in the transformers library, therefore we need our model input and output data as PyTorch tensors." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "juMKH1GbxL9u", "colab": { "base_uri": "https://localhost:8080/", "height": 170 }, "outputId": "07aa115d-267c-4f78-dba2-58e37a8d8d31" }, "source": [ "import torch\n", "\n", "train_x = torch.tensor(train_x)\n", "test_x = torch.tensor(test_x)\n", "val_x = torch.tensor(val_x)\n", "train_y = torch.tensor(train_y)\n", "test_y = torch.tensor(test_y)\n", "val_y = torch.tensor(val_y)\n", "train_m = torch.tensor(train_m)\n", "test_m = torch.tensor(test_m)\n", "val_m = torch.tensor(val_m)\n", "\n", "print(train_x.shape)\n", "print(test_x.shape)\n", "print(val_x.shape)\n", "print(train_y.shape)\n", "print(test_y.shape)\n", "print(val_y.shape)\n", "print(train_m.shape)\n", "print(test_m.shape)\n", "print(val_m.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "torch.Size([1576, 55])\n", "torch.Size([197, 55])\n", "torch.Size([197, 55])\n", "torch.Size([1576])\n", "torch.Size([197])\n", "torch.Size([197])\n", "torch.Size([1576, 55])\n", "torch.Size([197, 55])\n", "torch.Size([197, 55])\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "1nKiIAUwxL92" }, "source": [ "To feed data into the model for training, we use Pytorch’s Dataset, DataLoader, and Sampler. For feeding training data, which drives model weights updates, we use the RandomSampler. For feeding the validation data we can use the SequentialSampler." ] }, { "cell_type": "code", "metadata": { "id": "4DNG5OZMxL93", "colab": { "base_uri": "https://localhost:8080/", "height": 357 }, "outputId": "bda2e676-451a-4066-f1a1-2530d40f0fda" }, "source": [ " # Checking if GPU is available or not\n", " !nvidia-smi" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Mon Oct 19 17:12:28 2020 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 455.23.05 Driver Version: 418.67 CUDA Version: 10.1 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n", "| N/A 61C P8 11W / 70W | 0MiB / 15079MiB | 0% Default |\n", "| | | ERR! |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "| No running processes found |\n", "+-----------------------------------------------------------------------------+\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "oBt7jkeixL97" }, "source": [ "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", "\n", "batch_size = 8\n", "\n", "train_data = TensorDataset(train_x, train_m, train_y)\n", "train_sampler = RandomSampler(train_data)\n", "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", "\n", "val_data = TensorDataset(val_x, val_m, val_y)\n", "val_sampler = SequentialSampler(val_data)\n", "val_dataloader = DataLoader(val_data, sampler=val_sampler, batch_size=batch_size)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "wqAFseKYxL-A" }, "source": [ "Here we instantiate our model class. We use a compact version, that is trained through model distillation from a base BERT model and modified to include a classification layer at the output. This compact version has 6 transformer layers instead of 12 as in the original BERT model. Please see [here]( https://github.com/huggingface/transformers/tree/master/examples/distillation) for more details." ] }, { "cell_type": "code", "metadata": { "id": "rXb2yLsQxL-B" }, "source": [ "from transformers import DistilBertForSequenceClassification, DistilBertConfig, AdamW\n", "\n", "num_labels = len(set(labels))\n", "\n", "model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=num_labels,\n", " output_attentions=False, output_hidden_states=False)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "x6M2UXGYxL-E", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "619e5727-f53e-44c1-9d3d-d8ba3a19708d" }, "source": [ "num_labels" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "4" ] }, "metadata": { "tags": [] }, "execution_count": 26 } ] }, { "cell_type": "markdown", "metadata": { "id": "SJN9aDmSxL-H" }, "source": [ "BERT is a very large model. Unless you are freezing model weights in all layers but the classification layer, it is recommended to train it on a GPU." ] }, { "cell_type": "code", "metadata": { "id": "wVU4dz37xL-H", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "d5e41c7d-d5fa-46b4-ea87-e922c217a1aa" }, "source": [ "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "print(device)\n", "model = model.to(device)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "cuda\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "ZRxdfioSxL-K" }, "source": [ "Here we print the model architecture and all model learnable parameters." ] }, { "cell_type": "code", "metadata": { "id": "d8W8unp5xL-K", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "ac57d081-56d5-4dbe-8f47-337c0b0268a0" }, "source": [ "def count_parameters(model):\n", " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "\n", "print('Number of trainable parameters:', count_parameters(model), '\\n', model)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Number of trainable parameters: 66956548 \n", " DistilBertForSequenceClassification(\n", " (distilbert): DistilBertModel(\n", " (embeddings): Embeddings(\n", " (word_embeddings): Embedding(30522, 768, padding_idx=0)\n", " (position_embeddings): Embedding(512, 768)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (transformer): Transformer(\n", " (layer): ModuleList(\n", " (0): TransformerBlock(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (attention): MultiHeadSelfAttention(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (q_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (k_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (v_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (out_lin): Linear(in_features=768, out_features=768, bias=True)\n", " )\n", " (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (ffn): FFN(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (lin1): Linear(in_features=768, out_features=3072, bias=True)\n", " (lin2): Linear(in_features=3072, out_features=768, bias=True)\n", " )\n", " (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " )\n", " (1): TransformerBlock(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (attention): MultiHeadSelfAttention(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (q_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (k_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (v_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (out_lin): Linear(in_features=768, out_features=768, bias=True)\n", " )\n", " (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (ffn): FFN(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (lin1): Linear(in_features=768, out_features=3072, bias=True)\n", " (lin2): Linear(in_features=3072, out_features=768, bias=True)\n", " )\n", " (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " )\n", " (2): TransformerBlock(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (attention): MultiHeadSelfAttention(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (q_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (k_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (v_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (out_lin): Linear(in_features=768, out_features=768, bias=True)\n", " )\n", " (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (ffn): FFN(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (lin1): Linear(in_features=768, out_features=3072, bias=True)\n", " (lin2): Linear(in_features=3072, out_features=768, bias=True)\n", " )\n", " (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " )\n", " (3): TransformerBlock(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (attention): MultiHeadSelfAttention(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (q_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (k_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (v_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (out_lin): Linear(in_features=768, out_features=768, bias=True)\n", " )\n", " (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (ffn): FFN(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (lin1): Linear(in_features=768, out_features=3072, bias=True)\n", " (lin2): Linear(in_features=3072, out_features=768, bias=True)\n", " )\n", " (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " )\n", " (4): TransformerBlock(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (attention): MultiHeadSelfAttention(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (q_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (k_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (v_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (out_lin): Linear(in_features=768, out_features=768, bias=True)\n", " )\n", " (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (ffn): FFN(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (lin1): Linear(in_features=768, out_features=3072, bias=True)\n", " (lin2): Linear(in_features=3072, out_features=768, bias=True)\n", " )\n", " (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " )\n", " (5): TransformerBlock(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (attention): MultiHeadSelfAttention(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (q_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (k_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (v_lin): Linear(in_features=768, out_features=768, bias=True)\n", " (out_lin): Linear(in_features=768, out_features=768, bias=True)\n", " )\n", " (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (ffn): FFN(\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (lin1): Linear(in_features=768, out_features=3072, bias=True)\n", " (lin2): Linear(in_features=3072, out_features=768, bias=True)\n", " )\n", " (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " )\n", " )\n", " )\n", " )\n", " (pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n", " (classifier): Linear(in_features=768, out_features=4, bias=True)\n", " (dropout): Dropout(p=0.2, inplace=False)\n", ")\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "W00zF-SdxL-N", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "9dbe11cc-fc55-4969-9e60-8c7eb4f52939" }, "source": [ "[n for n, p in model.named_parameters()]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['distilbert.embeddings.word_embeddings.weight',\n", " 'distilbert.embeddings.position_embeddings.weight',\n", " 'distilbert.embeddings.LayerNorm.weight',\n", " 'distilbert.embeddings.LayerNorm.bias',\n", " 'distilbert.transformer.layer.0.attention.q_lin.weight',\n", " 'distilbert.transformer.layer.0.attention.q_lin.bias',\n", " 'distilbert.transformer.layer.0.attention.k_lin.weight',\n", " 'distilbert.transformer.layer.0.attention.k_lin.bias',\n", " 'distilbert.transformer.layer.0.attention.v_lin.weight',\n", " 'distilbert.transformer.layer.0.attention.v_lin.bias',\n", " 'distilbert.transformer.layer.0.attention.out_lin.weight',\n", " 'distilbert.transformer.layer.0.attention.out_lin.bias',\n", " 'distilbert.transformer.layer.0.sa_layer_norm.weight',\n", " 'distilbert.transformer.layer.0.sa_layer_norm.bias',\n", " 'distilbert.transformer.layer.0.ffn.lin1.weight',\n", " 'distilbert.transformer.layer.0.ffn.lin1.bias',\n", " 'distilbert.transformer.layer.0.ffn.lin2.weight',\n", " 'distilbert.transformer.layer.0.ffn.lin2.bias',\n", " 'distilbert.transformer.layer.0.output_layer_norm.weight',\n", " 'distilbert.transformer.layer.0.output_layer_norm.bias',\n", " 'distilbert.transformer.layer.1.attention.q_lin.weight',\n", " 'distilbert.transformer.layer.1.attention.q_lin.bias',\n", " 'distilbert.transformer.layer.1.attention.k_lin.weight',\n", " 'distilbert.transformer.layer.1.attention.k_lin.bias',\n", " 'distilbert.transformer.layer.1.attention.v_lin.weight',\n", " 'distilbert.transformer.layer.1.attention.v_lin.bias',\n", " 'distilbert.transformer.layer.1.attention.out_lin.weight',\n", " 'distilbert.transformer.layer.1.attention.out_lin.bias',\n", " 'distilbert.transformer.layer.1.sa_layer_norm.weight',\n", " 'distilbert.transformer.layer.1.sa_layer_norm.bias',\n", " 'distilbert.transformer.layer.1.ffn.lin1.weight',\n", " 'distilbert.transformer.layer.1.ffn.lin1.bias',\n", " 'distilbert.transformer.layer.1.ffn.lin2.weight',\n", " 'distilbert.transformer.layer.1.ffn.lin2.bias',\n", " 'distilbert.transformer.layer.1.output_layer_norm.weight',\n", " 'distilbert.transformer.layer.1.output_layer_norm.bias',\n", " 'distilbert.transformer.layer.2.attention.q_lin.weight',\n", " 'distilbert.transformer.layer.2.attention.q_lin.bias',\n", " 'distilbert.transformer.layer.2.attention.k_lin.weight',\n", " 'distilbert.transformer.layer.2.attention.k_lin.bias',\n", " 'distilbert.transformer.layer.2.attention.v_lin.weight',\n", " 'distilbert.transformer.layer.2.attention.v_lin.bias',\n", " 'distilbert.transformer.layer.2.attention.out_lin.weight',\n", " 'distilbert.transformer.layer.2.attention.out_lin.bias',\n", " 'distilbert.transformer.layer.2.sa_layer_norm.weight',\n", " 'distilbert.transformer.layer.2.sa_layer_norm.bias',\n", " 'distilbert.transformer.layer.2.ffn.lin1.weight',\n", " 'distilbert.transformer.layer.2.ffn.lin1.bias',\n", " 'distilbert.transformer.layer.2.ffn.lin2.weight',\n", " 'distilbert.transformer.layer.2.ffn.lin2.bias',\n", " 'distilbert.transformer.layer.2.output_layer_norm.weight',\n", " 'distilbert.transformer.layer.2.output_layer_norm.bias',\n", " 'distilbert.transformer.layer.3.attention.q_lin.weight',\n", " 'distilbert.transformer.layer.3.attention.q_lin.bias',\n", " 'distilbert.transformer.layer.3.attention.k_lin.weight',\n", " 'distilbert.transformer.layer.3.attention.k_lin.bias',\n", " 'distilbert.transformer.layer.3.attention.v_lin.weight',\n", " 'distilbert.transformer.layer.3.attention.v_lin.bias',\n", " 'distilbert.transformer.layer.3.attention.out_lin.weight',\n", " 'distilbert.transformer.layer.3.attention.out_lin.bias',\n", " 'distilbert.transformer.layer.3.sa_layer_norm.weight',\n", " 'distilbert.transformer.layer.3.sa_layer_norm.bias',\n", " 'distilbert.transformer.layer.3.ffn.lin1.weight',\n", " 'distilbert.transformer.layer.3.ffn.lin1.bias',\n", " 'distilbert.transformer.layer.3.ffn.lin2.weight',\n", " 'distilbert.transformer.layer.3.ffn.lin2.bias',\n", " 'distilbert.transformer.layer.3.output_layer_norm.weight',\n", " 'distilbert.transformer.layer.3.output_layer_norm.bias',\n", " 'distilbert.transformer.layer.4.attention.q_lin.weight',\n", " 'distilbert.transformer.layer.4.attention.q_lin.bias',\n", " 'distilbert.transformer.layer.4.attention.k_lin.weight',\n", " 'distilbert.transformer.layer.4.attention.k_lin.bias',\n", " 'distilbert.transformer.layer.4.attention.v_lin.weight',\n", " 'distilbert.transformer.layer.4.attention.v_lin.bias',\n", " 'distilbert.transformer.layer.4.attention.out_lin.weight',\n", " 'distilbert.transformer.layer.4.attention.out_lin.bias',\n", " 'distilbert.transformer.layer.4.sa_layer_norm.weight',\n", " 'distilbert.transformer.layer.4.sa_layer_norm.bias',\n", " 'distilbert.transformer.layer.4.ffn.lin1.weight',\n", " 'distilbert.transformer.layer.4.ffn.lin1.bias',\n", " 'distilbert.transformer.layer.4.ffn.lin2.weight',\n", " 'distilbert.transformer.layer.4.ffn.lin2.bias',\n", " 'distilbert.transformer.layer.4.output_layer_norm.weight',\n", " 'distilbert.transformer.layer.4.output_layer_norm.bias',\n", " 'distilbert.transformer.layer.5.attention.q_lin.weight',\n", " 'distilbert.transformer.layer.5.attention.q_lin.bias',\n", " 'distilbert.transformer.layer.5.attention.k_lin.weight',\n", " 'distilbert.transformer.layer.5.attention.k_lin.bias',\n", " 'distilbert.transformer.layer.5.attention.v_lin.weight',\n", " 'distilbert.transformer.layer.5.attention.v_lin.bias',\n", " 'distilbert.transformer.layer.5.attention.out_lin.weight',\n", " 'distilbert.transformer.layer.5.attention.out_lin.bias',\n", " 'distilbert.transformer.layer.5.sa_layer_norm.weight',\n", " 'distilbert.transformer.layer.5.sa_layer_norm.bias',\n", " 'distilbert.transformer.layer.5.ffn.lin1.weight',\n", " 'distilbert.transformer.layer.5.ffn.lin1.bias',\n", " 'distilbert.transformer.layer.5.ffn.lin2.weight',\n", " 'distilbert.transformer.layer.5.ffn.lin2.bias',\n", " 'distilbert.transformer.layer.5.output_layer_norm.weight',\n", " 'distilbert.transformer.layer.5.output_layer_norm.bias',\n", " 'pre_classifier.weight',\n", " 'pre_classifier.bias',\n", " 'classifier.weight',\n", " 'classifier.bias']" ] }, "metadata": { "tags": [] }, "execution_count": 29 } ] }, { "cell_type": "markdown", "metadata": { "id": "JKXrzR16xL-Q" }, "source": [ "In the following 5 cells we define our PyTorch optimizer and corresponding parameters, learning rate scheduler, and the training loop for the fine-tuning procedure. We train for 10 epochs." ] }, { "cell_type": "code", "metadata": { "id": "HJiKuvNaxL-Q" }, "source": [ "learning_rate = 1e-5\n", "adam_epsilon = 1e-8\n", "\n", "no_decay = ['bias', 'LayerNorm.weight']\n", "optimizer_grouped_parameters = [\n", " {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.2},\n", " {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.0}\n", "]\n", "\n", "optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=adam_epsilon)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "YwUeY2FZxL-T" }, "source": [ "from transformers import get_linear_schedule_with_warmup\n", "\n", "num_epochs = 10\n", "total_steps = len(train_dataloader) * num_epochs\n", "\n", "scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=total_steps)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "4KZRZTkpxL-W" }, "source": [ "import time\n", "\n", "def epoch_time(start_time, end_time):\n", " elapsed_time = end_time - start_time\n", " elapsed_mins = int(elapsed_time / 60)\n", " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n", " return elapsed_mins, elapsed_secs" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "S3u370YPxL-Z" }, "source": [ "import numpy as np\n", "import random\n", "\n", "seed_val = 11\n", "\n", "random.seed(seed_val)\n", "np.random.seed(seed_val)\n", "torch.manual_seed(seed_val)\n", "torch.cuda.manual_seed_all(seed_val)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "UKKSJ6FbxL-a", "colab": { "base_uri": "https://localhost:8080/", "height": 782 }, "outputId": "d2c98140-3476-40f6-d8e2-d5abf5f399be" }, "source": [ "train_losses = []\n", "val_losses = []\n", "num_mb_train = len(train_dataloader)\n", "num_mb_val = len(val_dataloader)\n", "\n", "if num_mb_val == 0:\n", " num_mb_val = 1\n", "\n", "for n in range(num_epochs):\n", " train_loss = 0\n", " val_loss = 0\n", " start_time = time.time()\n", " \n", " for k, (mb_x, mb_m, mb_y) in enumerate(train_dataloader):\n", " optimizer.zero_grad()\n", " model.train()\n", " \n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " mb_y = mb_y.to(device)\n", " \n", " outputs = model(mb_x, attention_mask=mb_m, labels=mb_y)\n", " \n", " loss = outputs[0]\n", " #loss = model_loss(outputs[1], mb_y)\n", " loss.backward()\n", " torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n", " optimizer.step()\n", " scheduler.step()\n", " \n", " train_loss += loss.data / num_mb_train\n", " \n", " print (\"\\nTrain loss after itaration %i: %f\" % (n+1, train_loss))\n", " train_losses.append(train_loss.cpu())\n", " \n", " with torch.no_grad():\n", " model.eval()\n", " \n", " for k, (mb_x, mb_m, mb_y) in enumerate(val_dataloader):\n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " mb_y = mb_y.to(device)\n", " \n", " outputs = model(mb_x, attention_mask=mb_m, labels=mb_y)\n", " \n", " loss = outputs[0]\n", " #loss = model_loss(outputs[1], mb_y)\n", " \n", " val_loss += loss.data / num_mb_val\n", " \n", " print (\"Validation loss after itaration %i: %f\" % (n+1, val_loss))\n", " val_losses.append(val_loss.cpu())\n", " \n", " end_time = time.time()\n", " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", " print(f'Time: {epoch_mins}m {epoch_secs}s')" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.6/dist-packages/transformers/optimization.py:146: UserWarning: This overload of add_ is deprecated:\n", "\tadd_(Number alpha, Tensor other)\n", "Consider using one of the following signatures instead:\n", "\tadd_(Tensor other, *, Number alpha) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)\n", " exp_avg.mul_(beta1).add_(1.0 - beta1, grad)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\n", "Train loss after itaration 1: 1.216444\n", "Validation loss after itaration 1: 0.998652\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 2: 0.776469\n", "Validation loss after itaration 2: 0.836683\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 3: 0.609383\n", "Validation loss after itaration 3: 0.803564\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 4: 0.496733\n", "Validation loss after itaration 4: 0.834540\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 5: 0.400773\n", "Validation loss after itaration 5: 0.864585\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 6: 0.328608\n", "Validation loss after itaration 6: 0.890638\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 7: 0.267227\n", "Validation loss after itaration 7: 0.948643\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 8: 0.211375\n", "Validation loss after itaration 8: 1.047477\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 9: 0.182005\n", "Validation loss after itaration 9: 1.024981\n", "Time: 0m 13s\n", "\n", "Train loss after itaration 10: 0.161004\n", "Validation loss after itaration 10: 1.039541\n", "Time: 0m 13s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "gTPgR5hlxL-c" }, "source": [ "After training, we can save the model and necessary configuration parameters, to recreate it later and use it to score the test data. Here we also save the losses computed from both training and validation data." ] }, { "cell_type": "code", "metadata": { "id": "uosvFYBlxL-d" }, "source": [ "import pickle\n", "import os\n", "\n", "out_dir = './model'\n", "\n", "if not os.path.exists(out_dir):\n", " os.makedirs(out_dir)\n", " \n", "model_to_save = model.module if hasattr(model, 'module') else model\n", "model_to_save.save_pretrained(out_dir)\n", "tokenizer.save_pretrained(out_dir)\n", "\n", "with open(out_dir + '/train_losses.pkl', 'wb') as f:\n", " pickle.dump(train_losses, f)\n", " \n", "with open(out_dir + '/val_losses.pkl', 'wb') as f:\n", " pickle.dump(val_losses, f)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "yiRvHcASxL-f" }, "source": [ "out_dir = './model'\n", "\n", "model = DistilBertForSequenceClassification.from_pretrained(out_dir)\n", "\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "model = model.to(device)\n", "\n", "with open(out_dir + '/train_losses.pkl', 'rb') as f:\n", " train_losses = pickle.load(f)\n", " \n", "with open(out_dir + '/val_losses.pkl', 'rb') as f:\n", " val_losses = pickle.load(f)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "d_jycFxMxL-g", "colab": { "base_uri": "https://localhost:8080/", "height": 282 }, "outputId": "d4bddad0-030d-4902-8b41-e389e7dd896d" }, "source": [ "from matplotlib import pyplot as plt\n", "%matplotlib inline\n", "\n", "plt.figure()\n", "plt.plot(train_losses)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[]" ] }, "metadata": { "tags": [] }, "execution_count": 37 }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAfzklEQVR4nO3deXSV1b3/8fc380gGMhAyAQIiCogECFjBARVHtE6AeGuvqFjb29be21/7u/3dtna5Onhra2+drbWKQxV7reIAOKEyBweEIBDGBEIShgRCCJn274+T0oABAjnJc87J57UWS845D+d8PAs+bPZ+nv2Ycw4REQl+YV4HEBER/1Chi4iECBW6iEiIUKGLiIQIFbqISIiI8OqD09LSXL9+/bz6eBGRoLRy5cpdzrn09l7zrND79etHUVGRVx8vIhKUzGzrsV7TlIuISIhQoYuIhAgVuohIiFChi4iECBW6iEiIUKGLiIQIFbqISIgIukLfWFXLva8X09DU4nUUEZGAcsJCN7OnzKzSzFYf4/WbzWyVmX1hZovNbIT/Y/7Ttt11PLVoM/OLd3blx4iIBJ2OjNCfBiYf5/XNwETn3DDgF8Djfsh1TBMGp5OTEsvspce8WEpEpEc6YaE75z4E9hzn9cXOub2tD5cCOX7K1q7wMGP62DyWbtpDSeX+rvwoEZGg4u859NuAt471opndYWZFZlZUVVV1yh9yY0EukeHG7KXbTvk9RERCjd8K3cwuwFfo/+dYxzjnHnfOFTjnCtLT290srEPSEqK57KwsXvmkjLqGplN+HxGRUOKXQjez4cCTwBTn3G5/vOeJzCjMZ399E69/vqM7Pk5EJOB1utDNLA/4G3CLc2595yN1zOh+KQzOTNC0i4hIq46ctvgCsAQ43czKzOw2M5tlZrNaD/kvoDfwsJl9Zmbdssm5mTGjMJ8vttfweWl1d3ykiEhAO+ENLpxz007w+kxgpt8SnYRrR2bzq7e+ZPbSrYzITfYigohIwAi6K0XbSoyJZMrZ2by+agc1dY1exxER8VRQFzrAjMI86htbmPNJmddRREQ8FfSFfmbfJEbmJfPcsq0457yOIyLimaAvdIBbCvPZVHWAJRu75YxJEZGAFBKFfvmwLFLiIpm9TPu7iEjPFRKFHhMZzg0FucxbU0HFvnqv44iIeCIkCh1g+pg8mlscLy4v9TqKiIgnQqbQ+6XFc96gNF5Yvo2mZt38QkR6npApdPDt77JzXz3vflnpdRQRkW4XUoV+0ZAMspJidPMLEemRQqrQI8LDmDo6j4827GLLrgNexxER6VYhVegAU8fkEh5mPL9cuzCKSM8ScoWe2SuGS4Zm8nJRKfWNzV7HERHpNiFX6OBbHN1b18ibX5R7HUVEpNuEZKGPP603A9LitTgqIj1KSBa6mTF9bB6fbKumeMc+r+OIiHSLkCx0gOtH5RAdEab9XUSkxwjZQk+Oi+KqEX159dPt7K/XzS9EJPSFbKGDb3G0rqGZVz/d7nUUEZEuF9KFPiIniWHZScxeuk03vxCRkBfShW5mzCjMY13Ffoq27vU6johIlwrpQge4akRfEmMieHaJFkdFJLSFfKHHRUVw3Tk5vLW6nF21h7yOIyLSZUK+0AFmFObR2Ox4qUg3vxCR0NUjCn1gRiKFA1J5ftk2mlu0OCoioalHFDr4TmEs23uQD9dXeR1FRKRL9JhCv2RoH9ISorW/i4iErB5T6FERYUwdnct76yop21vndRwREb/rMYUOMG1sHga8oJtfiEgIOmGhm9lTZlZpZquP8bqZ2R/MrMTMVpnZOf6P6R/ZybFcOCSDv64opaGpxes4IiJ+1ZER+tPA5OO8fhkwqPXHHcAjnY/VdW4uzGdXbQPz1uz0OoqIiF+dsNCdcx8Ce45zyBTgGeezFEg2syx/BfS3iYPSyU2N1eKoiIQcf8yhZwNtr9gpa33uK8zsDjMrMrOiqipvTh8MCzOmj8ln2eY9bKjY70kGEZGu0K2Los65x51zBc65gvT09O786CPcWJBDVHgYzy3T4qiIhA5/FPp2ILfN45zW5wJW74RoLhvWh1dWllHX0OR1HBERv/BHob8G/Evr2S6FQI1zrtwP79ulbinMZ/+hJl77bIfXUURE/KIjpy2+ACwBTjezMjO7zcxmmdms1kPeBDYBJcATwLe6LK0fjcpPYUifRJ5dulU3vxCRkBBxogOcc9NO8LoD7vZbom5iZtxcmM//e3U1n5VWMzIvxetIIiKd0qOuFD3atSOziY8KZ/ZSLY6KSPDr0YWeEB3BNSOzmbtqB9V1DV7HERHplB5d6ODbVvdQUwtzVpZ5HUVEpFN6fKGfkdWLUfkpPLdsGy26+YWIBLEeX+jgu0Xd5l0HWLxxt9dRREROmQoduOysLFLiIrW/i4gENRU6EBMZzo0FuSxYW8HOmnqv44iInBIVeqvpY/NobnG8uEKnMIpIcFKht8rvHc+Ewem8uLyUpmbd/EJEgo8KvY0ZY/PYua+ed9ZWeh1FROSkqdDbuHBIBllJMTy3TIujIhJ8VOhtRISHMW1MHh9t2MXmXQe8jiMiclJU6EeZOjqXiDDjeY3SRSTIqNCPktErhkvP7MPLK8uob2z2Oo6ISIep0Ntxc2Ee1XWNvLEq4O/TISJymAq9HeMG9Oa09Hie1ZWjIhJEVOjtMDNuHpvPZ6XVrN5e43UcEZEOUaEfw3WjcoiJDNMpjCISNFTox5AUG8nVI/ry6qc72Fff6HUcEZETUqEfx4zCfA42NvO/n2z3OoqIyAmp0I9jeE4yw3OSmL10K757YYuIBC4V+gnMGJvPhspalm/e43UUEZHjUqGfwFUj+tIrJoLZy7StrogENhX6CcRGhXPdqBzeXl1O1f5DXscRETkmFXoH3Dw2n8Zmx0tFpV5HERE5JhV6BwzMSGDcgN48v2wbzS1aHBWRwKRC76AZhflsrz7IwvW6+YWIBCYVegddcmYm6YnRzF6qxVERCUwdKnQzm2xm68ysxMx+1M7reWb2vpl9amarzOxy/0f1VmR4GFNH5/L+ukpK99R5HUdE5CtOWOhmFg48BFwGDAWmmdnQow77CfCSc24kMBV42N9BA8G0MXkY8MJyjdJFJPB0ZIQ+Bihxzm1yzjUALwJTjjrGAb1af54E7PBfxMDRNzmWi87I5KWiUg416eYXIhJYOlLo2UDb8/XKWp9r62fADDMrA94EvtPeG5nZHWZWZGZFVVVVpxDXezMK89lV28Dbq3d6HUVE5Aj+WhSdBjztnMsBLgeeNbOvvLdz7nHnXIFzriA9Pd1PH929zhuYRn7vOJ7T4qiIBJiOFPp2ILfN45zW59q6DXgJwDm3BIgB0vwRMNCEhRnTx+SxfMse1u3c73UcEZHDOlLoK4BBZtbfzKLwLXq+dtQx24CLAMzsDHyFHpxzKh1wQ0EuURG6+YWIBJYTFrpzrgn4NjAPWIvvbJY1ZnavmV3detgPgNvN7HPgBeBWF8L7zabGR3HFsCxeLipj6abdXscREQHAvOrdgoICV1RU5Mln+0PFvnpufnIZpXvqeGj6OUwamul1JBHpAcxspXOuoL3XdKXoKcrsFcNLd45jSJ9E7py9kr99UuZ1JBHp4VTonZAaH8Vztxcytn8q97z0OU99vNnrSCLSg6nQOykhOoKnbh3NpWdmcu/cYh6Yv063qxMRT6jQ/SAmMpyHpp/DTQW5/OG9En762hpatM2uiHSzCK8DhIqI8DB+dd0wkuMieezDTVTXNfLfN4wgKkJ/Z4pI91Ch+5GZ8ePLzyA5Lopfv/0l++obeeTmUcRGhXsdTUR6AA0fu8Bd55/GL78+jA/XV3HLn5ZRc7DR60gi0gOo0LvItDF5/HH6Oawqq+Gmx5ZQua/e60giEuJU6F3o8mFZPHXraLbtqeP6R5ewbbdujCEiXUeF3sW+NiiN52aOZV99I9c/upgvd+7zOpKIhCgVejcYmZfCS3eOwwxufHQJK7fu9TqSiIQgFXo3GZyZyJxZ40mNj2LGk8tYuD5kN6MUEY+o0LtRbmocL88aT/+0eGb+ZQWvfx6Sd+oTEY+o0LtZemI0L95ZyMjcFP7txU+ZvVR7qouIf6jQPdArJpJnbhvDhadn8JNXV/PQ+yXa/0VEOk2F7pGYyHAevWUU147M5v5567jvjbXa/0VEOkWX/nsoMjyM394wgqTYSJ78eDN76xr59XXDiAjX37MicvJU6B4LCzN+etVQUuKi+N0769lX38j/TBtJTKT2fxGRk6OhYAAwM747aRA/v/pMFhRXcOufl7O/Xvu/iMjJUaEHkG+M78eDU8+maMtepj+xjN21h7yOJCJBRIUeYKacnc0T/1LA+or93PDYErZXH/Q6kogECRV6ALpgSAazZ46lav8hrn9kMSWVtV5HEpEgoEIPUKP7pfLXO8bR2Oy44dHFrCqr9jqSiAQ4FXoAG9q3F3NmjSM+OoJpjy9l8cZdXkcSkQCmQg9w/dLieeWu8WSnxHLrUyuYt2an15FEJECp0INAZq8YXrpzHGdm9+Ku2St5qajU60giEoBU6EEiOS6K52aO5dyBafxwziqe+HCT15FEJMCo0INIXFQET36jgCuGZXHfm2v5zdtfalMvETlMl/4HmeiIcP4wbSRJcZE8/MFGqg828ospZxEeZl5HExGPdWiEbmaTzWydmZWY2Y+OccyNZlZsZmvM7Hn/xpS2wsOM+645i7svOI3nl23jjmeKqNhX73UsEfHYCQvdzMKBh4DLgKHANDMbetQxg4AfA+c6584EvtcFWaUNM+M/Lh3Cz64aysclu7jotwv5y+ItNGsLXpEeqyMj9DFAiXNuk3OuAXgRmHLUMbcDDznn9gI45yr9G1OO5dZz+zP/+xMYmZfMT19bw9cfXsSaHTVexxIRD3Sk0LOBtufJlbU+19ZgYLCZLTKzpWY2ub03MrM7zKzIzIqqqnSTZH/J7x3PM/86hgenns326oNc/cdF3PdGMQcONXkdTUS6kb/OcokABgHnA9OAJ8ws+eiDnHOPO+cKnHMF6enpfvpoAd8UzJSzs3n3nvO5sSCXJz7azMUPLOSd4gqvo4lIN+lIoW8Hcts8zml9rq0y4DXnXKNzbjOwHl/BSzdLiovkl18fxpxZ40iIiWDmM0Xc+WwR5TXatVEk1HWk0FcAg8ysv5lFAVOB14465lV8o3PMLA3fFIyufPFQQb9U5n7nPH44+XQWrq9i0m8X8udFm7VoKhLCTljozrkm4NvAPGAt8JJzbo2Z3WtmV7ceNg/YbWbFwPvAfzjndndVaOmYqIgwvnX+QOZ/byKj+qXy89eLufbhRazerkVTkVBkXl1pWFBQ4IqKijz57J7IOcfcVeX8/PVi9hw4xK3j+3PPJYNJiNa1ZSLBxMxWOucK2ntNl/73EGbGVSP68u4PJjJtTB5/XuxbNJ2v3RtFQoYKvYdJio3kvmuHMWfWeJJiI7nj2ZXc/kwRO3SrO5Ggp0LvoUblp/D6d77Gjy4bwkcbqpj0wEL+9PFmmppbvI4mIqdIhd6DRYaHMWviaSz4/kTG9k/lF3OLuebhRbrdnUiQUqELualxPHXraB6++Rwq9x3imocW8bPX1rC/vtHraCJyElToAvgWTS8flsU7P5jIjMJ8/rJkC5MeWMjbq8u157pIkFChyxF6xURy75Sz+Ntd40mNj2bW7E+4/ZkitmvRVCTgqdClXSPzUnj92+fyn5efwaKS3Vz8wEKe+HCTFk1FApgKXY4pIjyM2ycMYME9Eygc0Jv73lzL1X9cxGelWjQVCUQqdDmhnJQ4/vSNAh65+Rx2HzjEtQ8v4qd/X80+LZqKBBQVunSImXHZsCzeuWci3xjXj2eWbuXiBxby5hdaNBUJFCp0OSmJMZH87OozefVb55KWEM23nvuE2/5SROmeOq+jifR4KnQ5JSNyk/n73efykyvOYOmm3Vzyuw/5n3c3UNeguySJeEWFLqcsIjyMmecNYME9E5kwOI3fLljPhN98wDNLttDQpLNhRLqbCl06LTs5lsduKeCVu8YzID2e//r7Gi564ANe/XQ7Lbqhhki3UaGL34zKT+GvdxTy9DdHkxgdyff++hmX/+Ej3v+yUgunIt1AhS5+ZWacf3oGc7/zNR6cejZ1Dc188+kV3PTYUoq27PE6nkhIU6FLlwgLM6acnc0790zkF9ecxebdB7j+0SXM/MsKvty5z+t4IiFJt6CTblHX0MSfF23h0YUbqT3UxLVnZ/P9iweTmxrndTSRoHK8W9Cp0KVbVdc18MjCjTy9aAstznHz2HzuvmAg6YnRXkcTCQoqdAk4O2vqefDdDbxUVEp0RBgzv9afmRMG0Csm0utoIgFNhS4Ba1NVLb9dsJ43VpWTEhfJ3RcMZEZhPjGR4V5HEwlIKnQJeF+U1fCbeV/y0YZdZCXF8P1Jg/n6OdlEhGvdXqSt4xW6/rRIQBiWk8Szt43l+ZljyegVww9fWcWlv/9Qd0wSOQkqdAko4wem8eq3xvPojFEAzJr9Cdc8vJjFJbs8TiYS+FToEnDMjMln9WHe9ybwm+uHU7WvnulPLuOWPy3ji7Iar+OJBCzNoUvAq29sZvbSrTz0fgl76xq5YngWP7h4MAPSE7yOJtLttCgqIWF/fSNPfLSZJz/axKGmFm4syOW7Fw2iT1KM19FEuk2nF0XNbLKZrTOzEjP70XGOu87MnJm1+2EinZEYE8k9Fw/mwx9ewC2F+cxZWcrE+9/nl2+upbquwet4Ip474QjdzMKB9cDFQBmwApjmnCs+6rhE4A0gCvi2c+64w2+N0KWzSvfU8bsF6/nfz7aTEB3BrImn8c1z+xEXFeF1NJEu09kR+higxDm3yTnXALwITGnnuF8AvwbqTzmpyEnITY3jgZvO5q3vnsfY/r25f946Jt7/AQ+9X0LlPv02lJ6nI4WeDZS2eVzW+txhZnYOkOuce+N4b2Rmd5hZkZkVVVVVnXRYkfYM6dOLJ79RwCt3jWNwZgL3z1vH+F+9x6xnV7JwfZVusiE9Rqf/bWpmYcADwK0nOtY59zjwOPimXDr72SJtjcpP5bmZhWyqquWvK0p5eWUZb6/ZSU5KLNPG5HHDqBwyemkBVUJXR+bQxwE/c85d2vr4xwDOuV+2Pk4CNgK1rb+kD7AHuPp48+iaQ5eudqipmflrKnhh+TYWb9xNeJgx6YwMpo3JY8KgdMLCzOuIIietU6ctmlkEvkXRi4Dt+BZFpzvn1hzj+A+Af9eiqASStqP2PQcayE6OZdqYXG4oyCVTo3YJIp0+D93MLgd+D4QDTznn7jOze4Ei59xrRx37ASp0CVDHG7WfNyidcI3aJcDpwiKRdmzedYAXV2xjTlEZuzVqlyChQhc5jkNNzSworuD5Zf8ctV80JINpY31z7Rq1SyA5XqHrCgzp8aIjwrlyeF+uHN73iFH7/OIKspNjmTo6lxtHa9QugU8jdJF2NDS1ML94Jy8s38aiEo3aJXBohC5ykqIiwg6P2rfsOsALGrVLENAIXaSDGppafHPty7ceHrVfOCSD6Rq1SzfSCF3ED6IiwrhieBZXDM9iy64DvLiilDkrS1nQOmq/aXQuNxbkajtf8YxG6CKd8I9R+wvLt/Fxya7Do/ZpY3KZMChdN7kWv9MIXaSLHG/UnhofxeSz+nDl8CzG9u+tKRnpchqhi/hZQ1ML731ZydxVO3h3bSUHG5tJT4zm8rP6cOWIvozKS9E+MnLKdGGRiEfqGpp85f55Oe+vq+RQUwtZSTFcPiyLK4dncXZuMmYqd+k4FbpIAKg91MQ7xRXMXbWDheuraGx25KTEcsXwLK4a3pcz+/ZSucsJqdBFAkzNwUbmr9nJ3FXlLCrZRVOLo1/vON+57yOyOD0zUeUu7VKhiwSwvQcaeHvNTuau2sGSjbtpcTAwI4Erh2dx5fC+DMxI8DqiBBAVukiQqNp/iLdXl/P6qnJWbNmDc3BGVi+ubJ2Wyesd53VE8ZgKXSQI7ayp580vypm7agefbKsGYHhOElcOz+KK4X3JTo71OKF4QYUuEuTK9ta1lns5q8pqADgnL5krh/fliuFZ2lOmB1Ghi4SQrbsPMHeVr9zXlu/DDEb3S+Wq4VlcNiyLtIRoryNKF1Khi4SojVW1zP3cNy2zobKWMINxp/XmyuF9mXxmH1Lio7yOKH6mQhfpAdbt3M/cVTuYu6qczbsOEBFmjB+YxqVnZnLxGZlkaFomJKjQRXoQ5xxrduxj7qpy3lpdztbddQCMyE3mkqGZXDI0k4EZCTrPPUip0EV6KOccGyprWVBcwfw1O/m8dUG1X+84Lh6aycVD+zAqP0UbhwURFbqIAFCxr54FxRUsKK5gycbdNDS3kBofxYVDMrhkaCbnDUonNirc65hyHCp0EfmK/fWNLFxfxYLiCt77spL99U3ERIbxtYHpXDI0k4vOyKC3zpgJONoPXUS+IjEm8vB9UxubW1i+ec/hqZl31lZgBgX5KYenZvqnxXsdWU5AI3QROcI/FlX/MTVTXL4P8O0v4yv3TM7OSdae7h7RlIuInLKyvXW8U1zB/OIKlm3eQ3OLIz0xmkln+M6YGXdab2IiNe/eXVToIuIXNXWNvL+ukgXFFXywrpIDDc3ER4UzYXA6l5yZyQWnZ5Acp4uZupIKXUT87lBTM4s37mZBcQXvFFdQuf8Q4WHGmH6ph6dmclO1O6S/dbrQzWwy8CAQDjzpnPvVUa/fA8wEmoAq4F+dc1uP954qdJHQ0dLiWLW9hgXFO5m/poINlbUADOmTyCVDM5kwOJ1BmYkkxUZ6nDT4darQzSwcWA9cDJQBK4BpzrniNsdcACxzztWZ2V3A+c65m473vip0kdC1ZdeBw4uqRVv30NJaMxmJ0QzKTGBQRiKnZSQwqPWHTo/suM6etjgGKHHObWp9sxeBKcDhQnfOvd/m+KXAjFOPKyLBrl9aPLdPGMDtEwawu/YQn26rpqSqlg0VtZRU1fJyUSkHGpoPH58SF3lkyWcmMDAjgT69YrRFwUnoSKFnA6VtHpcBY49z/G3AW50JJSKho3dCNJOGZjKJzMPPOecor6mnpLKWDZW1lFTup6Sylje/KKfmYOPh4xKiI44YyQ/M8I3uc1JiddpkO/x6YZGZzQAKgInHeP0O4A6AvLw8f360iAQRM6Nvcix9k2OZMDj98PPOOXbVNlDSWvK+sq9l4foq5qwsO3xcTGQYA9ISWqdvfEU/MCOR/N5xRIaHefG/FBA6Uujbgdw2j3NanzuCmU0C/hOY6Jw71N4bOeceBx4H3xz6SacVkZBmZqQnRpOeGM2403of8VpNXSMlVfvZUFF7uOiLtuzl75/tOHxMZLjRr3e8b8omPYGBmYkMykigf1p8jzhXviOFvgIYZGb98RX5VGB62wPMbCTwGDDZOVfp95Qi0uMlxUUyKj+VUfmpRzx/4FATG9vMz2+oqKV4xz7eXr3z8GJsmEFeahz5vePpmxxLdnLM4X8h9E2KpU9SDFERwT+yP2GhO+eazOzbwDx8py0+5ZxbY2b3AkXOudeA+4EE4OXWBYxtzrmruzC3iAgA8dERDM9JZnhO8hHP1zc2s3nXgSPm6bftqWP19hp2H2g44lgzSE+Ibi35GPomxR4u/OzkWLKSY+gdHxXwC7S6sEhEepyDDc2U1xxkR3U9O6oPsqPmoO+/bR7XN7Yc8WuiI8IOF35WUmy7I/3u2HpYuy2KiLQRGxXOgPQEBqQntPu6c469dY3sqD7I9uqDlFcfZEdNPdurfcX/8YZdVOyv5+jxcEpc5JEj+6SYIx6nJ0Z36c1EVOgiIkcxM1Ljo0iNj+Ks7KR2j2lsbmFnjW9EX96m7HdUH6R0Tx1LN+1mf33TEb8mIszI7BXDreP7cfuEAX7PrUIXETkFkeFh5KbGHXe/mn31jZRX17eZ0vFN62T06porY1XoIiJdpFdMJL36RHJ6n8Ru+bzgP09HREQAFbqISMhQoYuIhAgVuohIiFChi4iECBW6iEiIUKGLiIQIFbqISIjwbHMuM6sCjnsj6eNIA3b5MU6w0/dxJH0f/6Tv4kih8H3kO+fS23vBs0LvDDMrOtZuYz2Rvo8j6fv4J30XRwr170NTLiIiIUKFLiISIoK10B/3OkCA0fdxJH0f/6Tv4kgh/X0E5Ry6iIh8VbCO0EVE5CgqdBGREBF0hW5mk81snZmVmNmPvM7jJTPLNbP3zazYzNaY2Xe9zuQ1Mws3s0/NbK7XWbxmZslmNsfMvjSztWY2zutMXjGz77f+GVltZi+YWYzXmbpCUBW6mYUDDwGXAUOBaWY21NtUnmoCfuCcGwoUAnf38O8D4LvAWq9DBIgHgbedc0OAEfTQ78XMsoF/Awqcc2cB4cBUb1N1jaAqdGAMUOKc2+ScawBeBKZ4nMkzzrly59wnrT/fj+8PbLa3qbxjZjnAFcCTXmfxmpklAROAPwE45xqcc9XepvJUBBBrZhFAHLDD4zxdItgKPRsobfO4jB5cYG2ZWT9gJLDM2ySe+j3wQ6DF6yABoD9QBfy5dQrqSTOL9zqUF5xz24H/BrYB5UCNc26+t6m6RrAVurTDzBKAV4DvOef2eZ3HC2Z2JVDpnFvpdZYAEQGcAzzinBsJHAB65JqTmaXg+5d8f6AvEG9mM7xN1TWCrdC3A7ltHue0PtdjmVkkvjJ/zjn3N6/zeOhc4Goz24JvKu5CM5vtbSRPlQFlzrl//IttDr6C74kmAZudc1XOuUbgb8B4jzN1iWAr9BXAIDPrb2ZR+BY2XvM4k2fMzPDNka51zj3gdR4vOed+7JzLcc71w/f74j3nXEiOwjrCObcTKDWz01ufuggo9jCSl7YBhWYW1/pn5iJCdIE4wusAJ8M512Rm3wbm4Vupfso5t8bjWF46F7gF+MLMPmt97v865970MJMEju8Az7UOfjYB3/Q4jyecc8vMbA7wCb4zwz4lRLcA0KX/IiIhItimXERE5BhU6CIiIUKFLiISIlToIiIhQoUuIhIiVOgiIiFChS4iEiL+P2v9pfJSSonHAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "1OF56fTFxL-i" }, "source": [ "After instantiating a trained model, we can then score the test data and compute its accuracy. We then print the classification report and plot a confusion matrix.\n" ] }, { "cell_type": "code", "metadata": { "id": "LbZR-3elxL-i" }, "source": [ "batch_size = 8\n", "\n", "test_data = TensorDataset(test_x, test_m)\n", "test_sampler = SequentialSampler(test_data)\n", "test_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=batch_size)\n", "\n", "outputs = []\n", "with torch.no_grad():\n", " model.eval()\n", " for k, (mb_x, mb_m) in enumerate(test_dataloader):\n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " output = model(mb_x, attention_mask=mb_m)\n", " outputs.append(output[0].to('cpu'))\n", "\n", "outputs = torch.cat(outputs)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "JvaygEtyxL-j" }, "source": [ "_, predicted_values = torch.max(outputs, 1)\n", "predicted_values = predicted_values.numpy()\n", "true_values = test_y.numpy()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "2OpivSd7xL-l", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "08d7fdc5-84f7-44ad-8301-ad694959a21c" }, "source": [ "test_accuracy = np.sum(predicted_values == true_values) / len(true_values)\n", "print (\"Test Accuracy:\", test_accuracy)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Test Accuracy: 0.6903553299492385\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "WGFsZIRKxL-o" }, "source": [ "import warnings\n", "warnings.filterwarnings('always')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "SU4J7XrJxL-q", "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "outputId": "7259f700-41cc-43fc-b840-036f2ffb5e1a" }, "source": [ "from sklearn.metrics import classification_report\n", "\n", "print(classification_report(true_values, predicted_values, target_names=[str(l) for l in label_values]))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " zero 0.71 0.71 0.71 77\n", " one 0.91 0.89 0.90 46\n", " two 0.54 0.55 0.55 47\n", " three 0.52 0.52 0.52 27\n", "\n", " accuracy 0.69 197\n", " macro avg 0.67 0.67 0.67 197\n", "weighted avg 0.69 0.69 0.69 197\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "lw61mJaAxL-s" }, "source": [ "Analysis" ] }, { "cell_type": "code", "metadata": { "id": "RVx12wMxxL-s" }, "source": [ "import itertools\n", "\n", "# plot confusion matrix\n", "# code borrowed from scikit-learn.org\n", "def plot_confusion_matrix(cm, classes,\n", " normalize=False,\n", " title='Confusion matrix',\n", " cmap=plt.cm.Blues):\n", " \"\"\"\n", " This function prints and plots the confusion matrix.\n", " Normalization can be applied by setting `normalize=True`.\n", " \"\"\"\n", " if normalize:\n", " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", " print(\"Normalized confusion matrix\")\n", " else:\n", " print('Confusion matrix, without normalization')\n", "\n", " print(cm)\n", "\n", " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", " plt.title(title)\n", " plt.colorbar()\n", " tick_marks = np.arange(len(classes))\n", " plt.xticks(tick_marks, classes, rotation=45)\n", " plt.yticks(tick_marks, classes)\n", "\n", " fmt = '.2f' if normalize else 'd'\n", " thresh = cm.max() / 2.\n", " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", " plt.text(j, i, format(cm[i, j], fmt),\n", " horizontalalignment=\"center\",\n", " color=\"white\" if cm[i, j] > thresh else \"black\")\n", "\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "6vzjDOtTxL-u", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "77a6c01e-66dd-4a05-8da7-5fd3e8762338" }, "source": [ "from sklearn.metrics import classification_report, confusion_matrix\n", "\n", "cm_test = confusion_matrix(true_values, predicted_values)\n", "\n", "np.set_printoptions(precision=2)\n", "\n", "plt.figure(figsize=(6,6))\n", "plot_confusion_matrix(cm_test, classes=label_values, title='Confusion Matrix - Test Dataset')\n", "plt.figure(figsize=(6,6))\n", "plot_confusion_matrix(cm_test, classes=label_values, title='Confusion Matrix - Test Dataset', normalize=True)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Confusion matrix, without normalization\n", "[[55 2 16 4]\n", " [ 1 41 1 3]\n", " [14 1 26 6]\n", " [ 7 1 5 14]]\n", "Normalized confusion matrix\n", "[[0.71 0.03 0.21 0.05]\n", " [0.02 0.89 0.02 0.07]\n", " [0.3 0.02 0.55 0.13]\n", " [0.26 0.04 0.19 0.52]]\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaYAAAGlCAYAAACx5VKtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de/zX8/3/8du9VKSUFMLInOUQRZLS5MxWM+dTiBjNLIZtNmZ8x2bYgZ81zHGGzfncyCGHIssh5rQ0KaSEEjo8fn+8Xh8+Purz+VTvz/t1+NyvXd6Xz/v9Oj7er8+n9+P9eL6er+dLEYGZmVletMg6ADMzs9qcmMzMLFecmMzMLFecmMzMLFecmMzMLFecmMzMLFeWyzoAMzOrvJYrrRMxf27Fthdzp98fEbtXbIP1cGIyMyuhmD+XNhvtX7HtfTrhks4V21gDnJjMzEpJoGKerSlm1GZmVlqumMzMykiAlHUUS8WJycysrNyUZ2ZmtuxcMZmZlZWb8szMLD/cK8/MzKwiXDGZmZVVQZvyXDGZmVmuuGIyMysjUdhzTE5MZmalJDflmZmZVYIrJjOzsnJTnpmZ5Yqb8szMzJadKyYzs1LyyA9mZmYV4YrJzKyMfD8mMzPLHTflmZmZLTtXTGZmpVTczg9OTGZmZdWimOeYiplOzcystFwxmZmVkUcXNzOz3Clod/FiplMzMystV0xmZqVU3F55xYzazMxKyxWTmVlZFfQckxOTmVlZuSnPzMxs2bliMjMrI6nqTXmS3gQ+BhYA8yOil6ROwI1AN+BNYP+I+KC+7bhiMjMrK7Wo3KPxvhURPSKiV/r6dODBiNgAeDB9XS8nJjMza0qDgKvT51cDgxtawYnJFkvSCpLulPShpJuXYTuHSHqgkrFlQdK9koZkHYdZo9U051Xi0TgBPCBpvKRh6bTVImJa+vwdYLWGNuLEVAKSDpb0jKTZkqalH6A7VGDT+5L8Ea0SEfst7UYi4vqI2LUC8XyFpAGSQtKtdaZvmU5/uJHbOUvSdQ0tFxF7RMTVDS23JCT1S39vsyXNSeOeXeux9lJsMyStX8/8IyQtqLWPSZL+KmnDJdjHVZLOWdLYllS19mON0jn9nKl5DFvEMjtExNbAHsAJkvrXnhkRQZK86uXODwUnaQRJm+1xwP3A58DuJOXzmGXc/DrAqxExfxm305SmA30krRIRM9JpQ4BXK7UDSQIUEQsrtc0aEfEY0C7dTzdgEtCxCsf8yYjYQVJLkpPSJwPjJfWJiBebeN9WFRUf+eH9WueNFiki3k5/vpd+YdwWeFdS14iYJqkr8F5DO3LFVGCSOgBnAydExC0RMSci5kXEnRHx43SZNpIuljQ1fVwsqU06b4CkKZJOlvReWm0dmc77JfAL4ID0W/XQupWFpG7pt/Pl0tdHSPqvpI/Tb+GH1Jo+ptZ620t6Om0ifFrS9rXmPSzpV5IeT7fzgKTO9RyGz4HbgAPT9VsCBwDX1zlWv5f0lqSP0maGfun03YGf1nqfz9WK41xJjwOfAN9Mpx2dzv9/kv5Za/vnS3owTWIVIamDpCvS38vbks5J3x+S1pf0SHoM35d0Yzr90XT159L3c0B9+4iIBRHxRkQcDzwCnFVr/zdLeifdx6OSuqfThwGHAKem+7gznX66pDfS39tLkr5ba1uLjDedt7GkUZJmSnpF0v717ceWQBWb8iStKKl9zXNgV+BF4A6SL4ukP29vaFtOTMXWB1geuLWeZX4GbAf0ALYk+QZzRq35qwMdgDWBocAlklaOiDOB/wNujIh2EXFFfYGkf4h/APaIiPbA9sCERSzXCbg7XXYV4ELgbkmr1FrsYOBIYFWgNXBKffsGrgEOT5/vRvKfYWqdZZ4mOQadgL8BN0taPiLuq/M+t6y1zmHAMKA9MLnO9k4GNk+Tbj+SYzckbaqolKuA+cD6wFYk/9GPTuf9CngAWBlYC/gjQETUNJ1smb6fG2m8W4B+tV7fC2xA8nt4ljTZR8TI9Plv0n18O13+jXT9DsAvgevSb8iLjTf9uxlF8jtZleQLxqWSNq1nP5ZPqwFj0i9344C70/9f5wG7SHoN2Dl9XS8npmJbhaS8rq/Z5xDg7Ih4LyKmk3xgHFZr/rx0/ryIuAeYDWy0lPEsBDaTtEJETIuIiYtYZi/gtYi4NiLmR8QNwH+A2h86f42IVyNiLnATSUJZrIh4AugkaSOSBHXNIpa5LiJmpPv8HdCGht/nVRExMV1nXp3tfUJyHC8ErgN+EBFTGtheo0laDdgTOCmthN8DLiKtDEl+b+sAa0TEpxGxrM22kCTzTjUvIuLKiPg4Ij4jqaS2TKv0RYqImyNiakQsTBPiayRfhOqLd2/gzYj4a3qc/w38E1jqc5qWqrkfU5W6i0fEfyNiy/TRPSLOTafPiIiBEbFBROwcETMb2pYTU7HNIDkhWd+5wjX46rf9yem0L7ZRJ7F9QnrOY0lExBySJrTjgGmS7pa0cSPiqYlpzVqv31mKeK4FhgPfYhEVpKRTJL2cNiXNIvlWX18TIcBb9c2MiLHAf0k+Am5a3HKSJurLjgb9FrdcHesArUiO5aw05j+TVBUAp6b7HZdu/6hGbrc+awIz05hbSjovbZr7iOTCSKjnmEk6XNKEWvFuVmv5xcW7DtC7Zp10vUNIKnlbJqpqYqokd34otieBz0iuC/jHYpaZSvKfv6Z6WZuvN3M11hygba3XX/nwiIj7gfslrQCcA/yFrzYN1Y6ntrWB+5YyphrXAq8D10TEJ7VP9aTJ4FRgIDAxIhZK+oDkgxIW30uo3mY5SSeQVF5T0+3/epEbiei+BO+jxlskv9vOi6qII+Id4Jg0jh2Af0l6NCJeX4p91fgu8Fj6/GCSDjQ7kySlDsBij5mkdUh+3wNJOlYskDShZvnFxZu+z0ciYpfFxFTJplErCFdMBRYRH5J0ULhE0mBJbSW1krSHpN+ki90AnCGpS9qJ4BckTU9LYwLQX9LaaZPOT2pmSFpN0qD0nMFnJE2Ci+rFdg+woZIu7sulJ+c3Be5aypgAiIhJwI4k59Tqak9yrmY6sJykXwAr1Zr/LtBNavzXQiVdq88BDiVp0jtVUr1Njksive7jAeB3klaS1ELSepJ2TPe/n6S10sU/IPkArzne7wLfbOT7aClpXUl/BAaQNPVCcsw+I6nK25Kch6ut7j5WTGOYnm73SJKKqWY/i4v3LpK/h8PSv91WkraRtMmSvhdbhOpfx1QRTkwFl54vGUHSoWE6yTfQ4SQ91SD58HwGeB54geQk9lJdFxIRo0jGvHoeGM9Xk0mLNI6pJM1BOwLfX8Q2ZpCcVziZ5EPvVGDviHh/aWKqs+0xEbGoavB+korsVZJmw0/5ajNdzcXDMyQ929B+0qbT64DzI+K5iHiNpGfftUp7PFbI4SSdP14i+TD/B1DTmWAbYKyk2SS9nn4YEf9N550FXJ02je2/mG33Sdf9CHiYJFFvExEvpPOvITlWb6f7f6rO+lcAm6b7uC0iXgJ+R1LFvwtsDjxea/lFxhsRH5N06jiQ5G/nHeB8kkr0a/tp6IBZOaiynYjMzCwPWnRcJ9rs+NOKbe/TO44b39B1TJXic0xmZmVV0BsFuinPzMxyxRWTmVkZqeJDElWNE5OZWVm5Kc/MzGzZlbpi0nIrhFq3zzqMQuixyRLfXaFZm/NZngdcz5cVWrfMOoTCeOt/k5nx/vuVHAi4UpuqqnInptbtabPR4i7jsNoefeIPWYdQKGMnNTjcl6V6rNUx6xAKY2D/3hXblihuYnJTnpmZ5UqpKyYzs2ZLfDmyYcG4YjIzs1xxxWRmVkoq7DkmJyYzs5IqamJyU56ZmeWKKyYzs5IqasXkxGRmVlJFTUxuyjMzs1xxxWRmVka+jsnMzKwyXDGZmZWQfB2TmZnlTVETk5vyzMwsV1wxmZmVVFErJicmM7OSKmpiclOemZnliismM7My8nVMZmZmleGKycyspIp6jsmJycyshIp8ga2b8szMLFdcMZmZlVRRKyYnJjOzsipmXnJTnpmZ5YsrJjOzMlJxm/JcMZmZWa64YjIzK6miVkxOTE3sP3f/ko/nfMaChQuZv2AhOxzyG3527J4ctc/2TP9gNgBn/ukO7h/zUsaR5seUt95i2NAjeO+9d5HEkUOP4fjhJ2YdVq789mcnMvbhUXTs1JnL73zsi+m3XvcX7vjblbRo0ZLeO+7CsB+fmWGU+bVgwQJ27t+b1buuyQ3/uD3rcJqME5Mt1u7Dfs+MWXO+Mu2P143m4msfzCiifFtuueX4v/N/S4+ttubjjz+mX59t2Gngzmy8yaZZh5Ybuw0+kMEHD+X804d/MW3C2DE88eB9/Pm2h2ndug0fzJieYYT59udL/8AGG23Cxx99lHUotgg+x2S5s3rXrvTYamsA2rdvz0Ybb8zUt9/OOKp82WKb7WnfceWvTLvj73/lwGNOpHXrNgCsvEqXLELLvalvT2HU/fdy6JCjsg6lSdWM/FCpRzU5MTWxiODOS4fz+PWnctQ+fb+YftyB/Rl340+47MxD6Nh+hQwjzLfJb77J8xMm0Gvb3lmHkntvv/kGL45/iuEH7MaIw77Df174d9Yh5dLPTjuZM3/1a1q0aAYff6rgo4qawW8mWwOPvIjtDz6fwcMv5dgD+tF36/X4y82Psem3z6L3gefxzvsfcd6IfbIOM5dmz57NoQftx3kXXMhKK62UdTi5t2D+Aj768AP++Pf7GPbjszjnR0cTEVmHlSv333s3nbt0ocdWPbMOxepRiMQkqWXWMSytqdM/BGD6B7O546Hn2aZ7N96b+TELFwYRwZW3PE6vzdbJOMr8mTdvHoceuC/7H3gwgwY7cTdG59W70m+XvZHExltsjVq04MMPZmQdVq6Me+oJ7rvnLrbqvj7DjjiEMY+O5rijD886rKaRXsfkprwGSDpO0oT0MUnSaEm7SnpS0rOSbpbULl32TUnnS3oW2E/SQZJekPSipPOrGffSart8a9q1bfPF8537bMzEN6ayeucvv/0P2mlLXnpjWlYh5lJEcMKxR7PRxpvwgx/+KOtwCqPvwD2ZMHYMAFMmvcH8eZ/TYeVVMo4qX37+y3N54ZU3+ffE1xl51fXs0P9bXHb5NVmH1WSKmpiq2isvIi4DLpPUCngIuBI4A9g5IuZIOg0YAZydrjIjIraWtAbwFNAT+AB4QNLgiLit7j4kDQOGAdCqXVO/pXqtukp7brzwGACWa9mSG+99hlFPvMwVvzqcLTZai4hg8rSZ/OCcGzKNM2+efOJxbvjbdXTfbHO23zbpBHHm2eew2+57ZhxZfpx78jCeG/c4H86ayYEDtmDI8FPZfZ+DueCMH3L0t/uxXKtWnPrrPxW2u7A1b8qiDVrSpcB04GngKmBKOqs18GREDJX0JrBjREyWNAj4XkQcnq4/FOgeESPq20+LtqtGm432b6J3US7Tn/pD1iEUythJM7MOoTB6rNUx6xAKY2D/3kx4dnxFvk20XnX9WG2/31ViUwBMuXTw+IjoVbEN1qPq1zFJOgJYBxgO7AWMioiDFrP4nMVMNzOzkqr2OaaewCnAoRGxkKR5rq+k9dP5K0racBGrjgN2lNQ57QhxEPBIteI2MyukgnYXr3bFNBzoBIxO276fAY4AbpDUJl3mDODV2itFxDRJpwOjSQ7R3RFR3nFEzMwqoKjnGKvd+eHIxczaZhHLdqvz+gbAvQTMzErOY+WZmZVQFt28K8WJycyspIqamAox8oOZmTUfrpjMzErKFZOZmVkFuGIyMyurYhZMTkxmZmXlpjwzM7MKcMVkZlZGKm7F5MRkZlZCAgqal9yUZ2Zm+eKKycyslIo7JJErJjMzyxUnJjOzkpIq92j8PtVS0r8l3ZW+XlfSWEmvS7pRUuuGtuHEZGZWUjUjjFfisQR+CLxc6/X5wEURsT7wATC0oQ04MZmZWUVIWgvYC7g8fS1gJ+Af6SJXA4Mb2o47P5iZldESNsFVyMXAqUD79PUqwKyImJ++ngKs2dBGnJjMzEpIQIsWFc1MnSU9U+v1yIgY+cX+pL2B9yJivKQBy7IjJyYzM2uM9yOiVz3z+wLfkbQnsDywEvB7oKOk5dKqaS3g7YZ25HNMZmYlVc1eeRHxk4hYKyK6AQcCD0XEIcBoYN90sSHA7Q1ty4nJzKykMuqVV9dpwAhJr5Occ7qioRXclGdmZhUVEQ8DD6fP/wtsuyTrOzGZmZVRNr3yKsJNeWZmliuumMzMSii57UUxSyYnJjOzUvLo4mZmZhXhisnMrKQKWjA5MZmZlZWb8szMzCrAFZOZWRkV+DqmUiemrTZZm8fH/inrMAphs9PvzTqEQnnxvD2yDqEwPp23IOsQCkMUNJNUWKkTk5lZc+XrmMzMLHcKmpfc+cHMzPLFFZOZWUm5Kc/MzHKloHnJTXlmZpYvrpjMzMpIxW3Kc8VkZma54orJzKyEkuuYso5i6TgxmZmVku/HZGZmVhGumMzMSqqgBZMTk5lZWbkpz8zMrAJcMZmZlZHvx2RmZnlS5NteuCnPzMxyxRWTmVlJuWIyMzOrAFdMZmYlVdCCyYnJzKys3JRnZmZWAa6YzMzKyNcxmZlZnsiji5uZmVWGKyYzs5IqaMHkisnMzPLFFZOZWUm1KGjJ5MRkZlZSBc1LbsozM7N8cWKqkmOPPoq111iVnj02yzqUXGshuONHfRl5VE8ADuu7Ng+e3p/XL9iDldu2yji6fPLfVuN9+umnDOy3HTv03po+Pbfg1786K+uQmoyUjPxQqUc1OTFVyWFDjuD2u+7LOozcO6JfN15/d/YXr8dPmsXhf36aKTM/yTCqfPPfVuO1adOG2+/9F2PGPsujT43nwVH38/S4p7IOq8m0UOUeVY27urtrvnbo159OnTplHUaurd5heQZs0oWbxr31xbSXpn7E2x/MzTCq/PPfVuNJol27dgDMmzePefPmIwp6IqbEnJgsN84YtAnn3/UKEVlHYmW2YMEC+vXuyYbrdGXAwIH02rZ31iE1GTflmS2Db23ShRmzP2Pi2x9lHYqVXMuWLXls7HgmvjaZZ595mpcmvph1SFaHu4tbLvTstjIDN12NHTfuQpvlWtJu+eX43UFbcPINz2cdmpVUh44d6dd/AA+Oup9Nu5ez40hRu4tnmpgkjQCOSl9eDtwG3AuMAbYH3gYGRcRcSesBlwBdgE+AYyLiP9WP2prCBfe+ygX3vgpA7/U6MXTHdZ2UrOLenz6dVq1a0aFjR+bOncvoh/7FD0f8OOuwmoSgsOfPMmvKk9QTOBLoDWwHHAOsDGwAXBIR3YFZwPfSVUYCP4iInsApwKVVD3oZHH7oQQzo14dXX3mF9bqtxVVXXpF1SIVw+A7rMOaMb7F6h+W56+Qd+L/9yvnNdln4b6vx3nlnGt/efWf6brsVO/Xbjm/ttDO777l31mFZHVlWTDsAt0bEHABJtwD9gEkRMSFdZjzQTVI7kgrq5lon4dosaqOShgHDAL6x9tpNF/0Suua6G7IOoTDGvjGTsW/MBOCaMZO5ZszkjCPKN/9tNd5mm2/Bo089k3UYVVPtbt6VksdzTJ/Ver4AWIGkspsVET0aWjkiRpJUV/Ts2cv9u8ysecqgN12lZNkr7zFgsKS2klYEvptO+5qI+AiYJGk/ACW2rF6oZmZWLZklpoh4FrgKGAeMJen88EE9qxwCDJX0HDARGNTUMZqZFZlUuUc1ZdqUFxEXAhfWmbxZrfkX1Ho+Cdi9SqGZmVlG8niOyczMlpHw/ZjMzCxnCpqXPCSRmZnliysmM7OSKmp3cScmM7MSyqI3XaW4Kc/MzHLFFZOZWUm5V56ZmeVKMdOSm/LMzCxnXDGZmZVU6XrlSfojsNjRuSPixCaJyMzMmrX6Kqbmc9MSM7OSSYYkyjqKpbPYxBQRV9d+LaltRHzS9CGZmdkyK/P9mCT1kfQS8J/09ZaSCnVbczMzK47G9Mq7GNgNmAEQEc8B/ZsyKDMzW3bVvB+TpOUljZP0nKSJkn6ZTl9X0lhJr0u6UVLrhrbVqO7iEfFWnUkLGrOemZllR2lzXiUejfAZsFNEbAn0AHaXtB1wPnBRRKxPcjPYoQ1tqDGJ6S1J2wMhqZWkU4CXGxOlmZk1D5GYnb5slT4C2An4Rzr9amBwQ9tqTGI6DjgBWBOYSpIJT1jCmM3MrIpqeuVV6gF0lvRMrcewr+1TailpAvAeMAp4A5gVEfPTRaaQ5JJ6NXiBbUS8DxzS6KNhZmZl9H5E9KpvgYhYAPSQ1BG4Fdh4aXbUmF5535R0p6Tpkt6TdLukby7NzszMrHqqfI7pCxExCxgN9AE6SqopgtYC3m5o/cY05f0NuAnoCqwB3AzcsERRmplZ1amCjwb3JXVJKyUkrQDsQtIfYTSwb7rYEOD2hrbVmMTUNiKujYj56eM6YPlGrGdmZs1HV2C0pOeBp4FREXEXcBowQtLrwCrAFQ1tqL6x8jqlT++VdDrwd5IeFgcA9yxb/GZm1pSk6t6PKSKeB7ZaxPT/Atsuybbq6/wwniQR1byzY2vvC/jJkuzIzMyqq6AjEtU7Vt661QzEzMwMGnk/JkmbAZtS69xSRFzTVEGZmdmyK+ogrg0mJklnAgNIEtM9wB7AGMCJyczMKq4xvfL2BQYC70TEkcCWQIcmjcrMzJZZNQdxraTGNOXNjYiFkuZLWolkqIlvNHFcZma2DISq2iuvkhqTmJ5JL5r6C0lPvdnAk00alZmZNVuNGSvv+PTpZZLuA1ZK+6ubmVleZdAEVyn1XWC7dX3zIuLZpgnJzMwqoYy98n5Xz7yae2zk2iefL+C5ybOyDqMQXjxvj6xDKJRfjXo16xAK4/g+3bIOoTAWRmQdQi7Ud4Htt6oZiJmZVVajblGeQ426wNbMzIpFFLcpr6gJ1czMSsoVk5lZSbUoZsHUqDvYStKhkn6Rvl5b0hINYW5mZtZYjWnKu5Tk9rgHpa8/Bi5psojMzKwiWqhyj2pqTFNe74jYWtK/ASLiA0mtmzguMzNbBskYd8Vsy2tMxTRPUkuSa5eQ1AVY2KRRmZlZs9WYiukPwK3AqpLOJRlt/IwmjcrMzJZZUTs/NGasvOsljSe59YWAwRHxcpNHZmZmy6SgLXmNulHg2sAnwJ21p0XE/5oyMDMza54a05R3N8n5JZHcWn1d4BWgexPGZWZmy0BQ3vsxRcTmtV+no44fv5jFzczMlskSj/wQEc9K6t0UwZiZWeUUdcy5xpxjGlHrZQtga2Bqk0VkZmYVUdCWvEZVTO1rPZ9Pcs7pn00TjpmZNXf1Jqb0wtr2EXFKleIxM7MKkFS+zg+SlouI+ZL6VjMgMzOrjILmpXorpnEk55MmSLoDuBmYUzMzIm5p4tjMzKwZasw5puWBGcBOfHk9UwBOTGZmOVbGIYlWTXvkvciXCalGNGlUZmbWbNWXmFoC7fhqQqrhxGRmlmNlHflhWkScXbVIzMysogqal+q9MLigb8nMzIqsvoppYNWiMDOzysrgluiVstjEFBEzqxmImZlVlgra8FXUMf7MzKyklnh0cTMzy7+kV17WUSwdJyYzs5IqamJyU14TOuf04eyx7QYcvEefr827/vI/sd36KzNr5owMIsu/Y48+irXXWJWePTbLOpTc+Wj6NK47/TD+fOyejDxuL8bddvUX856+41ouG7Y7I4/bi4eu+E2GUebXhx/OYtiQA9lx280Z0HsLxo97KuuQrA5XTE1or30OYt9Dj+HsHx/3lenvTp3CuDGjWX2NtTKKLP8OG3IExx0/nKOPOjzrUHKnRcuW7Hz06ay+fnc++2Q2fz3xe6y7dV/mfPA+rz31IEdfcgfLtWrNnFn+0rMoZ55+MgMG7srIq//O559/zty5n2QdUpNRQS9kcsXUhLbati8rdVz5a9MvPvdnDD/trOJe/VYFO/TrT6dOnbIOI5fadVqV1dfvDkCbtu1YZe1vMvv9d3n27hvos98wlmvVGoAVO66SZZi59NGHHzL2icc46LAjAWjdujUdOnTMOCqry4mpyh4ddQ9dVu/KBptsnnUoVgKz3p3Cu2+8zBobb8nMqW/y1sRnuOqk/bj21EOZ+urzWYeXO2/97006de7CiBOOYbf+23LKicfxyZw5Da9YQDWdHyr1qKbMEpOkjpKOz2r/Wfh07idcddmFDDvpJ1mHYiXw+dw53HLuiew87Ke0aduOhQsWMPfjDxly0U0MHHoqt/76JCI8rGVt8+fP58Xn/s1hRw3j/kfH0bZtWy65+LdZh9U0lDTKVOpRTVlWTB2BZpWYpvxvEtPemsyhe/dj8I5bMP2dqQwZtCMzpr+bdWhWMAvmz+Of555I9wHfZuO+uwKwUufV2Gj7XZDEGhttgdSCTz76IONI86XrGmvSdY212LrXtgDs9Z19eOG5f2ccldWVZWI6D1hP0gRJf5X0HQBJt0q6Mn1+lKRz0+cjJL2YPk7KMO6ltv5G3bl33Gvc9sjz3PbI83RZfQ2uvv0RVumyWtahWYFEBHdf/DM6f+Ob9N7nyC+mb7jdzkx+fiwAM6ZMYsH8ebRd6evnOJuzVVdbnTXWXIs3XnsFgDGPjmaDjTbJOKqm0yK9vXolHlWNu6p7+6rTgTciogdwP9Avnb4msGn6vB/wqKSewJFAb2A74BhJW1U53iX285OGcsx+uzJ50ut8u2937rjp2qxDKozDDz2IAf368Oorr7Bet7W46sorsg4pN6a8NJ4XH7qdN597isuHD+Ly4YN4/elH2HLX7zHrnbcY+f29ue38EXx7xHmF7ZXVlH71m4v4wbAj2LlvT1564Tl+cPJpWYfUJIp8jikv3cUfA06StCnwErCypK5AH+BE4Cjg1oiYAyDpFpKk9bUaXNIwYBiQeXfsX11c/4fpbY/45PTiXHPdDVmHkFvf6N6Ln97zyiLnDfrxBVWOpni6b74l94x+MuswrB656JUXEW+TnHPaHXiUJFHtD8yOiI+XcFsjI6JXRPTq2Klz5YM1MysId35Ych8D7Wu9fgo4iS8T0ynpT9KfgyW1lbQi8N1a88zMrEQya8qLiBmSHpf0InAvSaLZNSJelzQZ6JROIyKelXQVMD5rjZQAABPfSURBVC5d/fKIcFcaM7PFEi0KetuLTM8xRcTBdSZdkU6fB6xYZ9kLgQurFJqZWaGJ4g4uk4tzTGZmZjXy0ivPzMwqqYy3Vjczs2Kr9oWxleKmPDMzyxVXTGZmJeTOD2ZmZhXiisnMrKSKeo7JicnMrKQKmpfclGdmZvniisnMrIREcSsPJyYzszIShb0fV1ETqpmZlZQrJjOzkipmveSKyczMcsaJycyshERyHVOlHg3uT/qGpNGSXpI0UdIP0+mdJI2S9Fr6c+WGtuXEZGZWUqrgoxHmAydHxKbAdsAJkjYFTgcejIgNgAfT1/VyYjIzs2UWEdMi4tn0+cfAy8CawCDg6nSxq4HBDW3LnR/MzEoqq97ikroBWwFjgdUiYlo66x1gtYbWd2IyMyslVfo6ps6Snqn1emREjPzaXqV2wD+BkyLio9oxRERIioZ25MRkZmaN8X5E9KpvAUmtSJLS9RFxSzr5XUldI2KapK7Aew3tyOeYzMxKqGZIoko9GtxfUhpdAbwcERfWmnUHMCR9PgS4vaFtuWIyMyupKg9J1Bc4DHhB0oR02k+B84CbJA0FJgP7N7QhJyYzM1tmETGGxfcsH7gk23JiMjMrKQ9JZGZmVgGumMzMyqjAt71wYjIzK6Ei3yiwqHGbmVlJuWIyMyspN+WZmVmuFDMtuSnPzMxyxhWTmVlJFbQlr9yJqVXLFnTtuHzWYVgJndh33axDKIw3p8/JOoTC+Hz+wqxDyIVSJyYzs+Yq6S5ezJLJicnMrKSK2pTnzg9mZpYrrpjMzEpJyE15ZmaWJ27KMzMzqwBXTGZmJVTkXnmumMzMLFdcMZmZlZGKe47JicnMrKSKmpjclGdmZrniisnMrKR8HZOZmeWGgBbFzEtuyjMzs3xxxWRmVlJuyjMzs1xxrzwzM7MKcMVkZlZSRW3Kc8VkZma54orJzKyEitxd3InJzKyUinujQDflmZlZrrhiMjMrI48ubmZmeVPQvOSmPDMzyxdXTGZmJZT0yitmzeSKyczMcsUVk5lZSRWzXnJiMjMrr4JmJjflmZlZrjgxVckbr73KHgN6f/Ho3m1Vrrjsj1mHlVvHHn0Ua6+xKj17bJZ1KIXQa/MNGNBnKwbu0Itdd9wu63By5ZzTh7PHthtw8B59vjbv+sv/xHbrr8ysmTMyiKzpqYL/qsmJqUrW22BD7n14LPc+PJa7HnyCFdq2Zbe9vpN1WLl12JAjuP2u+7IOo1D+edcoHhzzDA888lTWoeTKXvscxEVX/uNr09+dOoVxY0az+hprZRBVdUiVe1STE1MGHn90NGt3W5e1vrFO1qHk1g79+tOpU6esw7AS2GrbvqzUceWvTb/43J8x/LSzijs8Qok5MWXgjltv5jv77J91GFYiQhw4eE927d+ba/96edbh5N6jo+6hy+pd2WCTzbMOpUmpgo9qapLEJKmjpOPT5wMk3dUU+ymizz//nH/ddzd7fWefrEOxErnj/tGMemwc1//zTv56+f/jyccfyzqk3Pp07idcddmFDDvpJ1mHYovRVBVTR+D4JVlBUssmiiVXHv7X/Wy2RQ+6rLpa1qFYiXRdY00AunRZlT32HsS/xz+dcUT5NeV/k5j21mQO3bsfg3fcgunvTGXIoB2ZMf3drEOrvIKWTE11HdN5wHqSJgDzgDmS/gFsBowHDo2IkPQmcCOwC/AbSTOBXwJtgDeAIyNitqSewIVAO+B94IiImNZEsTepO265yc14VlFz5swhFi6kXfv2zJkzh0ce+hcjTvtZ1mHl1vobdefeca998Xrwjltw1a2j6dhplQyjqrwknxTz/FlTVUynA29ERA/gx8BWwEnApsA3gb61lp0REVsD/wLOAHZOXz8DjJDUCvgjsG9E9ASuBM5d3I4lDZP0jKRnZs6Y3gRvbel9MmcOjz3yELvvPSjrUHLv8EMPYkC/Prz6yius120trrryiqxDyq3333uX7+w+gJ369mSPnbZn5133YKedd8s6rNz4+UlDOWa/XZk86XW+3bc7d9x0bdYhWQOqNfLDuIiYApBWUd2AMem8G9Of25EkrseV9JJpDTwJbERSaY1Kp7cEFlstRcRIYCTAFj16RoXfxzJpu+KKPPfa21mHUQjXXHdD1iEUxjrrfpOHHh+fdRi59auL6/9Sc9sjz1cpkirz/Zga9Fmt5wvq7HdO+lPAqIg4qPaKkjYHJkbE16+OMzOzxSpoXmqypryPgfZLuM5TQF9J6wNIWlHShsArQBdJfdLprSR1r2i0ZmaWG01SMUXEDEmPS3oRmAs02N0lIqZLOgK4QVKbdPIZEfGqpH2BP0jqkMZ8MTCxKWI3MyuNgpZMTdaUFxEHL2b68FrPu9WZ9xCwzSLWmQD0r3CIZmYlVv0x7irFIz+YmVmu+H5MZmYlVdReea6YzMwsV1wxmZmVUBaDr1aKE5OZWVkVNDO5Kc/MzHLFFZOZWUkVtbu4E5OZWUm5V56ZmVkFuGIyMyupghZMrpjMzCxfXDGZmZVRgS9kcmIyMyupovbKc1OemZlVhKQrJb2X3vKoZlonSaMkvZb+XLmh7TgxmZmVkEi6i1fq0UhXAbvXmXY68GBEbAA8mL6ulxOTmVlJqYKPxoiIR4GZdSYPAq5On18NDG5oO05MZmbWlFaLiGnp83eA1RpawZ0fzMzKqrJ9HzpLeqbW65ERMXJJNhARISkaWs6JyczMGuP9iOi1FOu9K6lrREyT1BV4r6EV3JRnZlZSquC/ZXAHMCR9PgS4vaEVXDGZmZVUtQdxlXQDMICk2W8KcCZwHnCTpKHAZGD/hrbjxGRmZhUREQctZtbAJdmOE5OZWUkVc9wHJyYzs/IqaGZy5wczM8sVV0xmZiWUjNhQzJLJicnMrIyWbIy7XHFTnpmZ5YorJjOzkipoweSKyczM8sUVk5lZWRW0ZHJiMjMrpWUe4y4zpU5MLzz37PvrdF5hctZx1NEZeD/rIArEx6vxfKwaL6/Hap2sA8iDUiemiOiSdQx1SXpmKYeOb5Z8vBrPx6rxmsuxKmp38VInJjOz5mpJbomeN+6VZ2ZmueKKqfqW6FbE5uO1BHysGq95HKuClkyumKosIprHf4gK8fFqPB+rxvOxyjdXTGZmJeXu4mZmlitF7ZXnpjwzM8sVV0wZkqSIiKzjMLNyKmjB5MSUseWBuTUvnKgWT1IXYF5EzMo6ljzy387SW9yxk9QiIhZmEVNFFPh+TE5MGZE0FNhJ0v+AcRFxa0SEP2C+TtKJwG7AB5LeioifZB1T3tT8zUjaF9gYeBkYHxFvZhlX3tX+/ybpCGAFks/F/xcR87OMrTnzOaYMSNofOJnkWorPgB0lHQ9ffsBYQtKBwCDgUGAOsHm2EeWXpGOBM4HZwBDgaEnbZBtVvtVKSicBhwNvAccBw7KMq3JUwUf1ODFVmaTVgbWB30bEI8BFwGNAD0krZRpcPs0FTiP50OgGfBdA0lYZxpQ7kloC2wNHRMTFwKnprL7ZRVUMkpYHukfETkB3YBLwZ0lts42s+XJiqqL0G+0wkm/+R0taNyI+BO4EvgmsnmV8OdUe+BewS0TsFhHzJB0NDJW0Qsax5YKkbYG2JKNlHypp+Yj4D8lx21vSipkGmDPS1868tABWlXQLsC3wvYhYABwoaUC146sUkZxjqtSjmnyOqUokfZukGep3ETEpPZn/E0mXkAx1vwLgE/uApCNJjsmDEXGdpI1IPmA3AXYFhgIHRcTc+rbTHEjqBBwG3A3cBgwmqS5HAiuRNOtZqs45pf7AVGAGyfH6C3BARHwm6XCS5vY9Mwu2Agra98GJqalJagG0Bv4vnfTr9Oc/SM6dXAx8ApwQEe9VP8J8kTQI+D4wFjgybbI7l+Rc3MnAiiQfHi9nF2U+SNokIl6W9BwwKCK+n37h2U/SASTV5jERMSfbSPOjzjml7wKPAr2AM4BfAtdLui+dtn9E5O1+bs2CfK69aUnqGBGzJLUHbgcmRMSIWvNXIukG7W//SVL6CfDdiJgmaR+gP/A6MDIiPpe0nHtLgaR+wBUkX3DOAu4CxkTE2emXoY2B9yIijzfDq7o6lVJ34OKI2EXSn4BVSb7shKQNgYXA3Ih4O8OQl9mWW/WM+x5+smLbW6Njm/HVuoeVzzE1IUnHAVdIOhfYieQb2o6Szq9ZJiI+clKC9ETzOyQfqEcARMQtwMNAD2BY+oG7IKMQc0PScsALwH9Imu1OAm4FdpW0bUQsjIiXnJQSdZLSViSJaKKkU0jO7Q5Jk9KeJMn89aInpRqq4L9qclNeE5F0KEkX56OB3wOdIuJ2SbsBT0n6PCJ+nmmQOZEm8O2AycAI4FRJ70TEXyPiNknzgacLfbHjMpDUGlg/Il6StCuwGfAIcAxwLDCP5HTC9sD30ruzNstjtSi1ktI+JOcnDwN+BuweERun844BDgCeyCpO+5ITUxOQ1IOkOeAYoA/Jh8YP0i69c4FtgI7ZRZgfkr4HDCdJ4kcBHwJ/A46T1CkifhcRd2UZYw6sDVws6V2Sb/t3knSh/wB4EvhfRDwk6Q3gTSelr0s7M+xM0vlopqSLSZL4NcBzJH9/h5VuZJGC9n5wU16Fpf8B/g5sTXJi9ZCI2DU9L3IMcCIwKyImZRhmnmwEXBURE4BTSDqCrAz8kKQnXsdFdO9tViLideB5ks4yN0fEn0j+jtYEfgrcIKlrRNwfEa9kGGpupM2+tS0P7AVsmL5+CPg5yTVLc4CDI+LF6kVo9XHFVEGSfkhyxf2nJOdGWgJtJHUkOb90PHCgR3f4ipdIet/dExEvAZdJGg1cAuwREZ9mG15uXEbyzX5E2gx8DUni/j7Qm+RyA0vVVI1pR4fXI2KkpI+AcyRNjIjHSL4EnZllnE2tqN/onJgqRNK3gH1JzpUcA3Qg+TA5Arie5KT9IemHr33pYZKuuYdIepjkA7YdSVXppJRKq6bXJc0i+XD9kKQpbw1geET4eiW+6NzQLyL+kA7zdQIwVdJlEfF3Sa2AkZKOj4jR2UbbtLK4MLZSnJgq5wWSbs6fS1obWCMirk+//T8P/CMi5mUbYv6kXekvBfYBfkxyQejR7lG2aBFxp6R5wG+A+SRfdpyU+GJEh87AXpK6kvS460vyhXFXSe0j4ipJbYALJO3gHrH55MRUIXU+SJ8A1pa0C3AOyYeHk9JiRMRU4E+SriS5ts4XhNYjIu6TND59Pj3rePJA0qrAyhExStJOwEBgStqZ4XJJnwL90+GaLpN0U3NISr61utX2DkkHiFeAQyPitYzjKYSI+CTrGIrCCelrOpB8uZlCMpzV74EfSToxIv6QDm21PLCFpA7pGJXlV8y85MTURF4D7gd+7KRk1vQi4rV0aKZhwGlpM/pM4Nj0AtvfR8TlklaKiI8yDtca4MTUBNLrJA6IiM+yjsWsGandc3FmRNwo6T3gUkkzIuK65paUClowOTE1FScls+qq03Px3PTn8sDnwOOZBpcR98ozM8uBWj0XLyC5eHaoL2gvFicmMyudtOfis8nT5tpRpPqDr1aKE5OZlZLvb1ZcTkxmZiVUc2v1IvIgrmZmlitOTGZmlituyjMzKyk35Zk1EUkLJE2Q9KKkm9PbsC/ttq6StG/6/HJJm9az7ABJ2y/FPt6U1Lmx0+sss0QDsko6K71FuNnXFPXW6k5MVgRzI6JHRGxGcrHkcbVnSlqqyj8ijm7gNiQDSG5XbmZV5MRkRfMYsH5azTwm6Q7gJUktJf1W0tOSnpd0LCS3QpD0J0mvSPoXya3JSec9LKlX+nx3Sc9Kek7Sg5K6kSTAH6XVWj9JXST9M93H05L6puuuIukBSRMlXU4jRoKRdJuk8ek6w+rMuyid/qCkLum09STdl67zmKSNK3EwrcT05T2ZKvGoJp9jssJIK6M9gPvSSVsDm0XEpPTD/cOI2Ca9387jkh4AtiK5ffumwGokd8y9ss52uwB/Afqn2+qUjnd4GTA7Ii5Il/sbcFFEjEnvuXU/sAnJXVDHRMTZkvYChjbi7RyV7mMF4GlJ/4yIGcCKwDMR8SNJv0i3PRwYCRyXDlbaG7gU2GkpDqNZ7jkxWRGsIGlC+vwx4AqSJrZxtYaa2ZXklgb7pq87ABsA/YEbImIByZ1MH1rE9rcDHq3ZVkTMXEwcOwOb6suvjytJapfuY5903bslfdCI93SipO+mz7+RxjoDWAjcmE6/Drgl3cf2wM219t2mEfuwZkx4EFezpjQ3InrUnpB+QNe+oaCAH0TE/XWW27OCcbQAtqt7y3ctYTuHpAEkSa5PRHyi5Jbyyy9m8Uj3O6vuMTBrUEEzk88xWVncD3xfUisASRtKWhF4FDggPQfVFfjWItZ9iuTupuum63ZKp38MtK+13APAD2peSKpJFI8CB6fT9gBWbiDWDsAHaVLamKRiq9GC5FbgpNsck96qYZKk/dJ9SNKWDezDrLCcmKwsLic5f/SspBeBP5O0CNxKcuPGl4BrgCfrrpgO8jmMpNnsOb5sSrsT+G5N5wfgRKBX2rniJb7sHfhLksQ2kaRJ738NxHofsJykl4HzSBJjjTnAtul72Ak4O51+CDA0jW8iMKgRx8SauaJ2F1dEVHWHZmbW9Lbu2Ssee/Lpim2vXZsW4yOiV8U2WA9XTGZmlivu/GBmVlIF7fvgisnMzPLFFZOZWVkVtGRyYjIzK6mi3lrdTXlmZpYrrpjMzEqoyLdW93VMZmYlJOk+oN77fy2h9yNi9wpub7GcmMzMLFd8jsnMzHLFicnMzHLFicnMzHLFicnMzHLFicnMzHLl/wMc+Jb37hvYKAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAGlCAYAAABA7gkgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xUZfbH8c+BEIpIl5KEDgoEpcOCItiRagPsYP1ZsKzu2ta1d9e2Cq51VSwgWABFsOIqFppYQEWEAElABAEFEchwfn/cISShJOhkZnL9vn3Ny7n3PvPcM8NkzpznPveOuTsiIiLJqFyiAxAREdkVJSkREUlaSlIiIpK0lKRERCRpKUmJiEjSUpISEZGklZLoAEREJPbKV2vsnrcxZv35xh+nunufmHVYQkpSIiIh5HkbqbjfkJj199vckXVi1tkeUJISEQklAyv7R3TK/jMQEZHQUiUlIhJGBpglOoo/TElKRCSsNNwnIiJSelRJiYiElYb7REQkOWl2n4iISKlSJSUiElYhGO5TJSUiIklLlZSISBgZoTgmpSQlIhJKpuE+ERGR0qRKSkQkrDTcJyIiSUvDfSIiIqVHlZSISCjpihMiIiKlSpWUiEgY6fekREQkqWm4T0REpPSokhIRCaVwTJxQkhIRCatyZf+YVNlPsyIiElqqpEREwkhXQRcRkaQWginoZT/NiohIaKmSEhEJpXDM7iv7z0BEREJLlZSISFiF4JiUkpSISFhpuE9ERKT0qJISEQkjMw33iYhIEtNwn4iISOlRkpJdMrPKZjbJzNaZ2bg/0M8pZvZmLGNLBDN7w8yGJToOkRLbNuQXi1uCKEmFgJmdbGazzGy9mS2PfpgeFIOuTwDqAbXdffDv7cTdn3P3I2MQTyFm1tvM3MxeKbK+XXT9tBL2c4OZPVtcO3c/2t2f/p3h7mrfPaP/buvNbEM07vUFbo1+R59uZi12s324mUUK7GOxmf3XzPbdg308ZWa37Glseype+5HkpSRVxpnZZcD9wG0ECaURMAoYFIPuGwML3D0vBn2Vlh+B7mZWu8C6YcCCWO3AAqXyt+LuH7h7VXevCmRGV9fYts7dl5bGfoGPo/usDhwObARmm1nbUtqfxF30ihOxupVkj2Z9zOxbM1toZlftZHsjM3vPzD4zsy/MrG9xfSpJlWFmVh24CbjQ3V929w3uvsXdJ7n736NtKprZ/WaWG73db2YVo9t6m1m2mV1uZiujVdgZ0W03AtcBQ6Pfts8qWnGYWZPot/aU6PJwM1tkZr9Ev52fUmD9hwUe18PMZkaHEWeaWY8C26aZ2c1mNj3az5tmVmc3L8Nm4FXgxOjjywNDgeeKvFYPmNkyM/vZzGabWc/o+j7ANQWe5+cF4rjVzKYDvwLNouvOjm5/2MxeKtD/nWb2jlnsxkXMrLqZPRH9d8kxs1uizw8za2Fm70dfw1VmNja6/n/Rh38efT5Dd7cPd4+4+/fufgHwPnBDgf2PM7MV0X38z8wyo+vPBU4BrojuY1J0/VVm9n30322+mR1boK+dxhvd1srM3jKzn6IfcEN2tx/ZA3Ec7ou+N0cCRwNtgJPMrE2RZtcCL7p7B4K/2VHF9askVbZ1ByoBr+ymzT+AvwDtgXZAV4I3yjb1Cb5NpwNnASPNrKa7X09QnY2NfqN/YneBmNlewL+Bo919b6AHMHcn7WoBr0fb1gbuBV4vUgmdDJwB1AVSgb/tbt/AM8Dp0ftHAV8BuUXazCR4DWoBzwPjzKySu08p8jzbFXjMacC5wN7AkiL9XQ7sH03APQleu2Hu7sXEuieeAvKAFkAH4Ejg7Oi2m4E3gZpABvAggLsfHN3eLvp8xlJyLwM9Cyy/AbQk+HeYQzTxu/uj0ft3RfcxINr+++jjqwM3As+aWYPdxRt937xF8G9Sl+gHl5m12c1+JDl1BRa6+yJ33wyMYccRHQeqRe9XZ8e/0x0oSZVttYFVxQzHnQLc5O4r3f1Hgg+P0wps3xLdvsXdJwPrgf1+ZzxbgbZmVtndl7v7vJ206Qd85+6j3T3P3V8AvgEKfgD9190XuPtG4EWC5LJL7v4RUMvM9iNIVs/spM2z7r46us97gIoU/zyfcvd50cdsKdLfrwSv473As8BF7p5dTH8lZmb1gL7ApdEKeSVwH9GKkeDfrTGQ5u6/ufuHu+hqT+QSJHEA3P1Jd//F3TcRVFjtotX7Trn7OHfPdfet0eT4HcEH1+7i7Q9kuft/o6/zZ8BLwO8+BipR235PKn7DfenAsgLL2dF1Bd0AnGpm2cBk4KLiOlWSKttWA3W2DbftQhqFq4Al0XX5fRRJcr8CVfc0EHffQDDMdh6w3MxeN7NWJYhnW0wF38wrfkc8o4ERwCHspLI0s7+Z2dfR4aa1BN/idjeMCIX/4Hbg7p8Ciwg+Dl7cVTszm2fbJyn03FW7IhoDFQhey7XRmB8hqDYArojud0a0/zNL2O/upAM/RWMub2Z3RIfvfgayom12+ZqZ2elmNrdAvG0LtN9VvI2BbtseE33cKQQVvvwhMT8mVceCCVrbbuf+jqBOIvjyl0HwJWy0FXO8Vyfzlm0fA5uAY4Dxu2iTS/BBsK2qaUQJSuxd2ABUKbBc6IPE3acCU82sMnAL8BiFh48KxlNQI2DK74xpm9HAQuAZd/+14KGhaGK4AjgMmOfuW81sDcGHJgRDEDuz26E7M7uQoCLLjfZ/+047cc/c2fpiLCP4t62zs0rZ3VcA50TjOAh428z+5+4Lf8e+tjkW+CB6/2SCoZrDCRJUdWCXr5mZNSb49z6MYFJGxMzmbmu/q3ijz/N9dz9iFzHFcvhU/phV7t55N9tzgIYFljOi6wo6C+gD4O4fm1klgi8yK3fVqSqpMszd1xFMbhhpZseYWRUzq2BmR5vZXdFmLwDXmtk+FkxAuI5geOr3mAscbMEMnerA1ds2mFk9MxsUPcawiWDYcOtO+pgM7GvBtPmU6IH9NsBrvzMmANx9MdCL4BhcUXsTHNv5EUgxs+vYPi4O8APQpLhvdAVZMF37FuBUgmG/K8xst8OSe8LdlxMcw7nHzKqZWTkza25mvaL7H2xmGdHmawg+zLe93j8AzUr4PMqbWVMzexDoTTAcDMFrtomgWq9CcNyuoKL72Csaw4/Rfs8gqKS27WdX8b5G8H44LfrerWBmXcys9Z4+F9mJ+J4nNRNoGX0/pRIMTU8s0mYpwRcZov/GlYi+Z3ZFSaqMix5fuYxgMsSPBN9MRxDMeIPgg3QW8AXwJcEB8N913om7vwWMjfY1m8KJpVw0jlyCIaNewPk76WM1wXGIywk+AK8A+rv7qt8TU5G+P3T3nVWJUwkqtQUEQ4u/UXgob9uJyqvNbE5x+4kOrz4L3Onun7v7dwQzBEdbdOZkjJxOMHFkPsEH+3hg20SELsCnZrae4IPgEndfFN12A/B0dPhsyC767h597M/ANIKk3cXdv4xuf4bgtcqJ7v+TIo9/AmgT3cer7j4fuIeguv8B2B+YXqD9TuN1918IJoScSPDeWQHcSVCh7rCf4l4wSZxoxT+C4O/ta4JZfPPM7CYzGxhtdjlwjgWzaF8Ahhc32chiOxlJRESSQbkajb1ir2ti1t9vE8+bXcxwX6nQMSkRkbAKwVXQNdwnIiJJS5WUiEgYmZX0/KakpiQlIhJWGu4TEREpPaGupMpVqubl994n0WGUCfs3qpnoEMqU37ZEEh1CmVExpXyiQygzli7NYvWqVbG8SHGsukqYUCep8nvvQ+1j70x0GGXC9JHHJzqEMuXrnF8SHUKZ0azeXokOocw45MBuMevLCEeS0nCfiIgkrVBXUiIif1rG9istlmGqpEREJGmpkhIRCSULxTEpJSkRkZAKQ5LScJ+IiCQtVVIiIiEVhkpKSUpEJKTCkKQ03CciIklLlZSISBjpPCkREZHSpUpKRCSETOdJiYhIMgtDktJwn4iIJC1VUiIiIRWGSkpJSkQkpMKQpDTcJyIiSUuVlIhIGOk8KRERkdKlSkpEJKTCcExKSUpEJITCcjKvhvtERCRpqZISEQmpMFRSSlIiImFV9nOUhvtERCR5qZISEQkjC8dwnyopERFJWqqkRERCSpWUFHJIZj0+vOkoPr6lDyP67LfD9huHtOPtfx7O2/88nOk3H8W39w/M3/b8xQfx7f0DGT3iwHiGnDBvTp1Cu8xWtG3dkn/ddccO2zdt2sRpJ59I29YtOfjAv7AkKwuAmTNn0K1zh+DWqT0TXn0lzpEnxkfvv81xh3ZiUO/2/Pfhe3fY/uzjD3HCEV0Z2qcH550ygOXZS/O3jRh2HL0OaMQlZw2JZ8gJ8/abU+jSrg0d2+7Hff+6c4ftmzZt4szTTqJj2/04/ODuLF2SBcDSJVk0qFWVnt060bNbJ/560QVxjjz2zCxmt0RRJRUj5QxuP7kDQ+77gOVrfmXKNYfx5ue5LFj+S36b61/8PP/+WYc0p22jGvnLo95cQOXU8px+cLO4xp0IkUiEv14ygtcmv0l6RgY9u3elX/+BtG7TJr/NU/99gho1a/DV198xbuwYrr3mKkY/P4bMzLZM/2QmKSkpLF++nL90bk+//gNISQnvWzkSiXDHdZczavSr1KufzmmDDqHX4X1p1rJVfpv9Mg9g9MRpVK5chXHPPs4Dd1zHHQ89BcDp517Mbxs38tIL/03QM4ifSCTC3/96Ma+8NoW09AwO7fkXju43gFatt7+3Rj/1JNVr1GTOV9/y0rix3HDt1Tw5+gUAmjRrzgefzk5U+LITqqRipEPTWixeuZ6lqzawJeK8OnMZR7VL22X7Y7o24pUZy/KXP/xmJRt+y4tHqAk3a+YMmjdvQdNmzUhNTeWEIUN5bdKEQm1enzSRU08bBsCxx5/AtPfewd2pUqVKfkLa9NtvoRjOKM68z2fTsHEzMho1pUJqKkcOOI5pb71eqE2X7gdTuXIVAPbv0IWVK3Lzt3U9sDdVqlaNa8yJMnvWDJo1b06TpsF767gThjD5tYmF2rzx+kROOvU0AAYdezzvT3sXd09EuKVq2xUnynolpSQVIw1qVCb3p435y8vXbqRBzco7bZtRqwqN6lThw29Wxiu8pJKbk0N6Rkb+cnp6Brm5OTtp0xCAlJQUqlWvzurVqwGYMeNTOrVrS5eOB/DAQw+HuooCWLkil3oN0vOX69VP58cVy3fZfsLY0fTodUQ8Qks6y3NzSU9vmL+clp7B8tzcQm1yC7RJSUmhWrXq/BR9by3NWszBf+lMvyMP4aPpH8Qv8NJiMbyVZHdmfczsWzNbaGZX7WT7fWY2N3pbYGZri+sz3H/dSeqYLg15bXYOW8P35S0uunbtxuzPv+Kbr7/mnLOGc1Sfo6lUqVKiw0oKk18Zy/wvP+OxMZMTHUqZU69+A778djG1atdm7pzZnDL0eD6e/QXVqlVLdGhlgpmVB0YCRwDZwEwzm+ju87e1cfe/Fmh/EdChuH7LRCUVffJJbfnajaTV2l45NahRmeVrNu607aAuGYWG+v5s0tLTycnOzl/OyckmLS19J22C1ygvL4+f162jdu3ahdq0at2aqlWrMm/eV6UfdALVrZ/GD8u3V5o/rMhhn/oNdmj36Yfv8cTIf3HfY2NIrVgxniEmjQZpaeTkbP/bys3JpkFa4WH3tAJt8vLy+PnnddSqXZuKFStSK/oea9+xE02bNeP77xbEL/hYs7hPnOgKLHT3Re6+GRgDDNpN+5OAF4rrNK5JyszOK1DqLTaz98zsSDP72MzmmNk4M6sabZtlZnea2RxgsJmdZGZfmtlXZrbjlJ0Em5u1hmZ1q9KodhUqlDeO6dKQNz/fcUimRf29qVEllVmLVicgyuTQqXMXFi78jqzFi9m8eTPjXxxLv/4DC7Xp238Az45+GoBXXhpPr96HYmZkLV5MXl5w7G7pkiV8++03NG7cJN5PIa7aHNCRZVnfk7Msiy2bN/PmpJfpdXjfQm2+mfc5t/7jUu57bAy16uyToEgTr2OnLny/cCFLsoL31svjX+TofgMKtenTdwAvPDsagAmvvMTBvQ7BzFj1449EIhEAshYvYtHChTRpWrYnMsU4SdUxs1kFbucW2V06UPDbd3Z03c7iagw0Bd4t7jnEdbjP3f8D/MfMKhAE9yRwLXC4u28wsyuBy4Cbog9Z7e4dzSwN+AToBKwB3jSzY9z91aL7iL5w5wKUq1qn1J/TNpGtzjUvzOWFS3tSvpzxwvQsvl3+M1cMbMPcJWvyE9YxXRry6swdq6hX/96blvX3pkrFFObc2ZfLnp7NtPk/xC3+eEpJSeHe+x9kYL8+RLZGOH3YGbTJzOSmG66jY6fO9B8wkOFnnMVZw0+nbeuW1KxZi2eeDb5wfTT9Q+65+05SKlSgXLly3P/vkdSpE79/50RISUnhihv/xYjTjyOyNcKgwafSfN/WPHzvrbTZvwO9jujLA7f/k40bNnDlhcFkk/ppGdz3+BgAzhrch6xFC9i4YQNHd2/NP+94kB69Dk/kUyo1KSkp3HXvAxw/sC+RSIRTTh9O6zaZ3HbT9bTv2Jm+/Qdw2vAzOe+sYXRsux81a9bkiWeeB+Cj6R9w+803kJISvLfu+fdIataqleBnlFRWuXvnGPV1IjDe3SPFNbREzGoxs1HAj8BM4CmCjAuQCnzs7meZWRbQy92XmNkg4Hh3Pz36+LOATHe/bHf7qbBPc699bNIVXUlp8cjjEx1CmfJ1zi/FNxIAmtXbK9EhlBmHHNiNz+bMislUutS6Lbze4Hti0RUA2aOOmb27JGVm3YEb3P2o6PLVAO5++07afgZc6O4fFbffuE+cMLPhQGNgBNAPeMvdT9pF8w3xiktERP6QmUBLM2sK5BBUSycXbWRmrYCawMcl6TTex6Q6AX8DTnX3rQRDeAeaWYvo9r3MbN+dPHQG0MvM6kQnUZwEvB+vuEVEyqQ4TkF39zyC4mMq8DXworvPM7ObzKzgQecTgTFewmG8eFdSI4BawHvRA3GzgOHAC2a2bTrStUChKTXuvjw65/49gpfrdXcvfPaniIgUEu+TcN19MjC5yLrriizfsCd9xnvixBm72NRlJ22bFFl+gRJMVxQRkfDQybwiIiGU6MsZxYqSlIhISIUhSZWJK06IiMifkyopEZGQUiUlIiJSilRJiYiEVdkvpJSkRETCSsN9IiIipUiVlIhIGFk4KiklKRGREDIgBDlKw30iIpK8VEmJiIRSOC6LpEpKRESSliopEZGQCkEhpSQlIhJWGu4TEREpRaqkRETCyDTcJyIiScqAcuXKfpbScJ+IiCQtVVIiIiGl4T4REUlamt0nIiJSilRJiYiEUUhm96mSEhGRpKVKSkQkhIKf6ij7pZSSlIhIKOkq6CIiIqVKlZSISEiFoJBSkhIRCSsN94mIiJQiVVIiImEUkvOkQp2kDmhUk+mjTkh0GGVCzS4jEh1CmbJm5kOJDqHM+HVTXqJDKDMcT3QISSfUSUpE5M9K50mJiEhSC0GO0sQJERFJXqqkRERCKgzDfaqkRERCyix2t5Ltz/qY2bdmttDMrtpFmyFmNt/M5pnZ88X1qUpKRET+MDMrD4wEjgCygZlmNtHd5xdo0xK4GjjQ3deYWd3i+lWSEhEJI4v7cF9XYKG7LwIwszHAIGB+gTbnACPdfQ2Au68srlMN94mISCykA8sKLGdH1xW0L7CvmU03s0/MrE9xnaqSEhEJoeA8qZh2WcfMZhVYftTdH93DPlKAlkBvIAP4n5nt7+5rd/cAEREJnZj/ntQqd++8m+05QMMCyxnRdQVlA5+6+xZgsZktIEhaM3fVqYb7REQkFmYCLc2sqZmlAicCE4u0eZWgisLM6hAM/y3aXaeqpEREQiqe8ybcPc/MRgBTgfLAk+4+z8xuAma5+8TotiPNbD4QAf7u7qt316+SlIhISMX7ZF53nwxMLrLuugL3HbgseisRDfeJiEjSUiUlIhJG+j0pERFJVmH5qQ4N94mISNJSJSUiElKqpEREREqRKikRkZAKQSGlJCUiElYa7hMRESlFqqRERMJI50mJiEiysthfBT0hNNwnIiJJS5WUiEhIhaCQUiUlIiLJS5WUiEhIlQtBKaUkJSISUiHIURruExGR5KUkFUNvTp3CAZn7kdmqBXffdccO2zdt2sSpJw8ls1ULevboxpKsLADeefstenTtROf2+9OjayemvfdunCOPvyN6tObzV/7JVxOu529nHLHD9ob1azLl0Yv5+IUrmTH2ao46qA0AFVLK88gNpzLzxWv4dOxV9OzUMt6hJ4TeWyX3zltT6dohk84HtOL+e+7aYfumTZs46/ST6XxAK47o3YOlS7IAGDf2eXp175R/q7N3Kl9+MTfO0ceOWXDFiVjdEkVJKkYikQiXXnwhEya9wWdfzGfcmBf4ev78Qm2eevIJataoybxvFnLRJX/lH9dcCUDt2nUY/+okZs39kseefJozh5+WiKcQN+XKGfdfNYRBI0bR4fhbGNynE62a1S/U5sqz+/DSW3PoftKdnH71f3ng6qEAnHncgQB0GXIb/c97iDsuOzYU54Lsjt5bJReJRLjisot58eVJfDTrC14eN4Zvvi78Wj379JPUqFGDWV98w/kXXsKN/7wGgMFDT+b9j2fz/sezefixp2jcpCn7H9A+EU8jZspZ7G4Jew6J23W4zJwxg+bNW9C0WTNSU1MZPPREXps0oVCb1yZN4JTThgFw3PEnMO3dd3B32nfoQFpaGgBtMjP5beNGNm3aFPfnEC9d2jbh+2WryMpZzZa8COOmzqF/7wMKtXF3qu1VCYDqVSuz/Md1ALRqVp9pM78F4Mc161n3y0Y6tWkU3ycQZ3pvldycWTNo2qw5TZoGr9WxJwzljdcnFWrzxuuTOPGUIFkPPPZ4/jftXdy9UJuXxo/l2OOHxC1u2TUlqRjJzc0hI6Nh/nJ6egY5OTk7tmkYtElJSaFa9eqsXr26UJtXXn6J9h06UrFixdIPOkHS6lYn+4c1+cs5P6whfZ/qhdrc+shkTuzblYVTbuaVB8/nsjvHAfDlghz699qf8uXL0TitNh3aNCSjfs24xh9vem+V3PLcXNIzMvKX09LTWZ6bs0ObtIzCr9VPRV6rV18ax/GDh5Z+wKUsDMN9mt2XRObPm8e111zJa5PfTHQoCTekT2eenfQJD4x+l24HNOWJW06n0wm38fSEj2nVtB7Tn7uCpct/4pPPFxOJbE10uElP762SmzXzUypXrkzrzLaJDkVQJRUzaWnpZGcvy1/OyckmPT19xzbLgjZ5eXn8vG4dtWvXBiA7O5uhg4/l8SefoVnz5vELPAFyV64jo9726ie9Xk1yosN52ww7pjsvvTkHgE+/WEyl1ArUqbEXkchWrrjnZf5y4h0M+euj1Ni7Mt8tXRnX+ONN762Sa5CWRk52dv5ybk4ODdLSd2iTm134taoVfa0AXhn/IscNPjE+AZcys9jdEiWhScrMLjOzr6K3S82siZl9bWaPmdk8M3vTzCpH2zY3sylmNtvMPjCzVomMvajOXbqwcOF3ZC1ezObNmxk3dgz9+g8s1KZf/4E8N/ppAF5+aTy9DjkUM2Pt2rUcN7AfN996Bz0OPDAR4cfVrHlLaNFoHxqn1aZCSnkGH9WR16d9UajNshU/0bvrfgDs17QelSpW4Mc166lcqQJVKqUCcGi3VuRFtvLNohVxfw7xpPdWyXXo1IVF3y9kSVbwWr0yfixH9+1fqE2fvv0Z89xoACa+8hI9ex2SP5y1detWXn15PMedUPaPRxnRi8zG6L9ESdhwn5l1As4AuhG8np8C7wMtgZPc/RwzexE4HngWeBQ4z92/M7NuwCjg0IQEvxMpKSnc98BDDOh3FJFIhGHDz6RNZiY33XAdHTt1pv+AgQw/8yzOHH4ama1aULNmLUY/NwaA/4x6iO+/X8jtt9zE7bfcBMCkN96kbt26iXxKpSYS2cpf73yRSaMupHw54+kJn/D1ohX88/x+zJm/lNff/5Kr7n2FUf88iYtOPQR3OOe64ENln5p7M2nUhWzd6uT+uJazrn06wc+m9Om9VXIpKSncec8DDD6mH5FIhJNPG06rNpncfvMNtO/YiaP7DeDUYWdy/tnD6XxAK2rUrMnjTz2X//iPPvyA9IwMmjRtlsBnIQVZ0Vktcdux2SVAbXe/Lrp8M/AjcJG7t4yuuxKoANwf3fZtgS4qunvrnfR7LnAuQMNGjTot+H5JqT6PsKjZZUSiQyhT1sx8KNEhlBm/bspLdAhlxqE9uzF3zuyYlC01Grf2g//xTCy6AmDS/3Wd7e6dY9ZhCSXjxImC82MjQGWCYcm17l7sSQvu/ihB1UWnTp0Tk4FFRBItwbPyYiWRx6Q+AI4xsypmthdwbHTdDtz9Z2CxmQ0GsEC7+IUqIiKJkLAk5e5zgKeAGQTHox4H1uzmIacAZ5nZ58A8YFBpxygiUpaFYXZfQof73P1e4N4iq9sW2P6vAvcXA33iFJqIiCSBZDwmJSIif5Ch35MSEZEkFoIcpStOiIhI8lIlJSISUmGYgq4kJSISQomelRcrGu4TEZGkpUpKRCSkNLtPRESSVtlPURruExGRGDGzPmb2rZktNLOrdrJ9uJn9aGZzo7ezi+tTlZSISEjFc3afmZUHRgJHANnATDOb6O7zizQd6+4l/tmFXSYpM3sQ2OVVxN394pLuREREQq8rsNDdFwGY2RiCa6wWTVJ7ZHeV1Kw/0rGIiCROcFmkuO4yHVhWYDmb4EdtizrezA4GFgB/dfdlO2mTb5dJyt0L/eSpmVVx919LHq+IiCRM7H9Pqo6ZFSxeHo3+ft+emAS84O6bzOz/gKcp5hfWi504YWbdzWw+8E10uZ2ZjdrDwEREpGxb5e6dC9yKJqgcoGGB5Yzounzuvtrdt/2w7eNAp+J2WpLZffcDRwGrozv5HDi4BI8TEZEEivPvSc0EWppZUzNLBU4EJhaOxxoUWBwIfF1cpyWa3efuy4qUjZGSPE5ERBInnrP73D3PzEYAU4HywJPuPs/MbgJmuftE4GIzGwjkAT8Bw4vrtyRJapmZ9QDczCoAl1CC7CciIn8u7j4ZmFxk3XUF7l8NXJicvOkAACAASURBVL0nfZYkSZ0HPEAwcyOXIEteuCc7ERGR+ErA7L5SUWyScvdVwClxiEVERKSQkszua2Zmk6KXslhpZhPMrFk8ghMRkd/PotPQY3FLlJLM7nseeBFoAKQB44AXSjMoERH54yyGt0QpSZKq4u6j3T0vensWqFTagYmIiOzu2n21onffiF7NdgzBtfyGUmT2hoiIJBez8P+e1GyCpLTtWf5fgW3OHk4jFBGR+ApBjtrttfuaxjMQERGRokp0xQkzawu0ocCxKHd/prSCEhGRPy6Rs/JipdgkZWbXA70JktRk4GjgQ0BJSkRESlVJZvedABwGrHD3M4B2QPVSjUpERP6wOF9gtlSUZLhvo7tvNbM8M6sGrKTw5dhFRCTJGBb62X3bzDKzGsBjBDP+1gMfl2pUIiIilOzafRdE7/7HzKYA1dz9i9INS0RE/pAED9PFyu5O5u24u23uPqd0QhIRkVgI++y+e3azzSnmd+mTwbrftjB1/opEh1EmrJn5UKJDKFMGPfJJokMoM/4ztH2iQygzIlsTHUHy2d3JvIfEMxAREYmtkkzfTnYlOplXRETKFiMcw31hSLQiIhJSqqREREIqDD8fX5Jf5jUzO9XMrosuNzKzrqUfmoiI/NmVZLhvFNAdOCm6/AswstQiEhGRmChnsbslSkmG+7q5e0cz+wzA3deYWWopxyUiIn9AcM29sj/eV5JKaouZlSc4Nwoz2wfQbH4RESl1Jamk/g28AtQ1s1sJrop+balGJSIif1gYJk6U5Np9z5nZbIKf6zDgGHf/utQjExGRPyQEo30l+tHDRsCvwKSC69x9aWkGJiIiUpLhvtcJjkcZwc/HNwW+BTJLMS4REfkDDP4cvyfl7vsXXI5eHf2CXTQXERGJmT2+4oS7zzGzbqURjIiIxE4YrntXkmNSlxVYLAd0BHJLLSIREYmJEIz2laiS2rvA/TyCY1QvlU44IiIi2+02SUVP4t3b3f8Wp3hERCQGzCzcEyfMLMXd88zswHgGJCIisRGCHLXbSmoGwfGnuWY2ERgHbNi20d1fLuXYRETkT64kx6QqAauBQ9l+vpQDSlIiIkks7JdFqhud2fcV25PTNl6qUYmIiLD7JFUeqErh5LSNkpSISBL7M1xxYrm73xS3SEREJKbinaPMrA/wAEGR87i737GLdscD44Eu7j5rd33u7oTksp+CRUQkLqKnLI0EjgbaACeZWZudtNsbuAT4tCT97i5JHfY74hQRkWQQw5+OL+EEjK7AQndf5O6bgTHAoJ20uxm4E/itJJ3uMkm5+08lCktERJKSxfC/EkgHlhVYzo6u2x5PcIHyhu7+ekmfwx5fYFZERP6U6phZweNHj7r7oyV9sJmVA+4Fhu/JTpWkRERCKJjdF9MuV7l7591szwEaFljOiK7bZm+gLTDNghkd9YGJZjZwd5MnlKREREIqzifzzgRamllTguR0InDyto3uvg6os23ZzKYBf/sjs/tkD82Z/i4XDDyI8/p356UnHtxh+5QXn+bi4w/h0iGHc/WwgSz7/tv8beOf+Dfn9e/OBQMP4rPp78Uz7IR4c+oUDsjcj8xWLbj7rh1nqW7atIlTTx5KZqsW9OzRjSVZWQC88/Zb9Ojaic7t96dH105Me+/dOEeeGJ0bVefxk9vx31PbM6Rj2g7bj2i1D2PP7MSoofszauj+9Gm9T/62yed3y19/Q9994xl2Qrz/zpsc9pcDOKRLJg8/cPcO22d89CEDDu1Oy/pVmTxx+4VzcpYtYcCh3enXuxtHHdSR5556LJ5hl3nungeMAKYCXwMvuvs8M7vJzAb+3n5VScVIJBLhkduu4cZHxlK7XgP+fvLRdO19JA2b75ff5uC+x9FnyDAAZkybypP/uoHrH36BZd9/y4dTJvDgy9P4aeUPXPd/Qxg1cTrly5dP1NMpVZFIhEsvvpDX33iL9IwMDvpLF/r3H0jrNttnqz715BPUrFGTed8s5MWxY/jHNVfy7PNjqV27DuNfnURaWhrzvvqKAf2OYtGSnN3srewrZ3DhwU25euLXrFq/mQcHt+WTxWtYumZjoXb/+241Iz/I2uHxmyNbuWDsl3GKNrEikQjXX3Upz4x7nfpp6Rxz5EEc3qc/Lfdrnd8mLaMhdz34KI+Pur/QY/ep14Dxb0yjYsWKbFi/nj4Hd+LwPv2oV3/HLwVlhcX5RCl3nwxMLrLuul207V2SPlVJxch3X31Gg4ZNqJ/RmAoVUjmozyA+nTa1UJsqVbf/NNdvG3/NfwN9Om0qB/UZRIXUitTLaESDhk347qvP4hp/PM2cMYPmzVvQtFkzUlNTGTz0RF6bNKFQm9cmTeCU04KEftzxJzDt3Xdwd9p36EBaWvCh0SYzk982bmTTpk1xfw7xtF/dquSu+40VP28ib6sz7bvVdG9aM9FhJaXP58ykcZPmNGrSlNTUVPofM5i33nitUJuMRo1pnbk/5azwx19qaioVK1YEYPPmTWzdujVuccuuqZKKkZ9WrqBO/e2zLWvXbcB3X+6YaCaP+S8TRj9C3pYt3PzYuOCxP6xg3wM6bn9svTR+Wrmi9INOkNzcHDIyth9fTU/PYMaMT3ds0zBok5KSQrXq1Vm9ejV16uQPafPKyy/RvkPH/A+WsKpdNZUf12/OX161fjOt6lXdod2BzWvRNm1vctb+xiPTl+Q/JrV8OR4c3JbIVmfsnFw+XrwmbrHH24rluTRIz8hfbpCWztzZM0r8+NycZZx18nEsWfw9V11/W9muogjHBWYTVkmZWQ0zuyBR+0+UvieewSOvf8Lpl/6DcY/dX/wDZKfmz5vHtddcyUOjHkl0KEnhk8VrGPbMZ5w/9kvmZK/jb4c1z9922jNzuGjcV9zx1kLOO6gJDaqFO6n/EWnpDXnj/Zm8N+MrXh77LD+u/CHRIf1+FlwWKVa3REnkcF8NIDRJqlbd+qxasf3YyOqVy6lVr/4u2/fscwyfvjcleGy9+qz6IXf7Y3/IpVbdXT+2rEtLSyc7e/s5fzk52aSnp+/YZlnQJi8vj5/XraN27doAZGdnM3TwsTz+5DM0a96csFu9fjP7VE3NX65TNZVVGzYXavPLpjy2bA2u+zxl/kpa7rPX9sdv2ALAip838UXOzzQvsC1s6jdIY3lOdv7y8twc6jVI380jdq5e/TT2bZXJzE+mxzI8+R0SmaTuAJqb2Vwz+++22R9m9oqZPRm9f6aZ3Rq9f5mZfRW9XZrAuHeqZWZ7li9dzA/ZS9myZTMfTplA115HFWqTu2RR/v1Z/3ubBo2aAtC111F8OGUCWzZv4ofspSxfupiWbTvENf546tylCwsXfkfW4sVs3ryZcWPH0K9/4ck//foP5LnRTwPw8kvj6XXIoZgZa9eu5biB/bj51jvoceCf40ejv125nvTqlai3d0VSyhm9W9bmk6zCQ3a1qlTIv/+XJjXzJ1VUrVieCtExn2qVUshsUJWlPxWecBEmB3ToTNbihSxbksXmzZt57dVxHN6nX4keuzw3m982Bq/NurVrmPXpRzRrUbZnQ5aL/oR8LG6JkshjUlcBbd29vZmdCPQEJhJcRqNBtE1PYIyZdQLOALoRDLV+ambvu3vSzC4on5LCOVffxo3nn0Rka4TDjzmRRi324/mRd9Eisx1dex/F5DFP8vknH1C+QgWq7l2dS27+NwCNWuzHgUcOYMSxvShfPoVzr7kttDP7IDjGdN8DDzGg31FEIhGGDT+TNpmZ3HTDdXTs1Jn+AwYy/MyzOHP4aWS2akHNmrUY/dwYAP4z6iG+/34ht99yE7ffElykf9Ibb1K3bt1EPqVStdVh5AdZ3DawFeXMePPrlSz5aSOnd81gwcoNfJK1hkEH1Kd705pEtjq//JbHPe98D0CjmpW5uHcz3B0zY+yc3B1mBYZJSkoKN9x+H8OGDGDr1giDTxrGvq3acN8dN7F/+44c3qc/n382i/OHDWXdurW88+ZkHrjrFqZ+OIeFC77ltuuvwsxwd8658FJatWmb6Kf0u4XlmJS5J+anocysCfCau7c1s3TgJeBM4AqgJnAe8B7QJbq+9rapjGZ2M/Cju/97J/2eC5wLsE+D9E6PTdnteWISdVSb8A4vloZBj3yS6BDKjP8MbZ/oEMqMgYcfyJdzZ8cktTRqtb//7fGJsegKgEt6NptdzBUnSkVSzO5z9xwzqwH0Af4H1AKGAOvd/Zc9mesfvZbUowAtMtvpxxlF5E8rBL95mNBjUr8QXMtpm0+ASwmS1AfA36L/J/r/Y8ysipntBRxbYJuIiIRUwiopd19tZtPN7CvgDYKkc6S7LzSzJQTV1AfRtnPM7Clg2wkPjyfT8SgRkeRjlAvBb9cmdLjP3U8usuqJ6PotwF5F2t5LcJl3EREphqHhPhERkVKVFBMnREQkxkr+s+9JTUlKRCSkEnkSbqxouE9ERJKWKikRkRDSxAkREZFSpkpKRCSkwnBMSklKRCSkQpCjNNwnIiLJS5WUiEgIGeGoQpSkRETCyGBPfkEiWYUh0YqISEipkhIRCamyX0epkhIRkSSmSkpEJIQMnSclIiJJrOynKA33iYhIElMlJSISUiEY7VOSEhEJJ9N5UiIiIqVJlZSISAjpskgiIpLUNNwnIiJSilRJiYiEVNmvo1RJiYhIElMlJSISRiH5qQ4lKRGREArL7L4wPAcREUkCZtbHzL41s4VmdtVOtp9nZl+a2Vwz+9DM2hTXp5KUiEhImVnMbiXYV3lgJHA00AY4aSdJ6Hl339/d2wN3AfcW16+SlIhISFkMbyXQFVjo7ovcfTMwBhhUsIG7/1xgcS/Ai+tUx6RERKQk6pjZrALLj7r7owWW04FlBZazgW5FOzGzC4HLgFTg0OJ2qiQlIhJSMZ7ct8rdO//RTtx9JDDSzE4GrgWG7a59qJNUeTOqpVZIdBhlQl5ka6JDKFPuHpiZ6BDKjL9PnJfoEMqM7LUbEx3CH5EDNCywnBFdtytjgIeL61THpEREQiiYgm4xu5XATKClmTU1s1TgRGBioZjMWhZY7Ad8V1ynoa6kRET+zOJ5Lq+755nZCGAqUB540t3nmdlNwCx3nwiMMLPDgS3AGooZ6gMlKRERiRF3nwxMLrLuugL3L9nTPpWkRERCybAQXGJWSUpEJKRCcOk+TZwQEZHkpUpKRCSEts3uK+tUSYmISNJSJSUiEkYWjmNSSlIiIiEVhiSl4T4REUlaqqREREJK50mJiEhSMqBc2c9RGu4TEZHkpUpKRCSkNNwnIiJJS7P7RERESpEqKRGRkArDcJ8qKRERSVqqpEREQigsU9CVpEREQikcP3qo4T4REUlaqqRERMJIV0EXEZFkFoIcpeE+ERFJXqqkRERCKJjdV/ZrKVVSIiKStFRJiYiEVNmvo5SkRETCKwRZSsN9IiKStJSkYmjGB+9w+tHdOPWoLjz/2AM7bB/31CjO6N+DswcdzOVnHMuKnGX5237IzebvZ53A8H7dOaN/D1bkLI1n6HH31ptT6LB/a9q12Zd77r5zh+2bNm1i2Kkn0q7NvhzSsztLsrIKbV+2dCn1a1fjgfvuiVPEiTV92lsM7N2R/j3b8cTIe3fYPvvT6Qzt25OOTWvy1uuvFtp2323Xcdzh3Tju8G5MmfhSvEJOmPbp1Xjg+EwePCGTYw6ot8P23i1q88RJB3D3oNbcPag1h+1bG4AmtSpza//9uO/YNtxzTGt6NK0Z79BjzmL4X6JouC9GIpEID9x8JXc/MZ596qVx/pAj6HFIH5q02C+/TYvW+/PwuLepVLkKE154kkf/dQPX3fcEAHdcdQGn/N9ldD6wNxs3rMfKhff7QyQS4fJLLmLC61NJz8ig14Hd6Nd/AK1at8lv88xTT1KjRk0+n7+A8S+O4bprr+LpZ8fkb7/6yss54qg+iQg/7iKRCLddezmPPDeBeg3SOXlAb3of0Zfm+7bKb1M/LYOb73mYpx/5d6HH/u+dKXzz1ee8OGU6mzdv4uwhfTnokCOoune1eD+NuChncHb3Rtw0dQE/bdjCHQNbMWvpOrLX/lao3UeL1/DEJ8sKrduUt5UH/5fFip83UbNyBe4a1Jq5OT/z6+ZIPJ9CTIVgcp8qqVj55os5pDdqSlrDJlRITeXQvsfy0btvFGrToVtPKlWuAkCbdp358YflAGQt/JZIJELnA3sDUHmvqvntwmjWzBk0a96cps2akZqayvGDh/LapImF2rw+aQInn3o6AMccdwLT3nsXdwdg0sRXadykKa1bZ8Y99kT4au4sGjZpRkbjplRITaXPgOOZ9ubrhdqkN2zMvq3bUq7Il5tF331Lx249SElJoUqVvWjZui3Tp70dz/DjqkWdvVjx82+s/GUzeVud6YvW0KVRjRI9dvnPm1jx8yYA1mzcwrrftlCtkr7HJ5qSVIysWrmcuvXT8pfr1EvLT0I7M/ml5+ja8zAAsrO+p+re1bjuomGce9wh/Ofu64lEyu63t+Isz80hPaNh/nJ6ejrLc3MKtcnNzSUj2iYlJYXq1aqzevVq1q9fz3333M3V/7gurjEn0soVy6mflpG/XLdBGj/8kFuix+7bpi0fTXubjRt/Zc1Pq5n50QesWJ5T/APLqFp7VWDVhi35y6s3bKZWlQo7tPtLk5rcc0xrLj+kGbX32nF7izpVSCln/BBNWmWVxfCWKKWSpMyshpldEL3f28xeK439lFVvTXyRBV/NZehZIwCIRPL4cvYnnHfFjTz84lssX7aEqa+8kOAok9Ntt9zIiIsuoWrVqokOpUzocfBhHHTokQw79giuGnEm7Tp1oXyIh5JLYtaytZz/4pdc/urXfJH7MyN6Nim0vUblFC7q1ZSRHyzBExOiFFBatWwN4AJgVEkfYGbl3b3Mlg916jZg5Yrt325X/ZDLPvUa7NBu9kfv89wj93HfMxNJTa0IwD710mjeqi1pDZsAcOBhffn681lxiTsRGqSlk5O9/XhATk4ODdLSC7VJS0sjO3sZ6RkZ5OXlse7nddSuXZtZM2Yw4eWX+Oc1V7Fu3VrKlStHpUqV+L/zL4z304ibuvUbsCI3O3955fJc6tVL280jCjvnor9zzkV/B+Cqi86kcbMWMY8xWfy0YQt1ClRGtfdK5adftxRqs37T9o+Zdxas4tQu26vUyhXKcc0RLXlhdg7f/bih9AMubTomtUt3AM3NbC5wN1DVzMab2Tdm9pxZcDjPzLLM7E4zmwMMNrMjzexjM5tjZuPMrGq0XScze9/MZpvZVDPb8dM/wVrt34GcJYtYnr2ELZs38+7kV+h+SOED+9/N/4J7b7icW0Y+S83a++Sv32//Dqz/5WfW/rQKgM8+/YDGzfcjrDp17sL3CxeStXgxmzdv5qVxY+nXf0ChNn37D+T5Z58B4NWXx9Or9yGYGW+++z7zFixi3oJFXDDiEi6/4upQJyiAzHadWLp4EdlLs9iyeTNTJr1EryP6luixkUiEtWtWA7Dg669Y8PU8uh98WGmGm1ALV22gQfVK1K2aSko548BmNZm5dG2hNjUqb/9u3rlRDXLWbgQgpZxxxWHNeX/haj7JKvyYsigYptPsvl25Cmjr7u3NrDcwAcgEcoHpwIHAh9G2q929o5nVAV4GDnf3DWZ2JXCZmd0OPAgMcvcfzWwocCtw5s52bGbnAucC1Cswjl/ayqekcNG1d3Dl2YOJbN3K0cedTNOWrfjvv29n37btOfDQo3nk7hv47dcN3PjXswCo2yCdW0c9R/ny5Tnv7zfytzOOw93ZN7Md/QafFrfY4y0lJYV/3f9vjhlwNFsjEU4bdgat22Ryy43X06FTJ/r1H8jpw8/knDNPp12bfalZqxb/feb5RIedMCkpKVx9892cf9qxbI1EOGboabTYrzUj77mFzP070vvIvnz1+Wz+es4p/LxuLe+//Qaj7r2NV96ZQd6WLZxxfPBlaa+99+a2Bx4jJSW8kwG2Ojz+8VKuPaol5cx497tVZK/9jaEdGvD9ql+ZtWwdfdvUpUujGkTcWb8pwkMfZAHQvWlNWtffm6oVU+jdMpiWPvKDLLJ+2pjAZyS2bcZUTDs1awK85u5to0nqH+5+RHTbw8B0d3/WzLKAXu6+xMz6A08B28Y1UoGPgfuAj4BF0fXlgeXufmRxcezXtr3/Z/w7sXpaodYlBOeExNOilSEYCoqTm95akOgQyox3bjqNNVnzY1K2tDmgg4+e+H4sugKgc9Pqs929c8w6LKF4faUqOEUmUmS/2/7aDXjL3U8q+EAz2x+Y5+7dSzdEEZFwCcEhqVI7JvULsPcePuYT4EAzawFgZnuZ2b7At8A+ZtY9ur6Cmf05TpARESlDzKyPmX1rZgvN7KqdbL/MzOab2Rdm9o6ZNS6uz1JJUu6+GphuZl8RTJwoyWN+BIYDL5jZFwRDfa3cfTNwAnCnmX0OzAV6lEbcIiKhEscTpcysPDASOBpoA5xkZm2KNPsM6OzuBwDjgbuK67fUhvvc/eRdrB9R4H6TItveBbrs5DFzgYNjHKKISIjFfVZeV2Chuy8CMLMxwCBg/rYG7v5egfafAKcW1+mf+6w+ERGJlXSg4AURs6PrduUs4I3dbAd0gVkRkdCK8QVm65hZwasMPOruj/6ejszsVKAz0Ku4tkpSIiJSEquKmYKeAzQssJwRXVeImR0O/IPg9KNiL46o4T4RkRCK5ZyJEhZkM4GWZtbUzFKBE4FCP29gZh2AR4CB7r6yJJ2qkhIRCas4zptw9zwzGwFMJbjowpPuPs/MbgJmuftEopfJA8ZFr4631N0H7q5fJSkREYkJd58MTC6y7roC9w/f0z6VpEREQiqRF4aNFSUpEZGQ0s/Hi4iIlCJVUiIiIRWCQkqVlIiIJC9VUiIiYbQHJzglMyUpEZGQCsPsPg33iYhI0lIlJSISQkY4pqArSYmIhFQIcpSG+0REJHmpkhIRCasQlFKqpEREJGmpkhIRCakwTEFXkhIRCakwzO7TcJ+IiCQtVVIiIiEVgkJKSUpEJLRCkKU03CciIklLlZSISAgFF0Ev+6WUkpSISBiZZveJiIiUKlVSIiIhFYJCSpWUiIgkL1VSIiJhFYJSSklKRCSUTLP7kt2CeZ+vOrR1nSWJjqOIOsCqRAdRhuj1Kjm9ViWXrK9V40QHkGxCnaTcfZ9Ex1CUmc1y986JjqOs0OtVcnqtSu7P8lqFYQp6qJOUiMiflRGKQ1Ka3SciIslLlVT8PZroAMoYvV4lp9eq5P4cr1UISilVUnHm7n+OP44Y0etVcnqtSk6vVdmhSkpEJKQ0BV1ERJJWGGb3abhPRESSliqpBDIzc3dPdBwiEk4hKKSUpBKsErBx24KS1q6Z2T7AFndfm+hYkpHeO7/frl47Myvn7lsTEVNMhOT3pJSkEsTMzgIONbOlwAx3f8XdXR82OzKzi4GjgDVmtszdr050TMlm23vGzE4AWgFfA7PdPSuRcSW7gn9vZjYcqEzwufiwu+clMjYJ6JhUApjZEOBygnM1NgG9zOwC2P5hIwEzOxEYBJwKbAD2T2xEycvM/g+4HlgPDAPONrMuiY0quRVIUJcCpwPLgPOAcxMZV+xYDG8l2JtZHzP71swWmtlVO9l+sJnNMbO86BeqYilJxZmZ1QcaAXe7+/vAfcAHQHszq5bQ4JLTRuBKgg+QJsCxAGbWIYExJR0zKw/0AIa7+/3AFdFNByYuqrLBzCoBme5+KJAJLAYeMbMqiY2sbIm+B0cCRwNtgJPMrE2RZkuB4cDzJe1XSSqOot90zyWoCM42s6buvg6YBDQD6icyviS1N/A2cIS7H+XuW8zsbOAsM6uc4NiSgpl1BaoQXNX7VDOr5O7fELxu/c1sr4QGmGTMdjhSUw6oa2YvA12B4909ApxoZr3jHV+sGMExqVjdSqArsNDdF7n7ZmAMwShIPnfPcvcvgBIf61OSihMzG0AwVDXa3R8GpgJXm1k7oA/BWLgmBQBmdoaZ3WBmPd39WeBBIN3MWpvZJcDFBMcMNu6+p/Azs1rAaQQV06sEf/ynRzdXIxj6k6gix6AONrMWQEWCofe/APe7+yYzO51gSH5x4qL94+I72Ec6wXDpNtnRdX+IJk6UMjMrB6QCt0VX3R79/3iCbxn3A78CF7r7yvhHmFzMbBBwPvApcEZ0WO9WgmN3lwN7AUPd/evERZkczKy1u39tZp8Dg9z9/OgsyMFmNpSgCj3H3TckNtLkUeQY1LHA/4DOwLXAjcBzZjYlum6Iuyfb79ElUh0zm1Vg+dF4XF5KSar0VXP3tWbWA5hA8EF7mbvPB+ab2UiCqdWqCoIEdTXBB+5yMzsOOBg4G7jL3TebWYpmXYGZ9QSeMLPxwA3AEDO7zt1vMrNXCWb4rXT3ZPxhv7grUkFlAv3cvZeZPQT8Asxx99lm9h5BNbrR3XMSGHJMxHgK+qpifoMrB2hYYDkjuu4P0XBfKTKz8wg+SG4FDiX45tbLzO7c1sbdf1aCguhB6hUEH67DAdz9ZWAa0B44N1qVRhIUYtIwsxTgS+AbgqG9S4FXgCPNrKu7b3X3+UpQgSIJqgNQF5hnZn8jOBY8LHr6R1+CxL4wDAkKtv2AfGz+K4GZQEsza2pmqcCJwMQ/+hyUpEqJmZ1KMG36HwRDB32ikySOAo43s5sTGV8yiSbzUUBf4DJgmJmdAeDurxIcaxkX/fD9003RN7PUbbOkzOxIgmNyzYFzCI6lbCE4bNDj/9u7+1gt6zqO4+8P5tD5wENSwxbVKjPHigjzKZ2SMZWtBuFsQqsp4ENio+Xa2rJitrXS+TBiROiYw3wgq0UxoCTGwWiRTIiDY9B0jrmeRLEU0uzTH7/fkbPjwXNg9zn3fe7zeW2M677OfV3X79y7z/W9fr/re/2+lO9W/q676RagZgK3AduBicBc21fYPihpHuW7F8eojnDcRLnf/hTwiO1OSYskZ2ITjgAABtpJREFUfQZA0tmS9gFXUjIoO/vab4b7BoCkSZQhg3nAeZQTyIKaonkQOBsY3bwWtg5Jn6N8secA1wAHKOmp10saa/sO279qZhtbwATgLkl/o/QCVlPS8l8AtgDP2t4g6S/AM0N6loQBUhMhLgXusL1f0l2UgH4/JWjNAb7QdjOaDPKME7bXAGt6rLu12/JWyjBgv+WKq8HqH8NDwGTKTdnZtqfVq4x5lKvgF20P6ayhBvoQsML2k8DXKEkkY4CvUNKnR/eSMjys2N4L7KAk2qyyvZjyPXoX8A3gQUnjba+zvbuJTW0ZvfQmTwCmA2fU1xuAb1Ky914Grra9c/BaGP2VnlQD1fToLwKHKPdSjgNGShpNuR91I/D54Thk9RZ2UbL41tRkkqX15vUPgcttH2pu81rGUsoV/1clvWr7fkoQvwE4h/IIQ1RdvcmaJLHX9jJJLwG3Seq03UG5IPpWM9s50Nrh6i5BqkEkXQLMojxrMQ8YRTmxfAl4gHLDf3Y9EcdhGyn37GZL2kg52Z5M6W0mQFW1N7VX0ouUE+0BynDf6cBNtvM8FG8kRlxo+x6Vqca+DDwnaanthyQdDyyTdKPt3zW3tQPrKB7CbWkJUo3zZ2BGTZOeAJxu+4HaK9gB/NT2a81tYuup6flLgJnALZSHT+cmM613tldLeg34PvBfyoVPAhRvzCRxGjBd0nhK5t4FlIvHaZJOsb1C0kjgdkmfTGZt60uQapAeJ9XfAxMkfZqSTTQ7AerIbD8HLJZ0H6A8fPrWbK+V9ERd/kez29MKJL0DGGP7N5KmAp8C9tVEiOWSDgEX1Smjlkp6ZDgEqJSPjyP5KyV5Yjcwx/aeJrdnSLD9SrPbMFQkOL3JKMqFzj7gPcDdwEJJN9u+x/ZKlYlkPyJpVH0cpP0N/RiVIDVA9lCeFbglASpi4NneU6eHmg98vQ617weuqw/z3m17uaRTbb/U5ObGUUiQGgD1OYyrbP+n2W2JGEa6Z0Dut/2wpL8DSyQ9b3vlcAtQbdCRSpAaKAlQEYOrRwbkd+v/JwCvAo83tXFNkuy+iIgW0y0D8nbKg7rX5uH5oStBKiLaTs2A3FYWh2uSSb8nhm1pCVIR0ZZSn609JEhFRLShrvLxQ10mmI2IiJaVIBURES0rw30REW0qw30Rg0DS65KelLRT0qpaav5Y97VC0qy6vLyr4u0R3nuxpPOP4RjPSDqtv+t7vOeoJouV9O1aBj3iTQa5fPyASJCKoeCg7Um2J1IezLy++w8lHdOIgO25fZROuZhSkj0imiRBKoaaDuADtZfTIemXwC5Jx0n6gaStknZIug5K+QZJiyXtlvRbSvl16s82SppSly+TtE3SdkmPSXovJRgurL24CyWNk/RoPcZWSRfUbd8uab2kTknL6cdsNJJ+IemJus38Hj+7s65/TNK4uu79ktbWbTokndmIDzPamA7XlGrEv2bJPakYMmqP6XJgbV01GZho++l6oj9g++xaL+hxSeuBj1FK1J8FvJNSCfi+HvsdB/wYuKjua2ydf3Ep8G/bt9f3/QS40/bmWjNsHfBhSnXXzbYXSZoOXNuPX+eaeowTga2SHrX9PHAS8CfbCyXdWvd9E7AMuL5OpHoOsASYegwfY8SQkiAVQ8GJkp6syx3AvZRhuD92m+5mGqUMw6z6ehTwQeAi4EHbr1MqtG7oZf/nApu69mV7/xHacSlwlg5fVp4q6eR6jJl1219LeqEfv9PNkmbU5XfXtj4P/A94uK5fCfysHuN8YFW3Y4/sxzFiGBOZYDZisBy0Pan7inqy7l4cUcAC2+t6vO+KBrZjBHBuz7L2OsqxEEkXUwLeebZfkbSRMhFqb1yP+2LPzyCiT20QpXJPKtrFOuAGSccDSDpD0knAJuCqes9qPHBJL9v+gVK19X1127F1/b+AU7q9bz2woOuFpK6gsQm4uq67HBjTR1tHAS/UAHUmpSfXZQSl3Dl1n5treYmnJV1ZjyFJH+3jGBFtIUEq2sVyyv2mbZJ2Aj+ijBT8nFKEchdwP7Cl54Z1AtL5lKG17RweblsNzOhKnABuBqbUxIxdHM4y/A4lyHVShv2e7aOta4G3SXoK+B4lSHZ5GfhE/R2mAovq+tnAtbV9ncBn+/GZxDDXDinost20g0dExMCY/PEp7tiytWH7O3nkiCdsT2nYDvspPamIiGhZSZyIiGhTbZA3kZ5URES0rvSkIiLaVRt0pRKkIiLaVDuUj89wX0REtKz0pCIi2lC7lI/Pc1IREW1I0lrgLeuXHaV/2r6sgfvrlwSpiIhoWbknFRERLStBKiIiWlaCVEREtKwEqYiIaFkJUhER0bL+Dzh27+QxrvF0AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "RW-K4QBYxL-v" }, "source": [ "# END" ] }, { "cell_type": "markdown", "metadata": { "id": "RkgaNWv4YOjP" }, "source": [ "# BERT BASE" ] }, { "cell_type": "markdown", "metadata": { "id": "b8d_wx8rxCga" }, "source": [ "" ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "-XfKhUITYOjR", "colab": { "base_uri": "https://localhost:8080/", "height": 680 }, "outputId": "1691126c-2748-4630-84e5-61a54905ea37" }, "source": [ "!pip install transformers==2.3.0" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Collecting transformers==2.3.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/50/10/aeefced99c8a59d828a92cc11d213e2743212d3641c87c82d61b035a7d5c/transformers-2.3.0-py3-none-any.whl (447kB)\n", "\u001b[K |████████████████████████████████| 450kB 4.5MB/s \n", "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (4.41.1)\n", "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (1.18.5)\n", "Collecting boto3\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/2f/08/f1ff665147a5d75b871bbe5ba76916f6490419c52a33e588385c4b69281b/boto3-1.15.18-py2.py3-none-any.whl (129kB)\n", "\u001b[K |████████████████████████████████| 133kB 19.8MB/s \n", "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (2019.12.20)\n", "Collecting sentencepiece\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n", "\u001b[K |████████████████████████████████| 1.1MB 16.7MB/s \n", "\u001b[?25hCollecting sacremoses\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n", "\u001b[K |████████████████████████████████| 890kB 52.9MB/s \n", "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (2.23.0)\n", "Collecting jmespath<1.0.0,>=0.7.1\n", " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", "Collecting s3transfer<0.4.0,>=0.3.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)\n", "\u001b[K |████████████████████████████████| 71kB 10.7MB/s \n", "\u001b[?25hCollecting botocore<1.19.0,>=1.18.18\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/2d/72/984ac8f33b5c8df5ff63f323a8724f65b4d0f8956968b942b77d35d3a1ef/botocore-1.18.18-py2.py3-none-any.whl (6.7MB)\n", "\u001b[K |████████████████████████████████| 6.7MB 34.1MB/s \n", "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (1.15.0)\n", "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (7.1.2)\n", "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (0.16.0)\n", "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (2.10)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (2020.6.20)\n", "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (3.0.4)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (1.24.3)\n", "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.19.0,>=1.18.18->boto3->transformers==2.3.0) (2.8.1)\n", "Building wheels for collected packages: sacremoses\n", " Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=e53998ce0e113f8a4f2ff855aabb9227bf26e6ae7a2c2a2a86837d227d84b3ea\n", " Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n", "Successfully built sacremoses\n", "Installing collected packages: jmespath, botocore, s3transfer, boto3, sentencepiece, sacremoses, transformers\n", "Successfully installed boto3-1.15.18 botocore-1.18.18 jmespath-0.10.0 s3transfer-0.3.3 sacremoses-0.0.43 sentencepiece-0.1.91 transformers-2.3.0\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "EB6spMp1YOjY" }, "source": [ "We then load and inspect the dataset we had previously prepared. In order to train faster, we use the sampled version, which contains 10% of the original prepared dataset." ] }, { "cell_type": "code", "metadata": { "id": "Eq5_BeDgYOjZ" }, "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('/content/result_31.csv', encoding='latin-1')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "PERgleXTYOjf", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "e0fee2a7-d07d-4d71-89c6-c5a9635b74ee" }, "source": [ "df.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1970, 4)" ] }, "metadata": { "tags": [] }, "execution_count": 4 } ] }, { "cell_type": "code", "metadata": { "id": "8QHny7VEYw6U", "colab": { "base_uri": "https://localhost:8080/", "height": 419 }, "outputId": "e5f5e08d-d2d9-4c7b-f2e1-2be159fd2609" }, "source": [ "df" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labels
01.005030e+05threeSnake oil Salesmen are not gone. They have jus...3
11.009760e+05threeFact check: Gargling water with salt or vinega...3
21.011710e+05zero.. this is great advice. Also, stop fighting f...0
31.017920e+05three14 people in Iran died from alcohol poisoning,...3
41.037400e+05oneTo be fair, China is not the only country alle...1
...............
19651.230000e+18one@TheSecondRevol1 @VincentCrypt46 I have been s...1
19661.230000e+18oneIf anyone is aware of the 5G theory related to...1
19671.230000e+18one@ClydeLewis @ClydeLewis have you looked into 5...1
19681.230000e+18oneHow many have the “Virus” where there’s...1
19691.230000e+18oneSo why am I still so thoroughly stunned by the...1
\n", "

1970 rows × 4 columns

\n", "
" ], "text/plain": [ " ID ... Product_Labels\n", "0 1.005030e+05 ... 3\n", "1 1.009760e+05 ... 3\n", "2 1.011710e+05 ... 0\n", "3 1.017920e+05 ... 3\n", "4 1.037400e+05 ... 1\n", "... ... ... ...\n", "1965 1.230000e+18 ... 1\n", "1966 1.230000e+18 ... 1\n", "1967 1.230000e+18 ... 1\n", "1968 1.230000e+18 ... 1\n", "1969 1.230000e+18 ... 1\n", "\n", "[1970 rows x 4 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "markdown", "metadata": { "id": "gCKjSkGfZkBZ" }, "source": [ "Clean(tweet)" ] }, { "cell_type": "code", "metadata": { "id": "9Rp6tZmH1lJT", "cellView": "both" }, "source": [ "\n", "import re\n", "import string\n", "def clean(tweet):\n", " \n", " # Special characters\n", " tweet = re.sub(r\"\\x89Û_\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÒ\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÓ\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÏWhen\", \"When\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÏ\", \"\", tweet)\n", " tweet = re.sub(r\"China\\x89Ûªs\", \"China's\", tweet)\n", " tweet = re.sub(r\"let\\x89Ûªs\", \"let's\", tweet)\n", " tweet = re.sub(r\"\\x89Û÷\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Ûª\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û\\x9d\", \"\", tweet)\n", " tweet = re.sub(r\"å_\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û¢\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û¢åÊ\", \"\", tweet)\n", " tweet = re.sub(r\"fromåÊwounds\", \"from wounds\", tweet)\n", " tweet = re.sub(r\"åÊ\", \"\", tweet)\n", " tweet = re.sub(r\"åÈ\", \"\", tweet)\n", " tweet = re.sub(r\"JapÌ_n\", \"Japan\", tweet) \n", " tweet = re.sub(r\"Ì©\", \"e\", tweet)\n", " tweet = re.sub(r\"å¨\", \"\", tweet)\n", " tweet = re.sub(r\"Surṳ\", \"Suruc\", tweet)\n", " tweet = re.sub(r\"åÇ\", \"\", tweet)\n", " tweet = re.sub(r\"å£3million\", \"3 million\", tweet)\n", " tweet = re.sub(r\"åÀ\", \"\", tweet)\n", " \n", " # Contractions\n", " tweet = re.sub(r\"he's\", \"he is\", tweet)\n", " tweet = re.sub(r\"there's\", \"there is\", tweet)\n", " tweet = re.sub(r\"We're\", \"We are\", tweet)\n", " tweet = re.sub(r\"That's\", \"That is\", tweet)\n", " tweet = re.sub(r\"won't\", \"will not\", tweet)\n", " tweet = re.sub(r\"they're\", \"they are\", tweet)\n", " tweet = re.sub(r\"Can't\", \"Cannot\", tweet)\n", " tweet = re.sub(r\"wasn't\", \"was not\", tweet)\n", " tweet = re.sub(r\"don\\x89Ûªt\", \"do not\", tweet)\n", " tweet = re.sub(r\"aren't\", \"are not\", tweet)\n", " tweet = re.sub(r\"isn't\", \"is not\", tweet)\n", " tweet = re.sub(r\"What's\", \"What is\", tweet)\n", " tweet = re.sub(r\"haven't\", \"have not\", tweet)\n", " tweet = re.sub(r\"hasn't\", \"has not\", tweet)\n", " tweet = re.sub(r\"There's\", \"There is\", tweet)\n", " tweet = re.sub(r\"He's\", \"He is\", tweet)\n", " tweet = re.sub(r\"It's\", \"It is\", tweet)\n", " tweet = re.sub(r\"You're\", \"You are\", tweet)\n", " tweet = re.sub(r\"I'M\", \"I am\", tweet)\n", " tweet = re.sub(r\"shouldn't\", \"should not\", tweet)\n", " tweet = re.sub(r\"wouldn't\", \"would not\", tweet)\n", " tweet = re.sub(r\"i'm\", \"I am\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªm\", \"I am\", tweet)\n", " tweet = re.sub(r\"I'm\", \"I am\", tweet)\n", " tweet = re.sub(r\"Isn't\", \"is not\", tweet)\n", " tweet = re.sub(r\"Here's\", \"Here is\", tweet)\n", " tweet = re.sub(r\"you've\", \"you have\", tweet)\n", " tweet = re.sub(r\"you\\x89Ûªve\", \"you have\", tweet)\n", " tweet = re.sub(r\"we're\", \"we are\", tweet)\n", " tweet = re.sub(r\"what's\", \"what is\", tweet)\n", " tweet = re.sub(r\"couldn't\", \"could not\", tweet)\n", " tweet = re.sub(r\"we've\", \"we have\", tweet)\n", " tweet = re.sub(r\"it\\x89Ûªs\", \"it is\", tweet)\n", " tweet = re.sub(r\"doesn\\x89Ûªt\", \"does not\", tweet)\n", " tweet = re.sub(r\"It\\x89Ûªs\", \"It is\", tweet)\n", " tweet = re.sub(r\"Here\\x89Ûªs\", \"Here is\", tweet)\n", " tweet = re.sub(r\"who's\", \"who is\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªve\", \"I have\", tweet)\n", " tweet = re.sub(r\"y'all\", \"you all\", tweet)\n", " tweet = re.sub(r\"can\\x89Ûªt\", \"cannot\", tweet)\n", " tweet = re.sub(r\"would've\", \"would have\", tweet)\n", " tweet = re.sub(r\"it'll\", \"it will\", tweet)\n", " tweet = re.sub(r\"we'll\", \"we will\", tweet)\n", " tweet = re.sub(r\"wouldn\\x89Ûªt\", \"would not\", tweet)\n", " tweet = re.sub(r\"We've\", \"We have\", tweet)\n", " tweet = re.sub(r\"he'll\", \"he will\", tweet)\n", " tweet = re.sub(r\"Y'all\", \"You all\", tweet)\n", " tweet = re.sub(r\"Weren't\", \"Were not\", tweet)\n", " tweet = re.sub(r\"Didn't\", \"Did not\", tweet)\n", " tweet = re.sub(r\"they'll\", \"they will\", tweet)\n", " tweet = re.sub(r\"they'd\", \"they would\", tweet)\n", " tweet = re.sub(r\"DON'T\", \"DO NOT\", tweet)\n", " tweet = re.sub(r\"That\\x89Ûªs\", \"That is\", tweet)\n", " tweet = re.sub(r\"they've\", \"they have\", tweet)\n", " tweet = re.sub(r\"i'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"should've\", \"should have\", tweet)\n", " tweet = re.sub(r\"You\\x89Ûªre\", \"You are\", tweet)\n", " tweet = re.sub(r\"where's\", \"where is\", tweet)\n", " tweet = re.sub(r\"Don\\x89Ûªt\", \"Do not\", tweet)\n", " tweet = re.sub(r\"we'd\", \"we would\", tweet)\n", " tweet = re.sub(r\"i'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"weren't\", \"were not\", tweet)\n", " tweet = re.sub(r\"They're\", \"They are\", tweet)\n", " tweet = re.sub(r\"Can\\x89Ûªt\", \"Cannot\", tweet)\n", " tweet = re.sub(r\"you\\x89Ûªll\", \"you will\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªd\", \"I would\", tweet)\n", " tweet = re.sub(r\"let's\", \"let us\", tweet)\n", " tweet = re.sub(r\"it's\", \"it is\", tweet)\n", " tweet = re.sub(r\"can't\", \"cannot\", tweet)\n", " tweet = re.sub(r\"don't\", \"do not\", tweet)\n", " tweet = re.sub(r\"you're\", \"you are\", tweet)\n", " tweet = re.sub(r\"i've\", \"I have\", tweet)\n", " tweet = re.sub(r\"that's\", \"that is\", tweet)\n", " tweet = re.sub(r\"i'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"doesn't\", \"does not\", tweet)\n", " tweet = re.sub(r\"i'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"didn't\", \"did not\", tweet)\n", " tweet = re.sub(r\"ain't\", \"am not\", tweet)\n", " tweet = re.sub(r\"you'll\", \"you will\", tweet)\n", " tweet = re.sub(r\"I've\", \"I have\", tweet)\n", " tweet = re.sub(r\"Don't\", \"do not\", tweet)\n", " tweet = re.sub(r\"I'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"I'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"Let's\", \"Let us\", tweet)\n", " tweet = re.sub(r\"you'd\", \"You would\", tweet)\n", " tweet = re.sub(r\"It's\", \"It is\", tweet)\n", " tweet = re.sub(r\"Ain't\", \"am not\", tweet)\n", " tweet = re.sub(r\"Haven't\", \"Have not\", tweet)\n", " tweet = re.sub(r\"Could've\", \"Could have\", tweet)\n", " tweet = re.sub(r\"youve\", \"you have\", tweet) \n", " tweet = re.sub(r\"donå«t\", \"do not\", tweet) \n", " \n", " # Character entity references\n", " tweet = re.sub(r\">\", \">\", tweet)\n", " tweet = re.sub(r\"<\", \"<\", tweet)\n", " tweet = re.sub(r\"&\", \"&\", tweet)\n", " \n", " # Typos, slang and informal abbreviations\n", " tweet = re.sub(r\"w/e\", \"whatever\", tweet)\n", " tweet = re.sub(r\"w/\", \"with\", tweet)\n", " tweet = re.sub(r\"USAgov\", \"USA government\", tweet)\n", " tweet = re.sub(r\"recentlu\", \"recently\", tweet)\n", " tweet = re.sub(r\"Ph0tos\", \"Photos\", tweet)\n", " tweet = re.sub(r\"amirite\", \"am I right\", tweet)\n", " tweet = re.sub(r\"exp0sed\", \"exposed\", tweet)\n", " tweet = re.sub(r\"<3\", \"love\", tweet)\n", " tweet = re.sub(r\"amageddon\", \"armageddon\", tweet)\n", " tweet = re.sub(r\"Trfc\", \"Traffic\", tweet)\n", " tweet = re.sub(r\"8/5/2015\", \"2015-08-05\", tweet)\n", " tweet = re.sub(r\"WindStorm\", \"Wind Storm\", tweet)\n", " tweet = re.sub(r\"8/6/2015\", \"2015-08-06\", tweet)\n", " tweet = re.sub(r\"10:38PM\", \"10:38 PM\", tweet)\n", " tweet = re.sub(r\"10:30pm\", \"10:30 PM\", tweet)\n", " tweet = re.sub(r\"16yr\", \"16 year\", tweet)\n", " tweet = re.sub(r\"lmao\", \"laughing my ass off\", tweet) \n", " tweet = re.sub(r\"TRAUMATISED\", \"traumatized\", tweet)\n", " \n", " # re.sub(r\"humanconsumption\", \"human consumption\", tweet)\n", " # tweet = re.sub(r\"Typhoon-Devastated\", \"Typhoon Devastated\", tweet)\n", " # tweet = re.sub(r\"Meat-Loving\", \"Meat Loving\", tweet)\n", " # tweet = re.sub(r\"facialabuse\", \"facial abuse\", tweet)\n", " # tweet = re.sub(r\"LakeCounty\", \"Lake County\", tweet)\n", " # tweet = re.sub(r\"BeingAuthor\", \"Being Author\", tweet)\n", " # tweet = re.sub(r\"withheavenly\", \"with heavenly\", tweet)\n", " # tweet = re.sub(r\"thankU\", \"thank you\", tweet)\n", " # tweet = re.sub(r\"iTunesMusic\", \"iTunes Music\", tweet)\n", " # tweet = re.sub(r\"OffensiveContent\", \"Offensive Content\", tweet)\n", " # tweet = re.sub(r\"WorstSummerJob\", \"Worst Summer Job\", tweet)\n", " # tweet = re.sub(r\"HarryBeCareful\", \"Harry Be Careful\", tweet)\n", " # tweet = re.sub(r\"NASASolarSystem\", \"NASA Solar System\", tweet)\n", " # tweet = re.sub(r\"animalrescue\", \"animal rescue\", tweet)\n", " # tweet = re.sub(r\"KurtSchlichter\", \"Kurt Schlichter\", tweet)\n", " # tweet = re.sub(r\"aRmageddon\", \"armageddon\", tweet)\n", " # tweet = re.sub(r\"Throwingknifes\", \"Throwing knives\", tweet)\n", " # tweet = re.sub(r\"GodsLove\", \"God's Love\", tweet)\n", " # tweet = re.sub(r\"bookboost\", \"book boost\", tweet)\n", " # tweet = re.sub(r\"ibooklove\", \"I book love\", tweet)\n", " # tweet = re.sub(r\"NestleIndia\", \"Nestle India\", tweet)\n", " # tweet = re.sub(r\"realDonaldTrump\", \"Donald Trump\", tweet)\n", " # tweet = re.sub(r\"DavidVonderhaar\", \"David Vonderhaar\", tweet)\n", " # tweet = re.sub(r\"CecilTheLion\", \"Cecil The Lion\", tweet)\n", " # tweet = re.sub(r\"weathernetwork\", \"weather network\", tweet)\n", " # tweet = re.sub(r\"withBioterrorism # Hashtags and usernames\n", " # tweet = re.sub(r\"IranDeal\", \"Iran Deal\", tweet)\n", " # tweet = re.sub(r\"ArianaGrande\", \"Ariana Grande\", tweet)\n", " # tweet = re.sub(r\"camilacabello97\", \"camila cabello\", tweet) \n", " # tweet = re.sub(r\"RondaRousey\", \"Ronda Rousey\", tweet) \n", " # tweet = re.sub(r\"MTVHottest\", \"MTV Hottest\", tweet)\n", " # tweet = re.sub(r\"TrapMusic\", \"Trap Music\", tweet)\n", " # tweet = re.sub(r\"ProphetMuhammad\", \"Prophet Muhammad\", tweet)\n", " # tweet = re.sub(r\"PantherAttack\", \"Panther Attack\", tweet)\n", " # tweet = re.sub(r\"StrategicPatience\", \"Strategic Patience\", tweet)\n", " # tweet = re.sub(r\"socialnews\", \"social news\", tweet)\n", " # tweet = re.sub(r\"NASAHurricane\", \"NASA Hurricane\", tweet)\n", " # tweet = re.sub(r\"onlinecommunities\", \"online communities\", tweet)\n", " # tweet = &use\", \"with Bioterrorism & use\", tweet)\n", " # tweet = re.sub(r\"Hostage&2\", \"Hostage & 2\", tweet)\n", " # tweet = re.sub(r\"GOPDebate\", \"GOP Debate\", tweet)\n", " # tweet = re.sub(r\"RickPerry\", \"Rick Perry\", tweet)\n", " # tweet = re.sub(r\"frontpage\", \"front page\", tweet)\n", " # tweet = re.sub(r\"NewsInTweets\", \"News In Tweets\", tweet)\n", " # tweet = re.sub(r\"ViralSpell\", \"Viral Spell\", tweet)\n", " # tweet = re.sub(r\"til_now\", \"until now\", tweet)\n", " # tweet = re.sub(r\"volcanoinRussia\", \"volcano in Russia\", tweet)\n", " # tweet = re.sub(r\"ZippedNews\", \"Zipped News\", tweet)\n", " # tweet = re.sub(r\"MicheleBachman\", \"Michele Bachman\", tweet)\n", " # tweet = re.sub(r\"53inch\", \"53 inch\", tweet)\n", " # tweet = re.sub(r\"KerrickTrial\", \"Kerrick Trial\", tweet)\n", " # tweet = re.sub(r\"abstorm\", \"Alberta Storm\", tweet)\n", " # tweet = re.sub(r\"Beyhive\", \"Beyonce hive\", tweet)\n", " # tweet = re.sub(r\"IDFire\", \"Idaho Fire\", tweet)\n", " # tweet = re.sub(r\"DETECTADO\", \"Detected\", tweet)\n", " # tweet = re.sub(r\"RockyFire\", \"Rocky Fire\", tweet)\n", " # tweet = re.sub(r\"Listen/Buy\", \"Listen / Buy\", tweet)\n", " # tweet = re.sub(r\"NickCannon\", \"Nick Cannon\", tweet)\n", " # tweet = re.sub(r\"FaroeIslands\", \"Faroe Islands\", tweet)\n", " # tweet = re.sub(r\"yycstorm\", \"Calgary Storm\", tweet)\n", " # tweet = re.sub(r\"IDPs:\", \"Internally Displaced People :\", tweet)\n", " # tweet = re.sub(r\"ArtistsUnited\", \"Artists United\", tweet)\n", " # tweet = re.sub(r\"ClaytonBryant\", \"Clayton Bryant\", tweet)\n", " # tweet = re.sub(r\"jimmyfallon\", \"jimmy fallon\", tweet)\n", " # tweet = re.sub(r\"justinbieber\", \"justin bieber\", tweet) \n", " # tweet = re.sub(r\"UTC2015\", \"UTC 2015\", tweet)\n", " # tweet = re.sub(r\"Time2015\", \"Time 2015\", tweet)\n", " # tweet = re.sub(r\"djicemoon\", \"dj icemoon\", tweet)\n", " # tweet = re.sub(r\"LivingSafely\", \"Living Safely\", tweet)\n", " # tweet = re.sub(r\"FIFA16\", \"Fifa 2016\", tweet)\n", " # tweet = re.sub(r\"thisiswhywecanthavenicethings\", \"this is why we cannot have nice things\", tweet)\n", " # tweet = re.sub(r\"bbcnews\", \"bbc news\", tweet)\n", " # tweet = re.sub(r\"UndergroundRailraod\", \"Underground Railraod\", tweet)\n", " # tweet = re.sub(r\"c4news\", \"c4 news\", tweet)\n", " # tweet = re.sub(r\"OBLITERATION\", \"obliteration\", tweet)\n", " # tweet = re.sub(r\"MUDSLIDE\", \"mudslide\", tweet)\n", " # tweet = re.sub(r\"NoSurrender\", \"No Surrender\", tweet)\n", " # tweet = re.sub(r\"NotExplained\", \"Not Explained\", tweet)\n", " # tweet = re.sub(r\"greatbritishbakeoff\", \"great british bake off\", tweet)\n", " # tweet = re.sub(r\"LondonFire\", \"London Fire\", tweet)\n", " # tweet = re.sub(r\"KOTAWeather\", \"KOTA Weather\", tweet)\n", " # tweet = re.sub(r\"LuchaUnderground\", \"Lucha Underground\", tweet)\n", " # tweet = re.sub(r\"KOIN6News\", \"KOIN 6 News\", tweet)\n", " # tweet = re.sub(r\"LiveOnK2\", \"Live On K2\", tweet)\n", " # tweet = re.sub(r\"9NewsGoldCoast\", \"9 News Gold Coast\", tweet)\n", " # tweet = re.sub(r\"nikeplus\", \"nike plus\", tweet)\n", " # tweet = re.sub(r\"david_cameron\", \"David Cameron\", tweet)\n", " # tweet = re.sub(r\"peterjukes\", \"Peter Jukes\", tweet)\n", " # tweet = re.sub(r\"JamesMelville\", \"James Melville\", tweet)\n", " # tweet = re.sub(r\"megynkelly\", \"Megyn Kelly\", tweet)\n", " # tweet = re.sub(r\"cnewslive\", \"C News Live\", tweet)\n", " # tweet = re.sub(r\"JamaicaObserver\", \"Jamaica Observer\", tweet)\n", " # tweet = re.sub(r\"TweetLikeItsSeptember11th2001\", \"Tweet like it is september 11th 2001\", tweet)\n", " # tweet = re.sub(r\"cbplawyers\", \"cbp lawyers\", tweet)\n", " # tweet = re.sub(r\"fewmoretweets\", \"few more tweets\", tweet)\n", " # tweet = re.sub(r\"BlackLivesMatter\", \"Black Lives Matter\", tweet)\n", " # tweet = re.sub(r\"cjoyner\", \"Chris Joyner\", tweet)\n", " # tweet = re.sub(r\"ENGvAUS\", \"England vs Australia\", tweet)\n", " # tweet = re.sub(r\"ScottWalker\", \"Scott Walker\", tweet)\n", " # tweet = re.sub(r\"MikeParrActor\", \"Michael Parr\", tweet)\n", " # tweet = re.sub(r\"4PlayThursdays\", \"Foreplay Thursdays\", tweet)\n", " # tweet = re.sub(r\"TGF2015\", \"Tontitown Grape Festival\", tweet)\n", " # tweet = re.sub(r\"realmandyrain\", \"Mandy Rain\", tweet)\n", " # tweet = re.sub(r\"GraysonDolan\", \"Grayson Dolan\", tweet)\n", " # tweet = re.sub(r\"ApolloBrown\", \"Apollo Brown\", tweet)\n", " # tweet = re.sub(r\"saddlebrooke\", \"Saddlebrooke\", tweet)\n", " # tweet = re.sub(r\"TontitownGrape\", \"Tontitown Grape\", tweet)\n", " # tweet = re.sub(r\"AbbsWinston\", \"Abbs Winston\", tweet)\n", " # tweet = re.sub(r\"ShaunKing\", \"Shaun King\", tweet)\n", " # tweet = re.sub(r\"MeekMill\", \"Meek Mill\", tweet)\n", " # tweet = re.sub(r\"TornadoGiveaway\", \"Tornado Giveaway\", tweet)\n", " # tweet = re.sub(r\"GRupdates\", \"GR updates\", tweet)\n", " # tweet = re.sub(r\"SouthDowns\", \"South Downs\", tweet)\n", " # tweet = re.sub(r\"braininjury\", \"brain injury\", tweet)\n", " # tweet = re.sub(r\"auspol\", \"Australian politics\", tweet)\n", " # tweet = re.sub(r\"PlannedParenthood\", \"Planned Parenthood\", tweet)\n", " # tweet = re.sub(r\"calgaryweather\", \"Calgary Weather\", tweet)\n", " # tweet = re.sub(r\"weallheartonedirection\", \"we all heart one direction\", tweet)\n", " # tweet = re.sub(r\"edsheeran\", \"Ed Sheeran\", tweet)\n", " # tweet = re.sub(r\"TrueHeroes\", \"True Heroes\", tweet)\n", " # tweet = re.sub(r\"S3XLEAK\", \"sex leak\", tweet)\n", " # tweet = re.sub(r\"ComplexMag\", \"Complex Magazine\", tweet)\n", " # tweet = re.sub(r\"TheAdvocateMag\", \"The Advocate Magazine\", tweet)\n", " # tweet = re.sub(r\"CityofCalgary\", \"City of Calgary\", tweet)\n", " # tweet = re.sub(r\"EbolaOutbreak\", \"Ebola Outbreak\", tweet)\n", " # tweet = re.sub(r\"SummerFate\", \"Summer Fate\", tweet)\n", " # tweet = re.sub(r\"RAmag\", \"Royal Academy Magazine\", tweet)\n", " # tweet = re.sub(r\"offers2go\", \"offers to go\", tweet)\n", " # tweet = re.sub(r\"foodscare\", \"food scare\", tweet)\n", " # tweet = re.sub(r\"MNPDNashville\", \"Metropolitan Nashville Police Department\", tweet)\n", " # tweet = re.sub(r\"TfLBusAlerts\", \"TfL Bus Alerts\", tweet)\n", " # tweet = re.sub(r\"GamerGate\", \"Gamer Gate\", tweet)\n", " # tweet = re.sub(r\"IHHen\", \"Humanitarian Relief\", tweet)\n", " # tweet = re.sub(r\"spinningbot\", \"spinning bot\", tweet)\n", " # tweet = re.sub(r\"ModiMinistry\", \"Modi Ministry\", tweet)\n", " # tweet = re.sub(r\"TAXIWAYS\", \"taxi ways\", tweet)\n", " # tweet = re.sub(r\"Calum5SOS\", \"Calum Hood\", tweet)\n", " # tweet = re.sub(r\"po_st\", \"po.st\", tweet)\n", " # tweet = re.sub(r\"scoopit\", \"scoop.it\", tweet)\n", " # tweet = re.sub(r\"UltimaLucha\", \"Ultima Lucha\", tweet)\n", " # tweet = re.sub(r\"JonathanFerrell\", \"Jonathan Ferrell\", tweet)\n", " # tweet = re.sub(r\"aria_ahrary\", \"Aria Ahrary\", tweet)\n", " # tweet = re.sub(r\"rapidcity\", \"Rapid City\", tweet)\n", " # tweet = re.sub(r\"OutBid\", \"outbid\", tweet)\n", " # tweet = re.sub(r\"lavenderpoetrycafe\", \"lavender poetry cafe\", tweet)\n", " # tweet = re.sub(r\"EudryLantiqua\", \"Eudry Lantiqua\", tweet)\n", " # tweet = re.sub(r\"15PM\", \"15 PM\", tweet)\n", " # tweet = re.sub(r\"OriginalFunko\", \"Funko\", tweet)\n", " # tweet = re.sub(r\"rightwaystan\", \"Richard Tan\", tweet)\n", " # tweet = re.sub(r\"CindyNoonan\", \"Cindy Noonan\", tweet)\n", " # tweet = re.sub(r\"RT_America\", \"RT America\", tweet)\n", " # tweet = re.sub(r\"narendramodi\", \"Narendra Modi\", tweet)\n", " # tweet = re.sub(r\"BakeOffFriends\", \"Bake Off Friends\", tweet)\n", " # tweet = re.sub(r\"TeamHendrick\", \"Hendrick Motorsports\", tweet)\n", " # tweet = re.sub(r\"alexbelloli\", \"Alex Belloli\", tweet)\n", " # tweet = re.sub(r\"itsjustinstuart\", \"Justin Stuart\", tweet)\n", " # tweet = re.sub(r\"gunsense\", \"gun sense\", tweet)\n", " # tweet = re.sub(r\"DebateQuestionsWeWantToHear\", \"debate questions we want to hear\", tweet)\n", " # tweet = re.sub(r\"RoyalCarribean\", \"Royal Carribean\", tweet)\n", " # tweet = re.sub(r\"samanthaturne19\", \"Samantha Turner\", tweet)\n", " # tweet = re.sub(r\"JonVoyage\", \"Jon Stewart\", tweet)\n", " # tweet = re.sub(r\"renew911health\", \"renew 911 health\", tweet)\n", " # tweet = re.sub(r\"SuryaRay\", \"Surya Ray\", tweet)\n", " # tweet = re.sub(r\"pattonoswalt\", \"Patton Oswalt\", tweet)\n", " # tweet = re.sub(r\"minhazmerchant\", \"Minhaz Merchant\", tweet)\n", " # tweet = re.sub(r\"TLVFaces\", \"Israel Diaspora Coalition\", tweet)\n", " # tweet = re.sub(r\"pmarca\", \"Marc Andreessen\", tweet)\n", " # tweet = re.sub(r\"pdx911\", \"Portland Police\", tweet)\n", " # tweet = re.sub(r\"jamaicaplain\", \"Jamaica Plain\", tweet)\n", " # tweet = re.sub(r\"Japton\", \"Arkansas\", tweet)\n", " # tweet = re.sub(r\"RouteComplex\", \"Route Complex\", tweet)\n", " # tweet = re.sub(r\"INSubcontinent\", \"Indian Subcontinent\", tweet)\n", " # tweet = re.sub(r\"NJTurnpike\", \"New Jersey Turnpike\", tweet)\n", " # tweet = re.sub(r\"Politifiact\", \"PolitiFact\", tweet)\n", " # tweet = re.sub(r\"Hiroshima70\", \"Hiroshima\", tweet)\n", " # tweet = re.sub(r\"GMMBC\", \"Greater Mt Moriah Baptist Church\", tweet)\n", " # tweet = re.sub(r\"versethe\", \"verse the\", tweet)\n", " # tweet = re.sub(r\"TubeStrike\", \"Tube Strike\", tweet)\n", " # tweet = re.sub(r\"MissionHills\", \"Mission Hills\", tweet)\n", " # tweet = re.sub(r\"ProtectDenaliWolves\", \"Protect Denali Wolves\", tweet)\n", " # tweet = re.sub(r\"NANKANA\", \"Nankana\", tweet)\n", " # tweet = re.sub(r\"SAHIB\", \"Sahib\", tweet)\n", " # tweet = re.sub(r\"PAKPATTAN\", \"Pakpattan\", tweet)\n", " # tweet = re.sub(r\"Newz_Sacramento\", \"News Sacramento\", tweet)\n", " # tweet = re.sub(r\"gofundme\", \"go fund me\", tweet)\n", " # tweet = re.sub(r\"pmharper\", \"Stephen Harper\", tweet)\n", " # tweet = re.sub(r\"IvanBerroa\", \"Ivan Berroa\", tweet)\n", " # tweet = re.sub(r\"LosDelSonido\", \"Los Del Sonido\", tweet)\n", " # tweet = re.sub(r\"bancodeseries\", \"banco de series\", tweet)\n", " # tweet = re.sub(r\"timkaine\", \"Tim Kaine\", tweet)\n", " # tweet = re.sub(r\"IdentityTheft\", \"Identity Theft\", tweet)\n", " # tweet = re.sub(r\"AllLivesMatter\", \"All Lives Matter\", tweet)\n", " # tweet = re.sub(r\"mishacollins\", \"Misha Collins\", tweet)\n", " # tweet = re.sub(r\"BillNeelyNBC\", \"Bill Neely\", tweet)\n", " # tweet = re.sub(r\"BeClearOnCancer\", \"be clear on cancer\", tweet)\n", " # tweet = re.sub(r\"Kowing\", \"Knowing\", tweet)\n", " # tweet = re.sub(r\"ScreamQueens\", \"Scream Queens\", tweet)\n", " # tweet = re.sub(r\"AskCharley\", \"Ask Charley\", tweet)\n", " # tweet = re.sub(r\"BlizzHeroes\", \"Heroes of the Storm\", tweet)\n", " # tweet = re.sub(r\"BradleyBrad47\", \"Bradley Brad\", tweet)\n", " # tweet = re.sub(r\"HannaPH\", \"Typhoon Hanna\", tweet)\n", " # tweet = re.sub(r\"meinlcymbals\", \"MEINL Cymbals\", tweet)\n", " # tweet = re.sub(r\"Ptbo\", \"Peterborough\", tweet)\n", " # tweet = re.sub(r\"cnnbrk\", \"CNN Breaking News\", tweet)\n", " # tweet = re.sub(r\"IndianNews\", \"Indian News\", tweet)\n", " # tweet = re.sub(r\"savebees\", \"save bees\", tweet)\n", " # tweet = re.sub(r\"GreenHarvard\", \"Green Harvard\", tweet)\n", " # tweet = re.sub(r\"StandwithPP\", \"Stand with planned parenthood\", tweet)\n", " # tweet = re.sub(r\"hermancranston\", \"Herman Cranston\", tweet)\n", " # tweet = re.sub(r\"WMUR9\", \"WMUR-TV\", tweet)\n", " # tweet = re.sub(r\"RockBottomRadFM\", \"Rock Bottom Radio\", tweet)\n", " # tweet = re.sub(r\"ameenshaikh3\", \"Ameen Shaikh\", tweet)\n", " # tweet = re.sub(r\"ProSyn\", \"Project Syndicate\", tweet)\n", " # tweet = re.sub(r\"Daesh\", \"ISIS\", tweet)\n", " # tweet = re.sub(r\"s2g\", \"swear to god\", tweet)\n", " # tweet = re.sub(r\"listenlive\", \"listen live\", tweet)\n", " # tweet = re.sub(r\"CDCgov\", \"Centers for Disease Control and Prevention\", tweet)\n", " # tweet = re.sub(r\"FoxNew\", \"Fox News\", tweet)\n", " # tweet = re.sub(r\"CBSBigBrother\", \"Big Brother\", tweet)\n", " # tweet = re.sub(r\"JulieDiCaro\", \"Julie DiCaro\", tweet)\n", " # tweet = re.sub(r\"theadvocatemag\", \"The Advocate Magazine\", tweet)\n", " # tweet = re.sub(r\"RohnertParkDPS\", \"Rohnert Park Police Department\", tweet)\n", " # tweet = re.sub(r\"THISIZBWRIGHT\", \"Bonnie Wright\", tweet)\n", " # tweet = re.sub(r\"Popularmmos\", \"Popular MMOs\", tweet)\n", " # tweet = re.sub(r\"WildHorses\", \"Wild Horses\", tweet)\n", " # tweet = re.sub(r\"FantasticFour\", \"Fantastic Four\", tweet)\n", " # tweet = re.sub(r\"HORNDALE\", \"Horndale\", tweet)\n", " # tweet = re.sub(r\"PINER\", \"Piner\", tweet)\n", " # tweet = re.sub(r\"BathAndNorthEastSomerset\", \"Bath and North East Somerset\", tweet)\n", " # tweet = re.sub(r\"thatswhatfriendsarefor\", \"that is what friends are for\", tweet)\n", " # tweet = re.sub(r\"residualincome\", \"residual income\", tweet)\n", " # tweet = re.sub(r\"YahooNewsDigest\", \"Yahoo News Digest\", tweet)\n", " # tweet = re.sub(r\"MalaysiaAirlines\", \"Malaysia Airlines\", tweet)\n", " # tweet = re.sub(r\"AmazonDeals\", \"Amazon Deals\", tweet)\n", " # tweet = re.sub(r\"MissCharleyWebb\", \"Charley Webb\", tweet)\n", " # tweet = re.sub(r\"shoalstraffic\", \"shoals traffic\", tweet)\n", " # tweet = re.sub(r\"GeorgeFoster72\", \"George Foster\", tweet)\n", " # tweet = re.sub(r\"pop2015\", \"pop 2015\", tweet)\n", " # tweet = re.sub(r\"_PokemonCards_\", \"Pokemon Cards\", tweet)\n", " # tweet = re.sub(r\"DianneG\", \"Dianne Gallagher\", tweet)\n", " # tweet = re.sub(r\"KashmirConflict\", \"Kashmir Conflict\", tweet)\n", " # tweet = re.sub(r\"BritishBakeOff\", \"British Bake Off\", tweet)\n", " # tweet = re.sub(r\"FreeKashmir\", \"Free Kashmir\", tweet)\n", " # tweet = re.sub(r\"mattmosley\", \"Matt Mosley\", tweet)\n", " # tweet = re.sub(r\"BishopFred\", \"Bishop Fred\", tweet)\n", " # tweet = re.sub(r\"EndConflict\", \"End Conflict\", tweet)\n", " # tweet = re.sub(r\"EndOccupation\", \"End Occupation\", tweet)\n", " # tweet = re.sub(r\"UNHEALED\", \"unhealed\", tweet)\n", " # tweet = re.sub(r\"CharlesDagnall\", \"Charles Dagnall\", tweet)\n", " # tweet = re.sub(r\"Latestnews\", \"Latest news\", tweet)\n", " # tweet = re.sub(r\"KindleCountdown\", \"Kindle Countdown\", tweet)\n", " # tweet = re.sub(r\"NoMoreHandouts\", \"No More Handouts\", tweet)\n", " # tweet = re.sub(r\"datingtips\", \"dating tips\", tweet)\n", " # tweet = re.sub(r\"charlesadler\", \"Charles Adler\", tweet)\n", " # tweet = re.sub(r\"twia\", \"Texas Windstorm Insurance Association\", tweet)\n", " # tweet = re.sub(r\"txlege\", \"Texas Legislature\", tweet)\n", " # tweet = re.sub(r\"WindstormInsurer\", \"Windstorm Insurer\", tweet)\n", " # tweet = re.sub(r\"Newss\", \"News\", tweet)\n", " # tweet = re.sub(r\"hempoil\", \"hemp oil\", tweet)\n", " # tweet = re.sub(r\"CommoditiesAre\", \"Commodities are\", tweet)\n", " # tweet = re.sub(r\"tubestrike\", \"tube strike\", tweet)\n", " # tweet = re.sub(r\"JoeNBC\", \"Joe Scarborough\", tweet)\n", " # tweet = re.sub(r\"LiteraryCakes\", \"Literary Cakes\", tweet)\n", " # tweet = re.sub(r\"TI5\", \"The International 5\", tweet)\n", " # tweet = re.sub(r\"thehill\", \"the hill\", tweet)\n", " # tweet = re.sub(r\"3others\", \"3 others\", tweet)\n", " # tweet = re.sub(r\"stighefootball\", \"Sam Tighe\", tweet)\n", " # tweet = re.sub(r\"whatstheimportantvideo\", \"what is the important video\", tweet)\n", " # tweet = re.sub(r\"ClaudioMeloni\", \"Claudio Meloni\", tweet)\n", " # tweet = re.sub(r\"DukeSkywalker\", \"Duke Skywalker\", tweet)\n", " # tweet = re.sub(r\"carsonmwr\", \"Fort Carson\", tweet)\n", " # tweet = re.sub(r\"offdishduty\", \"off dish duty\", tweet)\n", " # tweet = re.sub(r\"andword\", \"and word\", tweet)\n", " # tweet = re.sub(r\"rhodeisland\", \"Rhode Island\", tweet)\n", " # tweet = re.sub(r\"easternoregon\", \"Eastern Oregon\", tweet)\n", " # tweet = re.sub(r\"WAwildfire\", \"Washington Wildfire\", tweet)\n", " # tweet = re.sub(r\"fingerrockfire\", \"Finger Rock Fire\", tweet)\n", " # tweet = re.sub(r\"57am\", \"57 am\", tweet)\n", " # tweet = re.sub(r\"fingerrockfire\", \"Finger Rock Fire\", tweet)\n", " # tweet = re.sub(r\"JacobHoggard\", \"Jacob Hoggard\", tweet)\n", " # tweet = re.sub(r\"newnewnew\", \"new new new\", tweet)\n", " # tweet = re.sub(r\"under50\", \"under 50\", tweet)\n", " # tweet = re.sub(r\"getitbeforeitsgone\", \"get it before it is gone\", tweet)\n", " # tweet = re.sub(r\"freshoutofthebox\", \"fresh out of the box\", tweet)\n", " # tweet = re.sub(r\"amwriting\", \"am writing\", tweet)\n", " # tweet = re.sub(r\"Bokoharm\", \"Boko Haram\", tweet)\n", " # tweet = re.sub(r\"Nowlike\", \"Now like\", tweet)\n", " # tweet = re.sub(r\"seasonfrom\", \"season from\", tweet)\n", " # tweet = re.sub(r\"epicente\", \"epicenter\", tweet)\n", " # tweet = re.sub(r\"epicenterr\", \"epicenter\", tweet)\n", " # tweet = re.sub(r\"sicklife\", \"sick life\", tweet)\n", " # tweet = re.sub(r\"yycweather\", \"Calgary Weather\", tweet)\n", " # tweet = re.sub(r\"calgarysun\", \"Calgary Sun\", tweet)\n", " # tweet = re.sub(r\"approachng\", \"approaching\", tweet)\n", " # tweet = re.sub(r\"evng\", \"evening\", tweet)\n", " # tweet = re.sub(r\"Sumthng\", \"something\", tweet)\n", " # tweet = re.sub(r\"EllenPompeo\", \"Ellen Pompeo\", tweet)\n", " # tweet = re.sub(r\"shondarhimes\", \"Shonda Rhimes\", tweet)\n", " # tweet = re.sub(r\"ABCNetwork\", \"ABC Network\", tweet)\n", " # tweet = re.sub(r\"SushmaSwaraj\", \"Sushma Swaraj\", tweet)\n", " # tweet = re.sub(r\"pray4japan\", \"Pray for Japan\", tweet)\n", " # tweet = re.sub(r\"hope4japan\", \"Hope for Japan\", tweet)\n", " # tweet = re.sub(r\"Illusionimagess\", \"Illusion images\", tweet)\n", " # tweet = re.sub(r\"SummerUnderTheStars\", \"Summer Under The Stars\", tweet)\n", " # tweet = re.sub(r\"ShallWeDance\", \"Shall We Dance\", tweet)\n", " # tweet = re.sub(r\"TCMParty\", \"TCM Party\", tweet)\n", " # tweet = re.sub(r\"marijuananews\", \"marijuana news\", tweet)\n", " # tweet = re.sub(r\"onbeingwithKristaTippett\", \"on being with Krista Tippett\", tweet)\n", " # tweet = re.sub(r\"Beingtweets\", \"Being tweets\", tweet)\n", " # tweet = re.sub(r\"newauthors\", \"new authors\", tweet)\n", " # tweet = re.sub(r\"remedyyyy\", \"remedy\", tweet)\n", " # tweet = re.sub(r\"44PM\", \"44 PM\", tweet)\n", " # tweet = re.sub(r\"HeadlinesApp\", \"Headlines App\", tweet)\n", " # tweet = re.sub(r\"40PM\", \"40 PM\", tweet)\n", " # tweet = re.sub(r\"myswc\", \"Severe Weather Center\", tweet)\n", " # tweet = re.sub(r\"ithats\", \"that is\", tweet)\n", " # tweet = re.sub(r\"icouldsitinthismomentforever\", \"I could sit in this moment forever\", tweet)\n", " # tweet = re.sub(r\"FatLoss\", \"Fat Loss\", tweet)\n", " # tweet = re.sub(r\"02PM\", \"02 PM\", tweet)\n", " # tweet = re.sub(r\"MetroFmTalk\", \"Metro Fm Talk\", tweet)\n", " # tweet = re.sub(r\"Bstrd\", \"bastard\", tweet)\n", " # tweet = re.sub(r\"bldy\", \"bloody\", tweet)\n", " # tweet = re.sub(r\"MetrofmTalk\", \"Metro Fm Talk\", tweet)\n", " # tweet = re.sub(r\"terrorismturn\", \"terrorism turn\", tweet)\n", " # tweet = re.sub(r\"BBCNewsAsia\", \"BBC News Asia\", tweet)\n", " # tweet = re.sub(r\"BehindTheScenes\", \"Behind The Scenes\", tweet)\n", " # tweet = re.sub(r\"GeorgeTakei\", \"George Takei\", tweet)\n", " # tweet = re.sub(r\"WomensWeeklyMag\", \"Womens Weekly Magazine\", tweet)\n", " # tweet = re.sub(r\"SurvivorsGuidetoEarth\", \"Survivors Guide to Earth\", tweet)\n", " # tweet = re.sub(r\"incubusband\", \"incubus band\", tweet)\n", " # tweet = re.sub(r\"Babypicturethis\", \"Baby picture this\", tweet)\n", " # tweet = re.sub(r\"BombEffects\", \"Bomb Effects\", tweet)\n", " # tweet = re.sub(r\"win10\", \"Windows 10\", tweet)\n", " # tweet = re.sub(r\"idkidk\", \"I do not know I do not know\", tweet)\n", " # tweet = re.sub(r\"TheWalkingDead\", \"The Walking Dead\", tweet)\n", " # tweet = re.sub(r\"amyschumer\", \"Amy Schumer\", tweet)\n", " # tweet = re.sub(r\"crewlist\", \"crew list\", tweet)\n", " # tweet = re.sub(r\"Erdogans\", \"Erdogan\", tweet)\n", " # tweet = re.sub(r\"BBCLive\", \"BBC Live\", tweet)\n", " # tweet = re.sub(r\"TonyAbbottMHR\", \"Tony Abbott\", tweet)\n", " # tweet = re.sub(r\"paulmyerscough\", \"Paul Myerscough\", tweet)\n", " # tweet = re.sub(r\"georgegallagher\", \"George Gallagher\", tweet)\n", " # tweet = re.sub(r\"JimmieJohnson\", \"Jimmie Johnson\", tweet)\n", " # tweet = re.sub(r\"pctool\", \"pc tool\", tweet)\n", " # tweet = re.sub(r\"DoingHashtagsRight\", \"Doing Hashtags Right\", tweet)\n", " # tweet = re.sub(r\"ThrowbackThursday\", \"Throwback Thursday\", tweet)\n", " # tweet = re.sub(r\"SnowBackSunday\", \"Snowback Sunday\", tweet)\n", " # tweet = re.sub(r\"LakeEffect\", \"Lake Effect\", tweet)\n", " # tweet = re.sub(r\"RTphotographyUK\", \"Richard Thomas Photography UK\", tweet)\n", " # tweet = re.sub(r\"BigBang_CBS\", \"Big Bang CBS\", tweet)\n", " # tweet = re.sub(r\"writerslife\", \"writers life\", tweet)\n", " # tweet = re.sub(r\"NaturalBirth\", \"Natural Birth\", tweet)\n", " # tweet = re.sub(r\"UnusualWords\", \"Unusual Words\", tweet)\n", " # tweet = re.sub(r\"wizkhalifa\", \"Wiz Khalifa\", tweet)\n", " # tweet = re.sub(r\"acreativedc\", \"a creative DC\", tweet)\n", " # tweet = re.sub(r\"vscodc\", \"vsco DC\", tweet)\n", " # tweet = re.sub(r\"VSCOcam\", \"vsco camera\", tweet)\n", " # tweet = re.sub(r\"TheBEACHDC\", \"The beach DC\", tweet)\n", " # tweet = re.sub(r\"buildingmuseum\", \"building museum\", tweet)\n", " # tweet = re.sub(r\"WorldOil\", \"World Oil\", tweet)\n", " # tweet = re.sub(r\"redwedding\", \"red wedding\", tweet)\n", " # tweet = re.sub(r\"AmazingRaceCanada\", \"Amazing Race Canada\", tweet)\n", " # tweet = re.sub(r\"WakeUpAmerica\", \"Wake Up America\", tweet)\n", " # tweet = re.sub(r\"\\\\Allahuakbar\\\\\", \"Allahu Akbar\", tweet)\n", " # tweet = re.sub(r\"bleased\", \"blessed\", tweet)\n", " # tweet = re.sub(r\"nigeriantribune\", \"Nigerian Tribune\", tweet)\n", " # tweet = re.sub(r\"HIDEO_KOJIMA_EN\", \"Hideo Kojima\", tweet)\n", " # tweet = re.sub(r\"FusionFestival\", \"Fusion Festival\", tweet)\n", " # tweet = re.sub(r\"50Mixed\", \"50 Mixed\", tweet)\n", " # tweet = re.sub(r\"NoAgenda\", \"No Agenda\", tweet)\n", " # tweet = re.sub(r\"WhiteGenocide\", \"White Genocide\", tweet)\n", " # tweet = re.sub(r\"dirtylying\", \"dirty lying\", tweet)\n", " # tweet = re.sub(r\"SyrianRefugees\", \"Syrian Refugees\", tweet)\n", " # tweet = re.sub(r\"changetheworld\", \"change the world\", tweet)\n", " # tweet = re.sub(r\"Ebolacase\", \"Ebola case\", tweet)\n", " # tweet = re.sub(r\"mcgtech\", \"mcg technologies\", tweet)\n", " # tweet = re.sub(r\"withweapons\", \"with weapons\", tweet)\n", " # tweet = re.sub(r\"advancedwarfare\", \"advanced warfare\", tweet)\n", " # tweet = re.sub(r\"letsFootball\", \"let us Football\", tweet)\n", " # tweet = re.sub(r\"LateNiteMix\", \"late night mix\", tweet)\n", " # tweet = re.sub(r\"PhilCollinsFeed\", \"Phil Collins\", tweet)\n", " # tweet = re.sub(r\"RudyHavenstein\", \"Rudy Havenstein\", tweet)\n", " # tweet = re.sub(r\"22PM\", \"22 PM\", tweet)\n", " # tweet = re.sub(r\"54am\", \"54 AM\", tweet)\n", " # tweet = re.sub(r\"38am\", \"38 AM\", tweet)\n", " # tweet = re.sub(r\"OldFolkExplainStuff\", \"Old Folk Explain Stuff\", tweet)\n", " # tweet = re.sub(r\"BlacklivesMatter\", \"Black Lives Matter\", tweet)\n", " # tweet = re.sub(r\"InsaneLimits\", \"Insane Limits\", tweet)\n", " # tweet = re.sub(r\"youcantsitwithus\", \"you cannot sit with us\", tweet)\n", " # tweet = re.sub(r\"2k15\", \"2015\", tweet)\n", " # tweet = re.sub(r\"TheIran\", \"Iran\", tweet)\n", " # tweet = re.sub(r\"JimmyFallon\", \"Jimmy Fallon\", tweet)\n", " # tweet = re.sub(r\"AlbertBrooks\", \"Albert Brooks\", tweet)\n", " # tweet = re.sub(r\"defense_news\", \"defense news\", tweet)\n", " # tweet = re.sub(r\"nuclearrcSA\", \"Nuclear Risk Control Self Assessment\", tweet)\n", " # tweet = re.sub(r\"Auspol\", \"Australia Politics\", tweet)\n", " # tweet = re.sub(r\"NuclearPower\", \"Nuclear Power\", tweet)\n", " # tweet = re.sub(r\"WhiteTerrorism\", \"White Terrorism\", tweet)\n", " # tweet = re.sub(r\"truthfrequencyradio\", \"Truth Frequency Radio\", tweet)\n", " # tweet = re.sub(r\"ErasureIsNotEquality\", \"Erasure is not equality\", tweet)\n", " # tweet = re.sub(r\"ProBonoNews\", \"Pro Bono News\", tweet)\n", " # tweet = re.sub(r\"JakartaPost\", \"Jakarta Post\", tweet)\n", " # tweet = re.sub(r\"toopainful\", \"too painful\", tweet)\n", " # tweet = re.sub(r\"melindahaunton\", \"Melinda Haunton\", tweet)\n", " # tweet = re.sub(r\"NoNukes\", \"No Nukes\", tweet)\n", " # tweet = re.sub(r\"curryspcworld\", \"Currys PC World\", tweet)\n", " # tweet = re.sub(r\"ineedcake\", \"I need cake\", tweet)\n", " # tweet = re.sub(r\"blackforestgateau\", \"black forest gateau\", tweet)\n", " # tweet = re.sub(r\"BBCOne\", \"BBC One\", tweet)\n", " # tweet = re.sub(r\"AlexxPage\", \"Alex Page\", tweet)\n", " # tweet = re.sub(r\"jonathanserrie\", \"Jonathan Serrie\", tweet)\n", " # tweet = re.sub(r\"SocialJerkBlog\", \"Social Jerk Blog\", tweet)\n", " # tweet = re.sub(r\"ChelseaVPeretti\", \"Chelsea Peretti\", tweet)\n", " # tweet = re.sub(r\"irongiant\", \"iron giant\", tweet)\n", " # tweet = re.sub(r\"RonFunches\", \"Ron Funches\", tweet)\n", " # tweet = re.sub(r\"TimCook\", \"Tim Cook\", tweet)\n", " # tweet = re.sub(r\"sebastianstanisaliveandwell\", \"Sebastian Stan is alive and well\", tweet)\n", " # tweet = re.sub(r\"Madsummer\", \"Mad summer\", tweet)\n", " # tweet = re.sub(r\"NowYouKnow\", \"Now you know\", tweet)\n", " # tweet = re.sub(r\"concertphotography\", \"concert photography\", tweet)\n", " # tweet = re.sub(r\"TomLandry\", \"Tom Landry\", tweet)\n", " # tweet = re.sub(r\"showgirldayoff\", \"show girl day off\", tweet)\n", " # tweet = re.sub(r\"Yougslavia\", \"Yugoslavia\", tweet)\n", " # tweet = re.sub(r\"QuantumDataInformatics\", \"Quantum Data Informatics\", tweet)\n", " # tweet = re.sub(r\"FromTheDesk\", \"From The Desk\", tweet)\n", " # tweet = re.sub(r\"TheaterTrial\", \"Theater Trial\", tweet)\n", " # tweet = re.sub(r\"CatoInstitute\", \"Cato Institute\", tweet)\n", " # tweet = re.sub(r\"EmekaGift\", \"Emeka Gift\", tweet)\n", " # tweet = re.sub(r\"LetsBe_Rational\", \"Let us be rational\", tweet)\n", " # tweet = re.sub(r\"Cynicalreality\", \"Cynical reality\", tweet)\n", " # tweet = re.sub(r\"FredOlsenCruise\", \"Fred Olsen Cruise\", tweet)\n", " # tweet = re.sub(r\"NotSorry\", \"not sorry\", tweet)\n", " # tweet = re.sub(r\"UseYourWords\", \"use your words\", tweet)\n", " # tweet = re.sub(r\"WordoftheDay\", \"word of the day\", tweet)\n", " # tweet = re.sub(r\"Dictionarycom\", \"Dictionary.com\", tweet)\n", " # tweet = re.sub(r\"TheBrooklynLife\", \"The Brooklyn Life\", tweet)\n", " # tweet = re.sub(r\"jokethey\", \"joke they\", tweet)\n", " # tweet = re.sub(r\"nflweek1picks\", \"NFL week 1 picks\", tweet)\n", " # tweet = re.sub(r\"uiseful\", \"useful\", tweet)\n", " # tweet = re.sub(r\"JusticeDotOrg\", \"The American Association for Justice\", tweet)\n", " # tweet = re.sub(r\"autoaccidents\", \"auto accidents\", tweet)\n", " # tweet = re.sub(r\"SteveGursten\", \"Steve Gursten\", tweet)\n", " # tweet = re.sub(r\"MichiganAutoLaw\", \"Michigan Auto Law\", tweet)\n", " # tweet = re.sub(r\"birdgang\", \"bird gang\", tweet)\n", " # tweet = re.sub(r\"nflnetwork\", \"NFL Network\", tweet)\n", " # tweet = re.sub(r\"NYDNSports\", \"NY Daily News Sports\", tweet)\n", " # tweet = re.sub(r\"RVacchianoNYDN\", \"Ralph Vacchiano NY Daily News\", tweet)\n", " # tweet = re.sub(r\"EdmontonEsks\", \"Edmonton Eskimos\", tweet)\n", " # tweet = re.sub(r\"david_brelsford\", \"David Brelsford\", tweet)\n", " # tweet = re.sub(r\"TOI_India\", \"The Times of India\", tweet)\n", " # tweet = re.sub(r\"hegot\", \"he got\", tweet)\n", " # tweet = re.sub(r\"SkinsOn9\", \"Skins on 9\", tweet)\n", " # tweet = re.sub(r\"sothathappened\", \"so that happened\", tweet)\n", " # tweet = re.sub(r\"LCOutOfDoors\", \"LC Out Of Doors\", tweet)\n", " # tweet = re.sub(r\"NationFirst\", \"Nation First\", tweet)\n", " # tweet = re.sub(r\"IndiaToday\", \"India Today\", tweet)\n", " # tweet = re.sub(r\"HLPS\", \"helps\", tweet)\n", " # tweet = re.sub(r\"HOSTAGESTHROSW\", \"hostages throw\", tweet)\n", " # tweet = re.sub(r\"SNCTIONS\", \"sanctions\", tweet)\n", " # tweet = re.sub(r\"BidTime\", \"Bid Time\", tweet)\n", " # tweet = re.sub(r\"crunchysensible\", \"crunchy sensible\", tweet)\n", " # tweet = re.sub(r\"RandomActsOfRomance\", \"Random acts of romance\", tweet)\n", " # tweet = re.sub(r\"MomentsAtHill\", \"Moments at hill\", tweet)\n", " # tweet = re.sub(r\"eatshit\", \"eat shit\", tweet)\n", " # tweet = re.sub(r\"liveleakfun\", \"live leak fun\", tweet)\n", " # tweet = re.sub(r\"SahelNews\", \"Sahel News\", tweet)\n", " # tweet = re.sub(r\"abc7newsbayarea\", \"ABC 7 News Bay Area\", tweet)\n", " # tweet = re.sub(r\"facilitiesmanagement\", \"facilities management\", tweet)\n", " # tweet = re.sub(r\"facilitydude\", \"facility dude\", tweet)\n", " # tweet = re.sub(r\"CampLogistics\", \"Camp logistics\", tweet)\n", " # tweet = re.sub(r\"alaskapublic\", \"Alaska public\", tweet)\n", " # tweet = re.sub(r\"MarketResearch\", \"Market Research\", tweet)\n", " # tweet = re.sub(r\"AccuracyEsports\", \"Accuracy Esports\", tweet)\n", " # tweet = re.sub(r\"TheBodyShopAust\", \"The Body Shop Australia\", tweet)\n", " # tweet = re.sub(r\"yychail\", \"Calgary hail\", tweet)\n", " # tweet = re.sub(r\"yyctraffic\", \"Calgary traffic\", tweet)\n", " # tweet = re.sub(r\"eliotschool\", \"eliot school\", tweet)\n", " # tweet = re.sub(r\"TheBrokenCity\", \"The Broken City\", tweet)\n", " # tweet = re.sub(r\"OldsFireDept\", \"Olds Fire Department\", tweet)\n", " # tweet = re.sub(r\"RiverComplex\", \"River Complex\", tweet)\n", " # tweet = re.sub(r\"fieldworksmells\", \"field work smells\", tweet)\n", " # tweet = re.sub(r\"IranElection\", \"Iran Election\", tweet)\n", " # tweet = re.sub(r\"glowng\", \"glowing\", tweet)\n", " # tweet = re.sub(r\"kindlng\", \"kindling\", tweet)\n", " # tweet = re.sub(r\"riggd\", \"rigged\", tweet)\n", " # tweet = re.sub(r\"slownewsday\", \"slow news day\", tweet)\n", " # tweet = re.sub(r\"MyanmarFlood\", \"Myanmar Flood\", tweet)\n", " # tweet = re.sub(r\"abc7chicago\", \"ABC 7 Chicago\", tweet)\n", " # tweet = re.sub(r\"copolitics\", \"Colorado Politics\", tweet)\n", " # tweet = re.sub(r\"AdilGhumro\", \"Adil Ghumro\", tweet)\n", " # tweet = re.sub(r\"netbots\", \"net bots\", tweet)\n", " # tweet = re.sub(r\"byebyeroad\", \"bye bye road\", tweet)\n", " # tweet = re.sub(r\"massiveflooding\", \"massive flooding\", tweet)\n", " # tweet = re.sub(r\"EndofUS\", \"End of United States\", tweet)\n", " # tweet = re.sub(r\"35PM\", \"35 PM\", tweet)\n", " # tweet = re.sub(r\"greektheatrela\", \"Greek Theatre Los Angeles\", tweet)\n", " # tweet = re.sub(r\"76mins\", \"76 minutes\", tweet)\n", " # tweet = re.sub(r\"publicsafetyfirst\", \"public safety first\", tweet)\n", " # tweet = re.sub(r\"livesmatter\", \"lives matter\", tweet)\n", " # tweet = re.sub(r\"myhometown\", \"my hometown\", tweet)\n", " # tweet = re.sub(r\"tankerfire\", \"tanker fire\", tweet)\n", " # tweet = re.sub(r\"MEMORIALDAY\", \"memorial day\", tweet)\n", " # tweet = re.sub(r\"MEMORIAL_DAY\", \"memorial day\", tweet)\n", " # tweet = re.sub(r\"instaxbooty\", \"instagram booty\", tweet)\n", " # tweet = re.sub(r\"Jerusalem_Post\", \"Jerusalem Post\", tweet)\n", " # tweet = re.sub(r\"WayneRooney_INA\", \"Wayne Rooney\", tweet)\n", " # tweet = re.sub(r\"VirtualReality\", \"Virtual Reality\", tweet)\n", " # tweet = re.sub(r\"OculusRift\", \"Oculus Rift\", tweet)\n", " # tweet = re.sub(r\"OwenJones84\", \"Owen Jones\", tweet)\n", " # tweet = re.sub(r\"jeremycorbyn\", \"Jeremy Corbyn\", tweet)\n", " # tweet = re.sub(r\"paulrogers002\", \"Paul Rogers\", tweet)\n", " # tweet = re.sub(r\"mortalkombatx\", \"Mortal Kombat X\", tweet)\n", " # tweet = re.sub(r\"mortalkombat\", \"Mortal Kombat\", tweet)\n", " # tweet = re.sub(r\"FilipeCoelho92\", \"Filipe Coelho\", tweet)\n", " # tweet = re.sub(r\"OnlyQuakeNews\", \"Only Quake News\", tweet)\n", " # tweet = re.sub(r\"kostumes\", \"costumes\", tweet)\n", " # tweet = re.sub(r\"YEEESSSS\", \"yes\", tweet)\n", " # tweet = re.sub(r\"ToshikazuKatayama\", \"Toshikazu Katayama\", tweet)\n", " # tweet = re.sub(r\"IntlDevelopment\", \"Intl Development\", tweet)\n", " # tweet = re.sub(r\"ExtremeWeather\", \"Extreme Weather\", tweet)\n", " # tweet = re.sub(r\"WereNotGruberVoters\", \"We are not gruber voters\", tweet)\n", " # tweet = re.sub(r\"NewsThousands\", \"News Thousands\", tweet)\n", " # tweet = re.sub(r\"EdmundAdamus\", \"Edmund Adamus\", tweet)\n", " # tweet = re.sub(r\"EyewitnessWV\", \"Eye witness WV\", tweet)\n", " # tweet = re.sub(r\"PhiladelphiaMuseu\", \"Philadelphia Museum\", tweet)\n", " # tweet = re.sub(r\"DublinComicCon\", \"Dublin Comic Con\", tweet)\n", " # tweet = re.sub(r\"NicholasBrendon\", \"Nicholas Brendon\", tweet)\n", " # tweet = re.sub(r\"Alltheway80s\", \"All the way 80s\", tweet)\n", " # tweet = re.sub(r\"FromTheField\", \"From the field\", tweet)\n", " # tweet = re.sub(r\"NorthIowa\", \"North Iowa\", tweet)\n", " # tweet = re.sub(r\"WillowFire\", \"Willow Fire\", tweet)\n", " # tweet = re.sub(r\"MadRiverComplex\", \"Mad River Complex\", tweet)\n", " # tweet = re.sub(r\"feelingmanly\", \"feeling manly\", tweet)\n", " # tweet = re.sub(r\"stillnotoverit\", \"still not over it\", tweet)\n", " # tweet = re.sub(r\"FortitudeValley\", \"Fortitude Valley\", tweet)\n", " # tweet = re.sub(r\"CoastpowerlineTramTr\", \"Coast powerline\", tweet)\n", " # tweet = re.sub(r\"ServicesGold\", \"Services Gold\", tweet)\n", " # tweet = re.sub(r\"NewsbrokenEmergency\", \"News broken emergency\", tweet)\n", " # tweet = re.sub(r\"Evaucation\", \"evacuation\", tweet)\n", " # tweet = re.sub(r\"leaveevacuateexitbe\", \"leave evacuate exit be\", tweet)\n", " # tweet = re.sub(r\"P_EOPLE\", \"PEOPLE\", tweet)\n", " # tweet = re.sub(r\"Tubestrike\", \"tube strike\", tweet)\n", " # tweet = re.sub(r\"CLASS_SICK\", \"CLASS SICK\", tweet)\n", " # tweet = re.sub(r\"localplumber\", \"local plumber\", tweet)\n", " # tweet = re.sub(r\"awesomejobsiri\", \"awesome job siri\", tweet)\n", " # tweet = re.sub(r\"PayForItHow\", \"Pay for it how\", tweet)\n", " # tweet = re.sub(r\"ThisIsAfrica\", \"This is Africa\", tweet)\n", " # tweet = re.sub(r\"crimeairnetwork\", \"crime air network\", tweet)\n", " # tweet = re.sub(r\"KimAcheson\", \"Kim Acheson\", tweet)\n", " # tweet = re.sub(r\"cityofcalgary\", \"City of Calgary\", tweet)\n", " # tweet = re.sub(r\"prosyndicate\", \"pro syndicate\", tweet)\n", " # tweet = re.sub(r\"660NEWS\", \"660 NEWS\", tweet)\n", " # tweet = re.sub(r\"BusInsMagazine\", \"Business Insurance Magazine\", tweet)\n", " # tweet = re.sub(r\"wfocus\", \"focus\", tweet)\n", " # tweet = re.sub(r\"ShastaDam\", \"Shasta Dam\", tweet)\n", " # tweet = re.sub(r\"go2MarkFranco\", \"Mark Franco\", tweet)\n", " # tweet = re.sub(r\"StephGHinojosa\", \"Steph Hinojosa\", tweet)\n", " # tweet = re.sub(r\"Nashgrier\", \"Nash Grier\", tweet)\n", " # tweet = re.sub(r\"NashNewVideo\", \"Nash new video\", tweet)\n", " # tweet = re.sub(r\"IWouldntGetElectedBecause\", \"I would not get elected because\", tweet)\n", " # tweet = re.sub(r\"SHGames\", \"Sledgehammer Games\", tweet)\n", " # tweet = re.sub(r\"bedhair\", \"bed hair\", tweet)\n", " # tweet = re.sub(r\"JoelHeyman\", \"Joel Heyman\", tweet)\n", " # tweet = re.sub(r\"viaYouTube\", \"via YouTube\", tweet)\n", " \n", " # Urls\n", " tweet = re.sub(r\"https?:\\/\\/t.co\\/[A-Za-z0-9]+\", \"\", tweet)\n", " \n", " # Words with punctuations and special characters\n", " punctuations = '@#!?+&*[]-%.:/();$=><|{}^' + \"'`\"\n", " for p in punctuations:\n", " tweet = tweet.replace(p, f' {p} ')\n", " \n", " # ... and ..\n", " tweet = tweet.replace('...', ' ... ')\n", " if '...' not in tweet:\n", " tweet = tweet.replace('..', ' ... ') \n", " \n", " # # Acronyms\n", " # tweet = re.sub(r\"MH370\", \"Malaysia Airlines Flight 370\", tweet)\n", " # tweet = re.sub(r\"m̼sica\", \"music\", tweet)\n", " # tweet = re.sub(r\"okwx\", \"Oklahoma City Weather\", tweet)\n", " # tweet = re.sub(r\"arwx\", \"Arkansas Weather\", tweet) \n", " # tweet = re.sub(r\"gawx\", \"Georgia Weather\", tweet) \n", " # tweet = re.sub(r\"scwx\", \"South Carolina Weather\", tweet) \n", " # tweet = re.sub(r\"cawx\", \"California Weather\", tweet)\n", " # tweet = re.sub(r\"tnwx\", \"Tennessee Weather\", tweet)\n", " # tweet = re.sub(r\"azwx\", \"Arizona Weather\", tweet) \n", " # tweet = re.sub(r\"alwx\", \"Alabama Weather\", tweet)\n", " # tweet = re.sub(r\"wordpressdotcom\", \"wordpress\", tweet) \n", " # tweet = re.sub(r\"usNWSgov\", \"United States National Weather Service\", tweet)\n", " # tweet = re.sub(r\"Suruc\", \"Sanliurfa\", tweet) \n", " \n", " # Grouping same words without embeddings\n", " tweet = re.sub(r\"Bestnaijamade\", \"bestnaijamade\", tweet)\n", " tweet = re.sub(r\"SOUDELOR\", \"Soudelor\", tweet)\n", "\n", " tweet = re.sub(r'https?://\\S+|www\\.\\S+',r'',tweet)\n", " tweet = re.sub(r'<.*?>',r'',tweet)\n", " tweet = re.sub(\"[\"\n", " u\"\\U0001F600-\\U0001F64F\" # emoticons\n", " u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n", " u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n", " u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n", " u\"\\U00002702-\\U000027B0\"\n", " u\"\\U000024C2-\\U0001F251\"\n", " \"]+\",r'', tweet, flags=re.UNICODE)\n", " table=str.maketrans('','',string.punctuation)\n", " return tweet.translate(table).lower()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "B-H3tXT54cbp" }, "source": [ "df['clean_text'] = df['text'].apply(lambda x: clean(x))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "3ALnRX6ha-6R" }, "source": [ "df['Product_Label']=df['Product_Labels']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "PQ-B6WjjdqCB" }, "source": [ "df['Complaint']=df['clean_text']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "9uMfD8OFdn4S" }, "source": [ "df['Product']=df['Primary']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "XI2YwPFqZnpK", "colab": { "base_uri": "https://localhost:8080/", "height": 470 }, "outputId": "08b01782-578b-499c-9f6c-7fef13e6b0a3" }, "source": [ "df" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labelsclean_textProduct_LabelComplaintProduct
01.005030e+05threeSnake oil Salesmen are not gone. They have jus...3snake oil salesmen are not gone they have ju...3snake oil salesmen are not gone they have ju...three
11.009760e+05threeFact check: Gargling water with salt or vinega...3fact check gargling water with salt or vineg...3fact check gargling water with salt or vineg...three
21.011710e+05zero.. this is great advice. Also, stop fighting f...0this is great advice also stop fighting...0this is great advice also stop fighting...zero
31.017920e+05three14 people in Iran died from alcohol poisoning,...314 people in iran died from alcohol poisoning ...314 people in iran died from alcohol poisoning ...three
41.037400e+05oneTo be fair, China is not the only country alle...1to be fair china is not the only country alleg...1to be fair china is not the only country alleg...one
...........................
19651.230000e+18one@TheSecondRevol1 @VincentCrypt46 I have been s...1thesecondrevol1 vincentcrypt46 i have been...1thesecondrevol1 vincentcrypt46 i have been...one
19661.230000e+18oneIf anyone is aware of the 5G theory related to...1if anyone is aware of the 5g theory related to...1if anyone is aware of the 5g theory related to...one
19671.230000e+18one@ClydeLewis @ClydeLewis have you looked into 5...1clydelewis clydelewis have you looked into...1clydelewis clydelewis have you looked into...one
19681.230000e+18oneHow many have the “Virus” where there’s...1how many have the “virus” where there’s...1how many have the “virus” where there’s...one
19691.230000e+18oneSo why am I still so thoroughly stunned by the...1so why am i still so thoroughly stunned by the...1so why am i still so thoroughly stunned by the...one
\n", "

1970 rows × 8 columns

\n", "
" ], "text/plain": [ " ID ... Product\n", "0 1.005030e+05 ... three\n", "1 1.009760e+05 ... three\n", "2 1.011710e+05 ... zero\n", "3 1.017920e+05 ... three\n", "4 1.037400e+05 ... one\n", "... ... ... ...\n", "1965 1.230000e+18 ... one\n", "1966 1.230000e+18 ... one\n", "1967 1.230000e+18 ... one\n", "1968 1.230000e+18 ... one\n", "1969 1.230000e+18 ... one\n", "\n", "[1970 rows x 8 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "code", "metadata": { "id": "I49ZsYaYYOjk", "colab": { "base_uri": "https://localhost:8080/", "height": 359 }, "outputId": "6809d513-b368-484b-b856-acd1a7270581" }, "source": [ "df.head(10)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labelsclean_textProduct_LabelComplaintProduct
0100503.0threeSnake oil Salesmen are not gone. They have jus...3snake oil salesmen are not gone they have ju...3snake oil salesmen are not gone they have ju...three
1100976.0threeFact check: Gargling water with salt or vinega...3fact check gargling water with salt or vineg...3fact check gargling water with salt or vineg...three
2101171.0zero.. this is great advice. Also, stop fighting f...0this is great advice also stop fighting...0this is great advice also stop fighting...zero
3101792.0three14 people in Iran died from alcohol poisoning,...314 people in iran died from alcohol poisoning ...314 people in iran died from alcohol poisoning ...three
4103740.0oneTo be fair, China is not the only country alle...1to be fair china is not the only country alleg...1to be fair china is not the only country alleg...one
5108359.0threeCocaine does NOT cure #coronavirus: French gov...3cocaine does not cure coronavirus french g...3cocaine does not cure coronavirus french g...three
6108430.0three2008 Research paper demonstrating various esse...32008 research paper demonstrating various esse...32008 research paper demonstrating various esse...three
7110352.0three@SwerveChris @FakeNewsRehab @RudyGiuliani http...3swervechris fakenewsrehab rudygiuliani ...3swervechris fakenewsrehab rudygiuliani ...three
8110704.0three@wevarts @oliverdarcy @RudyGiuliani @charlieki...3wevarts oliverdarcy rudygiuliani charl...3wevarts oliverdarcy rudygiuliani charl...three
9110956.0zeroWasn't it great when our only health concerns ...0wasn t it great when our only health concerns...0wasn t it great when our only health concerns...zero
\n", "
" ], "text/plain": [ " ID Primary ... Complaint Product\n", "0 100503.0 three ... snake oil salesmen are not gone they have ju... three\n", "1 100976.0 three ... fact check gargling water with salt or vineg... three\n", "2 101171.0 zero ... this is great advice also stop fighting... zero\n", "3 101792.0 three ... 14 people in iran died from alcohol poisoning ... three\n", "4 103740.0 one ... to be fair china is not the only country alleg... one\n", "5 108359.0 three ... cocaine does not cure coronavirus french g... three\n", "6 108430.0 three ... 2008 research paper demonstrating various esse... three\n", "7 110352.0 three ... swervechris fakenewsrehab rudygiuliani ... three\n", "8 110704.0 three ... wevarts oliverdarcy rudygiuliani charl... three\n", "9 110956.0 zero ... wasn t it great when our only health concerns... zero\n", "\n", "[10 rows x 8 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 12 } ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "L24e7-s-YOjq", "colab": { "base_uri": "https://localhost:8080/", "height": 173 }, "outputId": "38032ad6-277f-4c0c-cc4a-be27f018679a" }, "source": [ "label_counts = pd.DataFrame(df['Product'].value_counts())\n", "label_counts" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Product
zero768
two468
one462
three272
\n", "
" ], "text/plain": [ " Product\n", "zero 768\n", "two 468\n", "one 462\n", "three 272" ] }, "metadata": { "tags": [] }, "execution_count": 13 } ] }, { "cell_type": "markdown", "metadata": { "id": "vYGlZYgtYOju" }, "source": [ "Here we create an array with the label names in the order they were numerically encoded. We use them later when plotting model performance data." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "oYKU_xolYOjw", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "12e838ef-7028-46ef-a5a0-7455f1aff224" }, "source": [ "label_values = list(label_counts.index)\n", "order = list(pd.DataFrame(df['Product_Label'].value_counts()).index)\n", "label_values = [l for _,l in sorted(zip(order, label_values))]\n", "\n", "label_values" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['zero', 'one', 'two', 'three']" ] }, "metadata": { "tags": [] }, "execution_count": 14 } ] }, { "cell_type": "markdown", "metadata": { "id": "_EWxTEOpYOj1" }, "source": [ "We need to create 2 arrays: one with the textual data, which is our feature data, and one with the numerically encoded labels, representing our target data." ] }, { "cell_type": "code", "metadata": { "id": "Fk1Rp-h6YOj2" }, "source": [ "texts = df['Complaint'].values\n", "labels = df['Product_Label'].values" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Rrds7zqdYOj7" }, "source": [ "BERT is a ‘heavy-weight’´model. This makes the training a very resource-intensive process, specially when we are fine-tuning for all model layers. To mitigate this, we can control the sequence length of our input text, which is given by the number of tokens in our input text, plus 2 special tokens to mark the beginning and ending of a text sequence." ] }, { "cell_type": "code", "metadata": { "id": "i38hSBadYOj8", "colab": { "base_uri": "https://localhost:8080/", "height": 51 }, "outputId": "d519bbca-2775-4426-f369-16e61f9ba259" }, "source": [ "text_lengths = [len(texts[i].split()) for i in range(len(texts))]\n", "print(min(text_lengths))\n", "print(max(text_lengths))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "1\n", "90\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "I6YdfO1HYOkB", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "e3bb61d4-2493-4eb0-9b46-e96cf5e00e3c" }, "source": [ "sum([1 for i in range(len(text_lengths)) if text_lengths[i] >= 55])" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "7" ] }, "metadata": { "tags": [] }, "execution_count": 17 } ] }, { "cell_type": "markdown", "metadata": { "id": "qeKHYy5sYOkH" }, "source": [ "Here we instantiate a BERT tokenizer and show an example of a tokenized text. " ] }, { "cell_type": "code", "metadata": { "scrolled": false, "id": "ibP4GASMYOkH", "colab": { "base_uri": "https://localhost:8080/", "height": 122 }, "outputId": "52baa0c7-8c3b-4c4a-a720-f1d481813ae4" }, "source": [ "from transformers import BertTokenizer\n", "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n", "\n", "\n", "\n", "print('Original Text: ', texts[0], '\\n')\n", "print('Tokenized Text: ', tokenizer.tokenize(texts[0]), '\\n')\n", "print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(texts[0])))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Original Text: snake oil salesmen are not gone they have just replaced the horse cart with twitter accounts and websites got an fda warning letter for advertising and selling the most powerful essential oil that can treat and defend against covid19 stay safe \n", "\n", "Tokenized Text: ['snake', 'oil', 'sales', '##men', 'are', 'not', 'gone', 'they', 'have', 'just', 'replaced', 'the', 'horse', 'cart', 'with', 'twitter', 'accounts', 'and', 'websites', 'got', 'an', 'fda', 'warning', 'letter', 'for', 'advertising', 'and', 'selling', 'the', 'most', 'powerful', 'essential', 'oil', 'that', 'can', 'treat', 'and', 'defend', 'against', 'co', '##vid', '##19', 'stay', 'safe'] \n", "\n", "Token IDs: [7488, 3514, 4341, 3549, 2024, 2025, 2908, 2027, 2031, 2074, 2999, 1996, 3586, 11122, 2007, 10474, 6115, 1998, 11744, 2288, 2019, 17473, 5432, 3661, 2005, 6475, 1998, 4855, 1996, 2087, 3928, 6827, 3514, 2008, 2064, 7438, 1998, 6985, 2114, 2522, 17258, 16147, 2994, 3647]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "E04gPWFoYOkM" }, "source": [ "We then tokenize and encode the entire dataset. In this process, we perform the following:\n", "- tokenize the text as shown above\n", "- encode it to the corresponding numeric values for each token.\n", "- truncate it to the maximum sequence length of 55.\n", "- pad the tokens positions greater than 55.\n", "- include the special token IDs to mark the beginning and end of each sequence." ] }, { "cell_type": "code", "metadata": { "id": "kiQEqbFFYOkN", "colab": { "base_uri": "https://localhost:8080/", "height": 952 }, "outputId": "d10bc592-6cb4-491e-ac8b-2a619265a38d" }, "source": [ "text_ids = [tokenizer.encode(text, max_length=55, pad_to_max_length=True) for text in texts]\n", "\n", "text_ids[0]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[101,\n", " 7488,\n", " 3514,\n", " 4341,\n", " 3549,\n", " 2024,\n", " 2025,\n", " 2908,\n", " 2027,\n", " 2031,\n", " 2074,\n", " 2999,\n", " 1996,\n", " 3586,\n", " 11122,\n", " 2007,\n", " 10474,\n", " 6115,\n", " 1998,\n", " 11744,\n", " 2288,\n", " 2019,\n", " 17473,\n", " 5432,\n", " 3661,\n", " 2005,\n", " 6475,\n", " 1998,\n", " 4855,\n", " 1996,\n", " 2087,\n", " 3928,\n", " 6827,\n", " 3514,\n", " 2008,\n", " 2064,\n", " 7438,\n", " 1998,\n", " 6985,\n", " 2114,\n", " 2522,\n", " 17258,\n", " 16147,\n", " 2994,\n", " 3647,\n", " 102,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0]" ] }, "metadata": { "tags": [] }, "execution_count": 20 } ] }, { "cell_type": "code", "metadata": { "id": "JldlWC2DYOkR", "colab": { "base_uri": "https://localhost:8080/", "height": 51 }, "outputId": "ac0b15ac-77ed-429c-95f9-5da141ba43e6" }, "source": [ "text_ids_lengths = [len(text_ids[i]) for i in range(len(text_ids))]\n", "print(min(text_ids_lengths))\n", "print(max(text_ids_lengths))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "55\n", "55\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "d8e1Az7iYOkW" }, "source": [ "To fine-tune our model, we need two inputs: one array of token IDs (created above) and one array of a corresponding binary mask, called attention mask in the BERT model specification. Each attention mask has the same length of the corresponding input sequence and has a 0 if the corresponding token is a pad token, or a 1 otherwise. " ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "M0LKIOSGYOkX", "colab": { "base_uri": "https://localhost:8080/", "height": 952 }, "outputId": "48253d8d-4966-4c82-c1c0-66a49c9793a5" }, "source": [ "att_masks = []\n", "for ids in text_ids:\n", " masks = [int(id > 0) for id in ids]\n", " att_masks.append(masks)\n", " \n", "att_masks[0]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0]" ] }, "metadata": { "tags": [] }, "execution_count": 22 } ] }, { "cell_type": "markdown", "metadata": { "id": "i3gBcJsNYOkb" }, "source": [ "Here we split the input and output arrays created before into train, validation, and test sets. We use 80% of the data for training, 10% for training validation, and 10% for final testing." ] }, { "cell_type": "code", "metadata": { "id": "t-xjCOmlYOkc" }, "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "train_x, test_val_x, train_y, test_val_y = train_test_split(text_ids, labels, random_state=11, test_size=0.2, stratify=labels)\n", "train_m, test_val_m = train_test_split(att_masks, random_state=11, test_size=0.2)\n", "\n", "test_x, val_x, test_y, val_y = train_test_split(test_val_x, test_val_y, random_state=11, test_size=0.5, stratify=test_val_y)\n", "test_m, val_m = train_test_split(test_val_m, random_state=11, test_size=0.5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "P0KwKWO2YOkg" }, "source": [ "We are working with the PyTorch artifacts in the transformers library, therefore we need our model input and output data as PyTorch tensors." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "3tWFKYqTYOkh", "colab": { "base_uri": "https://localhost:8080/", "height": 170 }, "outputId": "5eea6a57-b607-4340-ca99-ed94e368a137" }, "source": [ "import torch\n", "\n", "train_x = torch.tensor(train_x)\n", "test_x = torch.tensor(test_x)\n", "val_x = torch.tensor(val_x)\n", "train_y = torch.tensor(train_y)\n", "test_y = torch.tensor(test_y)\n", "val_y = torch.tensor(val_y)\n", "train_m = torch.tensor(train_m)\n", "test_m = torch.tensor(test_m)\n", "val_m = torch.tensor(val_m)\n", "\n", "print(train_x.shape)\n", "print(test_x.shape)\n", "print(val_x.shape)\n", "print(train_y.shape)\n", "print(test_y.shape)\n", "print(val_y.shape)\n", "print(train_m.shape)\n", "print(test_m.shape)\n", "print(val_m.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "torch.Size([1576, 55])\n", "torch.Size([197, 55])\n", "torch.Size([197, 55])\n", "torch.Size([1576])\n", "torch.Size([197])\n", "torch.Size([197])\n", "torch.Size([1576, 55])\n", "torch.Size([197, 55])\n", "torch.Size([197, 55])\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "2AZQKw8wYOkl" }, "source": [ "To feed data into the model for training, we use Pytorch’s Dataset, DataLoader, and Sampler. For feeding training data, which drives model weights updates, we use the RandomSampler. For feeding the validation data we can use the SequentialSampler." ] }, { "cell_type": "code", "metadata": { "id": "hA7DRz1gVzjh", "colab": { "base_uri": "https://localhost:8080/", "height": 357 }, "outputId": "c55bfb23-ee8a-4d55-c7a8-d35e92efbdf4" }, "source": [ " # Checking if GPU is available or not\n", " !nvidia-smi" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Mon Oct 19 17:09:35 2020 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 455.23.05 Driver Version: 418.67 CUDA Version: 10.1 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n", "| N/A 37C P8 10W / 70W | 0MiB / 15079MiB | 0% Default |\n", "| | | ERR! |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "| No running processes found |\n", "+-----------------------------------------------------------------------------+\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "KbF_AX6VYOkm" }, "source": [ "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", "\n", "batch_size = 8\n", "\n", "train_data = TensorDataset(train_x, train_m, train_y)\n", "train_sampler = RandomSampler(train_data)\n", "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", "\n", "val_data = TensorDataset(val_x, val_m, val_y)\n", "val_sampler = SequentialSampler(val_data)\n", "val_dataloader = DataLoader(val_data, sampler=val_sampler, batch_size=batch_size)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "KmZotr2qYOkr" }, "source": [ "Here we instantiate our model class. We use a compact version, that is trained through model distillation from a base BERT model and modified to include a classification layer at the output. This compact version has 6 transformer layers instead of 12 as in the original BERT model. Please see [here]( https://github.com/huggingface/transformers/tree/master/examples/distillation) for more details." ] }, { "cell_type": "code", "metadata": { "id": "KAErKspgYOks" }, "source": [ "from transformers import BertForSequenceClassification, BertConfig, AdamW\n", "\n", "num_labels = len(set(labels))\n", "\n", "model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=num_labels,\n", " output_attentions=False, output_hidden_states=False)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "t_KTIS4CzKtW", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "792e7c62-786d-4673-82f2-d6b8476847e6" }, "source": [ "num_labels" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "4" ] }, "metadata": { "tags": [] }, "execution_count": 28 } ] }, { "cell_type": "markdown", "metadata": { "id": "RhTSMq5EYOkw" }, "source": [ "BERT is a very large model. Unless you are freezing model weights in all layers but the classification layer, it is recommended to train it on a GPU." ] }, { "cell_type": "code", "metadata": { "id": "capa3O9cYOkw", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "f4079a1b-bf37-48cb-df55-86256a5858c4" }, "source": [ "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "print(device)\n", "model = model.to(device)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "cuda\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "1QqIWEEZYOk0" }, "source": [ "Here we print the model architecture and all model learnable parameters." ] }, { "cell_type": "code", "metadata": { "id": "X9eM3dZAYOk1", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "e4b140be-3b3c-4d42-8033-6554eeb5cae6" }, "source": [ "def count_parameters(model):\n", " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "\n", "print('Number of trainable parameters:', count_parameters(model), '\\n', model)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Number of trainable parameters: 109485316 \n", " BertForSequenceClassification(\n", " (bert): BertModel(\n", " (embeddings): BertEmbeddings(\n", " (word_embeddings): Embedding(30522, 768, padding_idx=0)\n", " (position_embeddings): Embedding(512, 768)\n", " (token_type_embeddings): Embedding(2, 768)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (encoder): BertEncoder(\n", " (layer): ModuleList(\n", " (0): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (1): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (2): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (3): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (4): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (5): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (6): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (7): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (8): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (9): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (10): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (11): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=768, out_features=768, bias=True)\n", " (key): Linear(in_features=768, out_features=768, bias=True)\n", " (value): Linear(in_features=768, out_features=768, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " )\n", " )\n", " (pooler): BertPooler(\n", " (dense): Linear(in_features=768, out_features=768, bias=True)\n", " (activation): Tanh()\n", " )\n", " )\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (classifier): Linear(in_features=768, out_features=4, bias=True)\n", ")\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "3KB4lAv8YOk7", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "c2ee9e7a-5426-4310-bfcb-b1168597ccb8" }, "source": [ "[n for n, p in model.named_parameters()]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['bert.embeddings.word_embeddings.weight',\n", " 'bert.embeddings.position_embeddings.weight',\n", " 'bert.embeddings.token_type_embeddings.weight',\n", " 'bert.embeddings.LayerNorm.weight',\n", " 'bert.embeddings.LayerNorm.bias',\n", " 'bert.encoder.layer.0.attention.self.query.weight',\n", " 'bert.encoder.layer.0.attention.self.query.bias',\n", " 'bert.encoder.layer.0.attention.self.key.weight',\n", " 'bert.encoder.layer.0.attention.self.key.bias',\n", " 'bert.encoder.layer.0.attention.self.value.weight',\n", " 'bert.encoder.layer.0.attention.self.value.bias',\n", " 'bert.encoder.layer.0.attention.output.dense.weight',\n", " 'bert.encoder.layer.0.attention.output.dense.bias',\n", " 'bert.encoder.layer.0.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.0.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.0.intermediate.dense.weight',\n", " 'bert.encoder.layer.0.intermediate.dense.bias',\n", " 'bert.encoder.layer.0.output.dense.weight',\n", " 'bert.encoder.layer.0.output.dense.bias',\n", " 'bert.encoder.layer.0.output.LayerNorm.weight',\n", " 'bert.encoder.layer.0.output.LayerNorm.bias',\n", " 'bert.encoder.layer.1.attention.self.query.weight',\n", " 'bert.encoder.layer.1.attention.self.query.bias',\n", " 'bert.encoder.layer.1.attention.self.key.weight',\n", " 'bert.encoder.layer.1.attention.self.key.bias',\n", " 'bert.encoder.layer.1.attention.self.value.weight',\n", " 'bert.encoder.layer.1.attention.self.value.bias',\n", " 'bert.encoder.layer.1.attention.output.dense.weight',\n", " 'bert.encoder.layer.1.attention.output.dense.bias',\n", " 'bert.encoder.layer.1.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.1.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.1.intermediate.dense.weight',\n", " 'bert.encoder.layer.1.intermediate.dense.bias',\n", " 'bert.encoder.layer.1.output.dense.weight',\n", " 'bert.encoder.layer.1.output.dense.bias',\n", " 'bert.encoder.layer.1.output.LayerNorm.weight',\n", " 'bert.encoder.layer.1.output.LayerNorm.bias',\n", " 'bert.encoder.layer.2.attention.self.query.weight',\n", " 'bert.encoder.layer.2.attention.self.query.bias',\n", " 'bert.encoder.layer.2.attention.self.key.weight',\n", " 'bert.encoder.layer.2.attention.self.key.bias',\n", " 'bert.encoder.layer.2.attention.self.value.weight',\n", " 'bert.encoder.layer.2.attention.self.value.bias',\n", " 'bert.encoder.layer.2.attention.output.dense.weight',\n", " 'bert.encoder.layer.2.attention.output.dense.bias',\n", " 'bert.encoder.layer.2.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.2.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.2.intermediate.dense.weight',\n", " 'bert.encoder.layer.2.intermediate.dense.bias',\n", " 'bert.encoder.layer.2.output.dense.weight',\n", " 'bert.encoder.layer.2.output.dense.bias',\n", " 'bert.encoder.layer.2.output.LayerNorm.weight',\n", " 'bert.encoder.layer.2.output.LayerNorm.bias',\n", " 'bert.encoder.layer.3.attention.self.query.weight',\n", " 'bert.encoder.layer.3.attention.self.query.bias',\n", " 'bert.encoder.layer.3.attention.self.key.weight',\n", " 'bert.encoder.layer.3.attention.self.key.bias',\n", " 'bert.encoder.layer.3.attention.self.value.weight',\n", " 'bert.encoder.layer.3.attention.self.value.bias',\n", " 'bert.encoder.layer.3.attention.output.dense.weight',\n", " 'bert.encoder.layer.3.attention.output.dense.bias',\n", " 'bert.encoder.layer.3.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.3.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.3.intermediate.dense.weight',\n", " 'bert.encoder.layer.3.intermediate.dense.bias',\n", " 'bert.encoder.layer.3.output.dense.weight',\n", " 'bert.encoder.layer.3.output.dense.bias',\n", " 'bert.encoder.layer.3.output.LayerNorm.weight',\n", " 'bert.encoder.layer.3.output.LayerNorm.bias',\n", " 'bert.encoder.layer.4.attention.self.query.weight',\n", " 'bert.encoder.layer.4.attention.self.query.bias',\n", " 'bert.encoder.layer.4.attention.self.key.weight',\n", " 'bert.encoder.layer.4.attention.self.key.bias',\n", " 'bert.encoder.layer.4.attention.self.value.weight',\n", " 'bert.encoder.layer.4.attention.self.value.bias',\n", " 'bert.encoder.layer.4.attention.output.dense.weight',\n", " 'bert.encoder.layer.4.attention.output.dense.bias',\n", " 'bert.encoder.layer.4.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.4.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.4.intermediate.dense.weight',\n", " 'bert.encoder.layer.4.intermediate.dense.bias',\n", " 'bert.encoder.layer.4.output.dense.weight',\n", " 'bert.encoder.layer.4.output.dense.bias',\n", " 'bert.encoder.layer.4.output.LayerNorm.weight',\n", " 'bert.encoder.layer.4.output.LayerNorm.bias',\n", " 'bert.encoder.layer.5.attention.self.query.weight',\n", " 'bert.encoder.layer.5.attention.self.query.bias',\n", " 'bert.encoder.layer.5.attention.self.key.weight',\n", " 'bert.encoder.layer.5.attention.self.key.bias',\n", " 'bert.encoder.layer.5.attention.self.value.weight',\n", " 'bert.encoder.layer.5.attention.self.value.bias',\n", " 'bert.encoder.layer.5.attention.output.dense.weight',\n", " 'bert.encoder.layer.5.attention.output.dense.bias',\n", " 'bert.encoder.layer.5.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.5.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.5.intermediate.dense.weight',\n", " 'bert.encoder.layer.5.intermediate.dense.bias',\n", " 'bert.encoder.layer.5.output.dense.weight',\n", " 'bert.encoder.layer.5.output.dense.bias',\n", " 'bert.encoder.layer.5.output.LayerNorm.weight',\n", " 'bert.encoder.layer.5.output.LayerNorm.bias',\n", " 'bert.encoder.layer.6.attention.self.query.weight',\n", " 'bert.encoder.layer.6.attention.self.query.bias',\n", " 'bert.encoder.layer.6.attention.self.key.weight',\n", " 'bert.encoder.layer.6.attention.self.key.bias',\n", " 'bert.encoder.layer.6.attention.self.value.weight',\n", " 'bert.encoder.layer.6.attention.self.value.bias',\n", " 'bert.encoder.layer.6.attention.output.dense.weight',\n", " 'bert.encoder.layer.6.attention.output.dense.bias',\n", " 'bert.encoder.layer.6.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.6.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.6.intermediate.dense.weight',\n", " 'bert.encoder.layer.6.intermediate.dense.bias',\n", " 'bert.encoder.layer.6.output.dense.weight',\n", " 'bert.encoder.layer.6.output.dense.bias',\n", " 'bert.encoder.layer.6.output.LayerNorm.weight',\n", " 'bert.encoder.layer.6.output.LayerNorm.bias',\n", " 'bert.encoder.layer.7.attention.self.query.weight',\n", " 'bert.encoder.layer.7.attention.self.query.bias',\n", " 'bert.encoder.layer.7.attention.self.key.weight',\n", " 'bert.encoder.layer.7.attention.self.key.bias',\n", " 'bert.encoder.layer.7.attention.self.value.weight',\n", " 'bert.encoder.layer.7.attention.self.value.bias',\n", " 'bert.encoder.layer.7.attention.output.dense.weight',\n", " 'bert.encoder.layer.7.attention.output.dense.bias',\n", " 'bert.encoder.layer.7.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.7.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.7.intermediate.dense.weight',\n", " 'bert.encoder.layer.7.intermediate.dense.bias',\n", " 'bert.encoder.layer.7.output.dense.weight',\n", " 'bert.encoder.layer.7.output.dense.bias',\n", " 'bert.encoder.layer.7.output.LayerNorm.weight',\n", " 'bert.encoder.layer.7.output.LayerNorm.bias',\n", " 'bert.encoder.layer.8.attention.self.query.weight',\n", " 'bert.encoder.layer.8.attention.self.query.bias',\n", " 'bert.encoder.layer.8.attention.self.key.weight',\n", " 'bert.encoder.layer.8.attention.self.key.bias',\n", " 'bert.encoder.layer.8.attention.self.value.weight',\n", " 'bert.encoder.layer.8.attention.self.value.bias',\n", " 'bert.encoder.layer.8.attention.output.dense.weight',\n", " 'bert.encoder.layer.8.attention.output.dense.bias',\n", " 'bert.encoder.layer.8.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.8.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.8.intermediate.dense.weight',\n", " 'bert.encoder.layer.8.intermediate.dense.bias',\n", " 'bert.encoder.layer.8.output.dense.weight',\n", " 'bert.encoder.layer.8.output.dense.bias',\n", " 'bert.encoder.layer.8.output.LayerNorm.weight',\n", " 'bert.encoder.layer.8.output.LayerNorm.bias',\n", " 'bert.encoder.layer.9.attention.self.query.weight',\n", " 'bert.encoder.layer.9.attention.self.query.bias',\n", " 'bert.encoder.layer.9.attention.self.key.weight',\n", " 'bert.encoder.layer.9.attention.self.key.bias',\n", " 'bert.encoder.layer.9.attention.self.value.weight',\n", " 'bert.encoder.layer.9.attention.self.value.bias',\n", " 'bert.encoder.layer.9.attention.output.dense.weight',\n", " 'bert.encoder.layer.9.attention.output.dense.bias',\n", " 'bert.encoder.layer.9.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.9.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.9.intermediate.dense.weight',\n", " 'bert.encoder.layer.9.intermediate.dense.bias',\n", " 'bert.encoder.layer.9.output.dense.weight',\n", " 'bert.encoder.layer.9.output.dense.bias',\n", " 'bert.encoder.layer.9.output.LayerNorm.weight',\n", " 'bert.encoder.layer.9.output.LayerNorm.bias',\n", " 'bert.encoder.layer.10.attention.self.query.weight',\n", " 'bert.encoder.layer.10.attention.self.query.bias',\n", " 'bert.encoder.layer.10.attention.self.key.weight',\n", " 'bert.encoder.layer.10.attention.self.key.bias',\n", " 'bert.encoder.layer.10.attention.self.value.weight',\n", " 'bert.encoder.layer.10.attention.self.value.bias',\n", " 'bert.encoder.layer.10.attention.output.dense.weight',\n", " 'bert.encoder.layer.10.attention.output.dense.bias',\n", " 'bert.encoder.layer.10.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.10.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.10.intermediate.dense.weight',\n", " 'bert.encoder.layer.10.intermediate.dense.bias',\n", " 'bert.encoder.layer.10.output.dense.weight',\n", " 'bert.encoder.layer.10.output.dense.bias',\n", " 'bert.encoder.layer.10.output.LayerNorm.weight',\n", " 'bert.encoder.layer.10.output.LayerNorm.bias',\n", " 'bert.encoder.layer.11.attention.self.query.weight',\n", " 'bert.encoder.layer.11.attention.self.query.bias',\n", " 'bert.encoder.layer.11.attention.self.key.weight',\n", " 'bert.encoder.layer.11.attention.self.key.bias',\n", " 'bert.encoder.layer.11.attention.self.value.weight',\n", " 'bert.encoder.layer.11.attention.self.value.bias',\n", " 'bert.encoder.layer.11.attention.output.dense.weight',\n", " 'bert.encoder.layer.11.attention.output.dense.bias',\n", " 'bert.encoder.layer.11.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.11.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.11.intermediate.dense.weight',\n", " 'bert.encoder.layer.11.intermediate.dense.bias',\n", " 'bert.encoder.layer.11.output.dense.weight',\n", " 'bert.encoder.layer.11.output.dense.bias',\n", " 'bert.encoder.layer.11.output.LayerNorm.weight',\n", " 'bert.encoder.layer.11.output.LayerNorm.bias',\n", " 'bert.pooler.dense.weight',\n", " 'bert.pooler.dense.bias',\n", " 'classifier.weight',\n", " 'classifier.bias']" ] }, "metadata": { "tags": [] }, "execution_count": 31 } ] }, { "cell_type": "markdown", "metadata": { "id": "zzqg1q-bYOk_" }, "source": [ "In the following 5 cells we define our PyTorch optimizer and corresponding parameters, learning rate scheduler, and the training loop for the fine-tuning procedure. We train for 10 epochs." ] }, { "cell_type": "code", "metadata": { "id": "fVdi9Q_NYOlA" }, "source": [ "learning_rate = 1e-5\n", "adam_epsilon = 1e-8\n", "\n", "no_decay = ['bias', 'LayerNorm.weight']\n", "optimizer_grouped_parameters = [\n", " {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.2},\n", " {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.0}\n", "]\n", "\n", "optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=adam_epsilon)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "yxg65ERWYOlG" }, "source": [ "from transformers import get_linear_schedule_with_warmup\n", "\n", "num_epochs = 10\n", "total_steps = len(train_dataloader) * num_epochs\n", "\n", "scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=total_steps)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "prV_sLUAYOlK" }, "source": [ "import time\n", "\n", "def epoch_time(start_time, end_time):\n", " elapsed_time = end_time - start_time\n", " elapsed_mins = int(elapsed_time / 60)\n", " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n", " return elapsed_mins, elapsed_secs" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "UdpuPsccYOlQ" }, "source": [ "import numpy as np\n", "import random\n", "\n", "seed_val = 11\n", "\n", "random.seed(seed_val)\n", "np.random.seed(seed_val)\n", "torch.manual_seed(seed_val)\n", "torch.cuda.manual_seed_all(seed_val)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "5gHBgvQrYOlU", "colab": { "base_uri": "https://localhost:8080/", "height": 782 }, "outputId": "5af0664d-239f-41f0-bff7-df7c1adb20dd" }, "source": [ "train_losses = []\n", "val_losses = []\n", "num_mb_train = len(train_dataloader)\n", "num_mb_val = len(val_dataloader)\n", "\n", "if num_mb_val == 0:\n", " num_mb_val = 1\n", "\n", "for n in range(num_epochs):\n", " train_loss = 0\n", " val_loss = 0\n", " start_time = time.time()\n", " \n", " for k, (mb_x, mb_m, mb_y) in enumerate(train_dataloader):\n", " optimizer.zero_grad()\n", " model.train()\n", " \n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " mb_y = mb_y.to(device)\n", " \n", " outputs = model(mb_x, attention_mask=mb_m, labels=mb_y)\n", " \n", " loss = outputs[0]\n", " #loss = model_loss(outputs[1], mb_y)\n", " loss.backward()\n", " torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n", " optimizer.step()\n", " scheduler.step()\n", " \n", " train_loss += loss.data / num_mb_train\n", " \n", " print (\"\\nTrain loss after itaration %i: %f\" % (n+1, train_loss))\n", " train_losses.append(train_loss.cpu())\n", " \n", " with torch.no_grad():\n", " model.eval()\n", " \n", " for k, (mb_x, mb_m, mb_y) in enumerate(val_dataloader):\n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " mb_y = mb_y.to(device)\n", " \n", " outputs = model(mb_x, attention_mask=mb_m, labels=mb_y)\n", " \n", " loss = outputs[0]\n", " #loss = model_loss(outputs[1], mb_y)\n", " \n", " val_loss += loss.data / num_mb_val\n", " \n", " print (\"Validation loss after itaration %i: %f\" % (n+1, val_loss))\n", " val_losses.append(val_loss.cpu())\n", " \n", " end_time = time.time()\n", " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", " print(f'Time: {epoch_mins}m {epoch_secs}s')" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.6/dist-packages/transformers/optimization.py:146: UserWarning: This overload of add_ is deprecated:\n", "\tadd_(Number alpha, Tensor other)\n", "Consider using one of the following signatures instead:\n", "\tadd_(Tensor other, *, Number alpha) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)\n", " exp_avg.mul_(beta1).add_(1.0 - beta1, grad)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\n", "Train loss after itaration 1: 1.264114\n", "Validation loss after itaration 1: 1.061481\n", "Time: 0m 24s\n", "\n", "Train loss after itaration 2: 0.837278\n", "Validation loss after itaration 2: 0.842118\n", "Time: 0m 24s\n", "\n", "Train loss after itaration 3: 0.623381\n", "Validation loss after itaration 3: 0.805889\n", "Time: 0m 24s\n", "\n", "Train loss after itaration 4: 0.467118\n", "Validation loss after itaration 4: 0.855168\n", "Time: 0m 24s\n", "\n", "Train loss after itaration 5: 0.312736\n", "Validation loss after itaration 5: 0.984129\n", "Time: 0m 24s\n", "\n", "Train loss after itaration 6: 0.233230\n", "Validation loss after itaration 6: 0.952618\n", "Time: 0m 25s\n", "\n", "Train loss after itaration 7: 0.161600\n", "Validation loss after itaration 7: 1.108665\n", "Time: 0m 25s\n", "\n", "Train loss after itaration 8: 0.121656\n", "Validation loss after itaration 8: 1.252152\n", "Time: 0m 25s\n", "\n", "Train loss after itaration 9: 0.087712\n", "Validation loss after itaration 9: 1.299764\n", "Time: 0m 25s\n", "\n", "Train loss after itaration 10: 0.072797\n", "Validation loss after itaration 10: 1.319620\n", "Time: 0m 25s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "pPSytOvwYOlY" }, "source": [ "After training, we can save the model and necessary configuration parameters, to recreate it later and use it to score the test data. Here we also save the losses computed from both training and validation data." ] }, { "cell_type": "code", "metadata": { "id": "U9tL0DGqYOlY" }, "source": [ "import pickle\n", "import os\n", "\n", "out_dir = './model'\n", "\n", "if not os.path.exists(out_dir):\n", " os.makedirs(out_dir)\n", " \n", "model_to_save = model.module if hasattr(model, 'module') else model\n", "model_to_save.save_pretrained(out_dir)\n", "tokenizer.save_pretrained(out_dir)\n", "\n", "with open(out_dir + '/train_losses.pkl', 'wb') as f:\n", " pickle.dump(train_losses, f)\n", " \n", "with open(out_dir + '/val_losses.pkl', 'wb') as f:\n", " pickle.dump(val_losses, f)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "YeSXfhL3YOlc" }, "source": [ "out_dir = './model'\n", "\n", "model = BertForSequenceClassification.from_pretrained(out_dir)\n", "\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "model = model.to(device)\n", "\n", "with open(out_dir + '/train_losses.pkl', 'rb') as f:\n", " train_losses = pickle.load(f)\n", " \n", "with open(out_dir + '/val_losses.pkl', 'rb') as f:\n", " val_losses = pickle.load(f)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "sKgIw7UPYOlf", "colab": { "base_uri": "https://localhost:8080/", "height": 282 }, "outputId": "8575213b-497b-4f20-bd20-4ab38ab22610" }, "source": [ "from matplotlib import pyplot as plt\n", "%matplotlib inline\n", "\n", "plt.figure()\n", "plt.plot(train_losses)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[]" ] }, "metadata": { "tags": [] }, "execution_count": 39 }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhV9b3v8fc3O3NIwpQAmUiYJxnDjAIOLRULtlYcKtYBKcWp9/S0p+d2uD3tuT2np7enrYpUUbQtHnFoKzjV1hYUEIGggExCGEMCJEwJkJDxd/9IpBEDCbCTtYfP63l4whqy18f9mE9+rL3W+plzDhERCX4RXgcQERH/UKGLiIQIFbqISIhQoYuIhAgVuohIiIj06sCdO3d22dnZXh1eRCQorV+//ohzLqWpbZ4VenZ2Nnl5eV4dXkQkKJnZvvNt0ykXEZEQoUIXEQkRKnQRkRChQhcRCREqdBGREKFCFxEJESp0EZEQEXSFvufIaf7t1S1U19Z5HUVEJKAEYaGf4plVe1myocjrKCIiASXoCn1y31T6dU3k8eX51NZpcg4RkU8EXaGbGfdP7sXuktO8teWQ13FERAJG0BU6wPVXdCOncwLzluWjKfREROoFZaH7IoxvTOzJlqIy3tlR4nUcEZGAEJSFDnDjsHS6Jccyb1m+11FERAJC0BZ6dGQEs6/qwbq9x1m755jXcUREPBe0hQ5w68gsOiVEa5QuIkILCt3MFppZsZltPs/2r5rZJjP7yMzeM7Mh/o/ZtLhoH/dMyOGdHSVsLixtq8OKiASklozQnwWmXGD7HmCic+4K4CfAk37I1WIzx3YnMSaSx5drlC4i4a3ZQnfOvQuc9yS1c+4959zxhsX3gQw/ZWuRpNgo7hzXnTc3HyK/+GRbHlpEJKD4+xz6vcCb59toZrPNLM/M8kpK/He54T3jc4iJjGD+8t1+e00RkWDjt0I3s8nUF/q/nG8f59yTzrlc51xuSkqTk1Zfkk7tYrhtVBavbCik4Fi5315XRCSY+KXQzWww8BQw3Tl31B+vebHuu7IHEQYLVmiULiLh6bIL3cyygD8CM51zOy4/0qVJax/Hl4dlsHhdAcUnz3gVQ0TEMy25bPF5YDXQ18wOmNm9ZjbHzOY07PJDoBPwuJltMLO8Vsx7QXMm9aSmto6nV+7xKoKIiGcim9vBOXdbM9tnAbP8lugy5HROYOrgNBat3sfcib1Ijo/yOpKISJsJ6jtFmzJ3Uk9OV9Xy29V7vY4iItKmQq7Q+3dL4pp+qSxctYfTlTVexxERaTMhV+gAcyf34kR5Nc+v3e91FBGRNhOShT6iewfG9ujEk+/uprKm1us4IiJtIiQLHeD+yb0oPlnJH9YXeh1FRKRNhGyhj+/ViSEZyfzmnV3U1NZ5HUdEpNWFbKGbGXMn92L/sXJe23TQ6zgiIq0uZAsd4Lr+XejTpR2PL8+nrk6TSYtIaAvpQo+IMOZO6sWOw6d4e9thr+OIiLSqkC50gBsGdyOzYxzzluXjnEbpIhK6Qr7QI30RzJnYk40HSlmV78mDIEVE2kTIFzrAV0ZkkJoYo8mkRSSkhUWhx0T6mH1VD1bvPsr6fceb/wYRkSAUFoUOcNuoLNrHRzFfk0mLSIgKm0JPiInk7nE5vL2tmG0Hy7yOIyLid2FT6ABfG9edhGgfjy/f5XUUERG/C6tCbx8fzR1ju/P6piL2HDntdRwREb8Kq0IHuHdCDpG+CJ54R6N0EQktYVfoqYmx3JKbyR8+OEDRiQqv44iI+E3YFTrA7Kt6UOdgwYrdXkcREfGbsCz0zI7x3Dg0nefX7ufoqUqv44iI+EVYFjrANyb1oLKmjmdW7fU6ioiIX4RtofdKTWTKwK78dvVeys5Uex1HROSyhW2hA8yd1IuTZ2r4/ep9XkcREblszRa6mS00s2Iz23ye7WZmj5hZvpltMrPh/o/ZOq7ISGZinxQWrtxDRZUmkxaR4NaSEfqzwJQLbP8C0Lvhz2xg/uXHajv3T+7F0dNVvLBuv9dRREQuS7OF7px7Fzh2gV2mA79z9d4H2ptZN38FbG2jcjoyMrsDT7y7m6oaTSYtIsHLH+fQ04GCRssHGtZ9hpnNNrM8M8srKSnxw6H9Y+7kXhwsPcMrHxZ6HUVE5JK16YeizrknnXO5zrnclJSUtjz0BU3qk8LAtCTmv7OLWk0mLSJByh+FXghkNlrOaFgXNMyM+yf3Ys+R07y5+aDXcURELok/Cn0pcGfD1S5jgFLnXNC14ucHdqVHSgLzlu3SZNIiEpRactni88BqoK+ZHTCze81sjpnNadjlDWA3kA8sAOa2WtpW5IswvjGxJ9sOlrHs42Kv44iIXLTI5nZwzt3WzHYH3O+3RB66cVg6v3p7J4/9PZ/JfVMxM68jiYi0WFjfKXquKF8EX5/Ygw/2n2DNngtdqSkiEnhU6OeYkZtJ53bRzFumyaRFJLio0M8RG+Xj3gk9WLHzCBsLTngdR0SkxVToTbhjTBZJsZE8vlyjdBEJHir0JiTGRnHXuGze2nKYnYdPeh1HRKRFVOjncdf4HOKifMxfrsmkRSQ4qNDPo2NCNLePzmLJxiL2Hy33Oo6ISLNU6Bdw35U98JnxxLsapYtI4FOhX0DX5FhuGpHBS3kHKC4743UcEZELUqE3Y87EHtTU1fHUyj1eRxERuSAVejO6d0rgi0PSWPT+Po6frvI6jojIeanQW2DupF6UV9Xy7Ht7vY4iInJeKvQW6Ns1kesGdOHZ9/ZyqrLG6zgiIk1SobfQ3Ek9Ka2o5n/W7PM6iohIk1ToLTQsqwPje3ViwYo9nKmu9TqOiMhnqNAvwv2TelFyspKX1h/wOoqIyGeo0C/C2J6dGJbVnife2UV1bZ3XcUREPkWFfhHMjPsn9eLA8Qpe3VjkdRwRkU9RoV+kq/ul0q9rIo8v30VdnSaTFpHAoUK/SBERxjcm9SS/+BR/2XrI6zgiImep0C/BDYPTyO4UzyN/y6eqRufSRSQwqNAvgS/C+Pbn+7H1YBkPL/6QGn1AKiIBQIV+iaYO7sYPbhjAm5sP8Z2XN+l8uoh4LtLrAMHs3gk5lFfW8Iu/7iAu2se/3zgIM/M6loiEqRaN0M1sipl9bGb5ZvbdJrZnmdkyM/vQzDaZ2fX+jxqYHri6F3Mm9uS5Nfv56RvbcE4jdRHxRrMjdDPzAfOA64ADwDozW+qc29pot+8DLzrn5pvZAOANILsV8gYcM+NfpvSlvKqGBSv2kBATyTev7eN1LBEJQy055TIKyHfO7QYws8XAdKBxoTsgqeHvyUBY3XVjZvzoiwMpr6rlV2/vJD7ax+yrenodS0TCTEsKPR0oaLR8ABh9zj4/Av5iZg8CCcC1Tb2Qmc0GZgNkZWVdbNaAFhFh/OymwVRU1/LTN7YTFx3JzDHdvY4lImHEX1e53AY865zLAK4Hfm9mn3lt59yTzrlc51xuSkqKnw4dOHwRxi9nDOWafqn84JXN/EEP8RKRNtSSQi8EMhstZzSsa+xe4EUA59xqIBbo7I+AwSY6MoJ5Xx3OuJ6d+PbLG3nzo4NeRxKRMNGSQl8H9DazHDOLBm4Flp6zz37gGgAz6099oZf4M2gwiY3yseDOXIZldeChxR+y7ONiryOJSBhottCdczXAA8BbwDbqr2bZYmY/NrNpDbt9C7jPzDYCzwN3uTC/fi8hJpKFd42kb9dE5vx+Pat3HfU6koiEOPOqd3Nzc11eXp4nx25Lx05XccsTqyk8UcGiWaMZntXB60giEsTMbL1zLrepbbr1v5V1TIjmuVmjSUmM4a6Fa9lSVOp1JBEJUSr0NpCaFMtzs0bTLiaSO59eS37xSa8jiUgIUqG3kYwO8SyaNRoz46tPrWH/0XKvI4lIiFGht6EeKe1YNGsUlTV13P7U+xwsrfA6koiEEBV6G+vXNYnf3TOKE+XVfHXBGkpOVnodSURChArdA4Mz2vPM3SMpKq1g5tNrOFFe5XUkEQkBKnSPjMzuyII7c9ldcpqvPbOOk2eqvY4kIkFOhe6hK3unMO+rw9lcWMq9v82joqrW60giEsRU6B67bkAXfnnLUNbtPcbXF62nskalLiKXRoUeAKYNSeNnXx7MuztKeOh5TTotIpdGhR4gZozM5P98cQBvbTnMP7+0UZNOi8hF0yTRAeTu8TmUV9Xy87c+Ji46kp9+SZNOi0jLqdADzP2Te3G6sobHl+8iPtrH96f2V6mLSIuo0APQtz/fl/KqWp5eWT/p9D9dp0mnRaR5KvQAZGb88IYBlFfV8Mjf6iednjNRk06LyIWp0ANURITxH18eTHlVLf/55nYSon3MHJvtdSwRCWAq9ADmizB+ectQzlTX8oMlW4iLjuQrIzK8jiUiAUqXLQa4KF8Ej90+nAm9OvOdlzfy+iZNOi0iTVOhB4HYKB9P3jmC4VkdeHjxh/x9+2GvI4lIAFKhB4n46EgW3j2S/t2SmLPoA97LP+J1JBEJMCr0IJIUG8Xv7hlFTqcEZv0uj/X7jnkdSUQCiAo9yHRIiOb3s0aRmhjDXc+sY3OhJp0WkXoq9CCUmhjLc/eNISk2iplPr2HHYU06LSIq9KCV3j6O52aNJtIXwc2/Wc2y7cVeRxIRj7Wo0M1sipl9bGb5Zvbd8+wzw8y2mtkWM/sf/8aUpmR3TuClr4+lW3Isdz+7jl/85WNq9ZRGkbDVbKGbmQ+YB3wBGADcZmYDztmnN/CvwHjn3EDgm62QVZqQ3TmBV+4fz80jMnj07/ncuXANR05p4mmRcNSSEfooIN85t9s5VwUsBqafs899wDzn3HEA55z+/d+GYqN8/PzmIfzXTYPJ23ucqY+sIG+vroARCTctKfR0oKDR8oGGdY31AfqY2Soze9/MpjT1QmY228zyzCyvpKTk0hLLec0Ymckf544jNsrHLU++z1MrduOcTsGIhAt/fSgaCfQGJgG3AQvMrP25OznnnnTO5TrnclNSUvx0aGlsYFoyrz44gWv7p/Lvr29jzqL1lJ2p9jqWiLSBlhR6IZDZaDmjYV1jB4Clzrlq59weYAf1BS8eSIqN4jd3jOB71/fn7W3FTHt0JVuLyryOJSKtrCWFvg7obWY5ZhYN3AosPWefV6gfnWNmnak/BbPbjznlIpkZ913Vg8Wzx1BeVcuXHl/Fi3kFzX+jiAStZgvdOVcDPAC8BWwDXnTObTGzH5vZtIbd3gKOmtlWYBnwbefc0dYKLS03Mrsjrz90JSO6d+A7L2/iOy9v5Ex1rdexRKQVmFcfmuXm5rq8vDxPjh2Oauscv/zrDh5blk//bknM/+pwsjsneB1LRC6Sma13zuU2tU13ioYJX4Txz5/vy8K7cik6UcEXH13Jnzcf8jqWiPiRCj3MXN2vC689OIGclATmLFrPT9/YRnVtndexRMQPVOhhKLNjPC/NGcvMMd158t3d3L7gfQ6XnfE6lohcJhV6mIqJ9PGTGwfx61uHsrmwjKmPrNCkGSJBToUe5qYPTWfpA+NJjovijqfXMG9ZPnV6wJdIUFKhC727JLL0gQlMHZzGz9/6mHt/u44T5VVexxKRi6RCFwASYiJ55Nah/GT6QFbmH2HqIyvZWHDC61gichFU6HKWmTFzbDYvzRkHwM2/Wc3vV+/VA75EgoQKXT5jaGZ7XntwAuN6deIHS7bwzRc2cLqyxutYItIMFbo0qUNCNAu/NpJvXdeHpRuLmD5vFfnFmrtUJJCp0OW8IiKMB6/pze/vGc3x01VMe2wVSzac+6BNEQkUKnRp1oTenXn9oSsZ0C2Jhxdv4IdLNlNZowd8iQQaFbq0SNfkWJ6fPYZZE3L43ep9zHjifQ4cL/c6log0okKXFovyRfD9GwbwmzuGs7v4FDc8upJlH2v6WJFAoUKXizZlUDeWPjiBrkmx3P3MOn7xl4+p1d2lIp5TocslyemcwCv3j+fmERk8+vd87ly4hiOnKr2OJRLWVOhyyWKjfPz85iH8102Dydt7nKmPrGDtnmNexxIJWyp0uWwzRmbyp7njiYvycduC95m/fJce8CXiARW6+MWAtCRefXACUwZ25Wd/3s6s3+Vx/LQe8CXSllTo4jeJsVE8dvsw/m3aQFbsLOGGR1fy4f7jXscSCRsqdPErM+Nr47J5ueEBXzOeWM3ClXv0gC+RNqBCl1YxJLM9bzx0JRP7pPDj17byjUUfUHam2utYIiFNhS6tJjk+igV35vK96/vz122HueGRlWwuLPU6lkjIUqFLqzIz7ruqBy/MHkNVTR1fnv8ei97fp1MwIq2gRYVuZlPM7GMzyzez715gv5vMzJlZrv8iSijIze7I6w9NYEyPTnz/lc08vHgDp/SMdRG/arbQzcwHzAO+AAwAbjOzAU3slwg8DKzxd0gJDZ3axfDsXSP558/14bVNRUx7bCXbD5V5HUskZLRkhD4KyHfO7XbOVQGLgelN7PcT4GfAGT/mkxATEWE8cHVvFs0aTVlFDTfOW8VLeQVexxIJCS0p9HSg8U/cgYZ1Z5nZcCDTOfe6H7NJCBvXszNvPDyBoZnt+fbLm/j2SxupqNIz1kUux2V/KGpmEcB/A99qwb6zzSzPzPJKSkou99AS5FITY3lu1hgevLoXL39wgBvnrWJXySmvY4kErZYUeiGQ2Wg5o2HdJxKBQcByM9sLjAGWNvXBqHPuSedcrnMuNyUl5dJTS8jwRRjf+lxfnr17FCWnKpn26EpNcydyiVpS6OuA3maWY2bRwK3A0k82OudKnXOdnXPZzrls4H1gmnMur1USS0ia2CeF1x+aQP+Gae6+96ePOFOtUzAiF6PZQnfO1QAPAG8B24AXnXNbzOzHZjattQNK+OiWHMfzs8fw9at68Nya/dw0/z32HT3tdSyRoGFe3eCRm5vr8vI0iJem/XXrYb714gacg5/fPJgpg7p5HUkkIJjZeudck/f66E5RCUjXDejC6w9dSY+UBOYs+oAfv7qVqpo6r2OJBDQVugSszI7xvDhnLHeNy2bhqj3MeGI1hScqvI4lErBU6BLQYiJ9/GjaQObdPpz84lNMfWQFf99+2OtYIgFJhS5BYergbrz64AS6Jcdxz7N5/OzP26mp1SkYkcZU6BI0cjon8Ke547htVBbzl+/i9gVrOFymJ02IfEKFLkElNsrHf3z5Cn55yxA+Kizl+l+vYMVO3XUsAip0CVJfGpbB0gfG0zEhmjsXruWXf91BbZ2esS7hTYUuQat3l0SWPDCeLw1N59d/28mdC9dQcrLS61ginlGhS1CLj47kFzOG8LObriBv73GmPrKCxWv3a/IMCUu6U1RCxraDZfzTixvZdrCM+GgfXxycxi2jMhmW2R4z8zqeiF9c6E5RFbqEFOccH+w/wQvr9vPqxoNUVNfSp0s7bhmZxZeGpdMxIdrriCKXRYUuYenkmWpe23SQF9YVsKHgBNG+CD43sAu3jsxiXM9ORERo1C7BR4UuYW/7oTJeWFfAnz4s5ER5NRkd4piRm8nNuRl0S47zOp5Ii6nQRRqcqa7lL1sP88K6/azKP0qE1T+L/ZaRWVzTP5Uon64TkMCmQhdpwv6j5by0voAX8wo4XFZJ53bR3DQig1tyM+mR0s7reCJNUqGLXEBNbR3v7ixh8doC/ra9mNo6x6jsjtwyMpPrr+hGXLTP64giZ6nQRVqo+OQZ/rC+kBfW7Wfv0XISYyKZPiyNW0dmMSg92et4Iip0kYvlnGPNnmO8uK6A1z86SGVNHQPTkrh1ZCbThqaTHBfldUQJUyp0kctQWlHN0g2FPL+2gK0Hy4iJjGDqFd2YMTKT0TkdddOStCkVuoifbC4sZfG6/Sz5sIiTlTXkdE5gRm4mN41IJzUx1ut4EgZU6CJ+VlFVyxsf1d+0tHbvMXwRxjX9Url1VCZX9U4hUpc/SitRoYu0ol0lp3gxr4A/rD/AkVNVdE2K5SsjMvjS8HR66vJH8TMVukgbqK6t42/binlh3X7e2VFCnYNB6UlMH5LOF4ek0TVZp2Tk8qnQRdrY4bIzvLbpIEs3FLLxQClmMDqnI9OHpvOFQV1pH6+HhMmlUaGLeGjPkdMs3VDEkg2F7D5ymiifMbFPKtOHpnFt/y66cUkuymUXuplNAX4N+ICnnHP/ec72fwJmATVACXCPc27fhV5ThS7hxjnHlqIylmwoZOnGIg6XVRIf7ePzA7sybWgaE3p11rNkpFmXVehm5gN2ANcBB4B1wG3Oua2N9pkMrHHOlZvZN4BJzrlbLvS6KnQJZ7V1jrV7jrF0YyGvbzpI2ZkaOiZEM/WKbkwfmsbwrA56vK806XILfSzwI+fc5xuW/xXAOfcf59l/GPCYc278hV5XhS5Sr7Kmlnd3HGHJhkLe3naYM9V1pLePY9rQNKYPTaNf1ySvI0oAuVChR7bg+9OBgkbLB4DRF9j/XuDN8wSZDcwGyMrKasGhRUJfTKSP6wZ04boBXThVWcNftx5iyYYinnx3N/OX76Jvl0SmDU1j2pA0MjvGex1XAlhLRuhfAaY452Y1LM8ERjvnHmhi3zuAB4CJzrkLTr+uEbrIhR09VckbHx1kyYYi8vYdB2BE9w5MH5rG9Vd0o3O7GI8Tihcud4ReCGQ2Ws5oWHfuQa4FvkcLylxEmtepXQwzx2Yzc2w2BcfKeXVTEUs3FPHDJVv4t1e3MqFXZ6YPTeNzA7vSLqYlP8oS6loyQo+k/kPRa6gv8nXA7c65LY32GQa8TP1IfmdLDqwRusil2X6orOEyyCIKT1QQExnBtQO6MH1IGhP7phATqcsgQ5k/Llu8HvgV9ZctLnTO/V8z+zGQ55xbamZvA1cABxu+Zb9zbtqFXlOFLnJ5nHN8sP84SzYU8fqmgxw9XUVSbCTXX9GNaUPSGN2jEz5dKRNydGORSIirrq1jVf4Rlm4o4q0thzhdVUtqYgxfGNSVYVkdGJSeTE7nBBV8CFChi4SRiqpa/rb9MEs2FPHujhIqa+oAiI/2MaBbEoPSkxmYVv+1d2o7PRkyyFzuh6IiEkTion3cMDiNGwanUVNbR37JKTYXlrG5sJQtRaW8mFdAeVUtADGREfTrlsSghoIflJZMn67tdB4+SGmELhJmausce46cZktRKZsLS+vLvqiUk2dqAIjyGX26JDIoLZlB6UkMTE+mf9ckPXMmQOiUi4hckHOOgmMVfFRYyuazRV/K8fJqACIMeqW2ayj5+j8D0pJ0uaQHdMpFRC7IzMjqFE9Wp3imDu4G1Jf8wdIzZ8t9c1EZK/OP8McPCxu+B3I6JTAwPZlBaUlckZ7MwLRkkuM1gbZXVOgi0iQzI619HGnt4/jcwK5n1xeXnWFLUVlDyZfywb7jvLqx6Oz2zI5xZ0fyn3z4qrta24YKXUQuSmpSLKlJsUzul3p23bHTVQ3n5MvOnrJ5c/Ohf3xPYgx9uiTSu0u7+q+p7ejdJZHkOI3m/UmFLiKXrWNCNFf2TuHK3iln15VWVLO1qIwtRaVsPVhGfvEpFq8toKK69uw+jYu+d2oifbqo6C+HCl1EWkVyXBRje3ZibM9OZ9fV1TkKT1Sws/gkOw6fYufhU+wsPtmyok9N1Pn5ZqjQRaTNREQYmR3jyewYz9X9upxdf7FF3yu1/tSNiv7TVOgi4rmWFP3Ow6fqy774JC+su3DR9+7Sjj5hWPQqdBEJWBdT9PnFJz91FyzUF/0/Ttskkt4hjg7xUXSIj6Z9fBTtYiIxC53n26jQRSToNFf0+cWn2HH45HmL/hOREUb7+Cjax0fTIT6K5Lj6rx0S6gu/fcNy+/hoOiTUL7ePjyI2KjDvmlWhi0jIaFz0jS+r/KToD5Wd4fjpKk5UVHOivIrj5dWcKP/k71UcOF7O5sJqjpdXnX2oWVPionz1vwAafhF0iI8mOT6q0eg/mvZxUfW/BOKj67fHRbX60y5V6CIS8hoXfUtVVNVyvLyqUeFXc6Kifvn46frl0or6r9sOlVFaXs2Jimpq687/OJWk2Eg6JEQzc0x3Zl3Zwx//aZ+iQhcRaUJctI+46Po7ZVuqrs5xsrKGEw2/CD7zC6Hha2vdOatCFxHxk4gIIzkuiuS4KLp3an5/vx+/7Q8pIiKtQYUuIhIiVOgiIiFChS4iEiJU6CIiIUKFLiISIlToIiIhQoUuIhIizLnz36baqgc2KwH2XeK3dwaO+DFOsNP78Wl6P/5B78WnhcL70d05l9LUBs8K/XKYWZ5zLtfrHIFC78en6f34B70Xnxbq74dOuYiIhAgVuohIiAjWQn/S6wABRu/Hp+n9+Ae9F58W0u9HUJ5DFxGRzwrWEbqIiJxDhS4iEiKCrtDNbIqZfWxm+Wb2Xa/zeMnMMs1smZltNbMtZvaw15m8ZmY+M/vQzF7zOovXzKy9mb1sZtvNbJuZjfU6k1fM7H81/IxsNrPnzSzW60ytIagK3cx8wDzgC8AA4DYzG+BtKk/VAN9yzg0AxgD3h/n7AfAwsM3rEAHi18CfnXP9gCGE6ftiZunAQ0Cuc24Q4ANu9TZV6wiqQgdGAfnOud3OuSpgMTDd40yecc4ddM590PD3k9T/wKZ7m8o7ZpYBTAWe8jqL18wsGbgKeBrAOVflnDvhbSpPRQJxZhYJxANFHudpFcFW6OlAQaPlA4RxgTVmZtnAMGCNt0k89SvgO0Cd10ECQA5QAjzTcArqKTNL8DqUF5xzhcD/A/YDB4FS59xfvE3VOoKt0KUJZtYO+APwTedcmdd5vGBmNwDFzrn1XmcJEJHAcGC+c24YcBoIy8+czKwD9f+SzwHSgAQzu8PbVK0j2Aq9EMhstJzRsC5smVkU9WX+nHPuj17n8dB4YJqZ7aX+VNzVZrbI20ieOgAccM598i+2l6kv+HB0LbDHOVfinKsG/giM8zhTqwi2Ql8H9DazHDOLpv6DjaUeZ/KMmRn150i3Oef+2+s8XnLO/atzLsM5l039/xd/d86F5CisJZxzh4ACM+vbsOoaYKuHkby0HxhjZvENPzPXEKIfEEd6HeBiOOdqzOwB4C3qP6le6Jzb4nEsL40HZgIfmdmGhnX/2zn3hoeZJHA8CDzXMPjZDdztcR5POOfWmNnLwAfUXxn2ISH6CADd+i8iEiKC7ZSLiJUJcLAAAAAqSURBVIichwpdRCREqNBFREKECl1EJESo0EVEQoQKXUQkRKjQRURCxP8HwfAyWj+d14cAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "jHgsB4duYOlo" }, "source": [ "After instantiating a trained model, we can then score the test data and compute its accuracy. We then print the classification report and plot a confusion matrix.\n" ] }, { "cell_type": "code", "metadata": { "id": "M2J_opCYYOlp" }, "source": [ "batch_size = 8\n", "\n", "test_data = TensorDataset(test_x, test_m)\n", "test_sampler = SequentialSampler(test_data)\n", "test_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=batch_size)\n", "\n", "outputs = []\n", "with torch.no_grad():\n", " model.eval()\n", " for k, (mb_x, mb_m) in enumerate(test_dataloader):\n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " output = model(mb_x, attention_mask=mb_m)\n", " outputs.append(output[0].to('cpu'))\n", "\n", "outputs = torch.cat(outputs)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "0Kn87Vq3YOls" }, "source": [ "_, predicted_values = torch.max(outputs, 1)\n", "predicted_values = predicted_values.numpy()\n", "true_values = test_y.numpy()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "pM9-0kkOYOlx", "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "outputId": "9b1bbddc-e0dc-42b2-81ed-ef92ce65fc0c" }, "source": [ "test_accuracy = np.sum(predicted_values == true_values) / len(true_values)\n", "print (\"Test Accuracy:\", test_accuracy)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Test Accuracy: 0.6802030456852792\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "10TOvX8VwZ_Y" }, "source": [ "import warnings\n", "warnings.filterwarnings('always')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "gjUCjpI_YOl0", "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "outputId": "ae85f4aa-7aab-4863-e1d7-3fecdbab8fd9" }, "source": [ "from sklearn.metrics import classification_report\n", "\n", "print(classification_report(true_values, predicted_values, target_names=[str(l) for l in label_values]))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " zero 0.73 0.66 0.69 77\n", " one 0.87 0.87 0.87 46\n", " two 0.54 0.64 0.58 47\n", " three 0.52 0.48 0.50 27\n", "\n", " accuracy 0.68 197\n", " macro avg 0.66 0.66 0.66 197\n", "weighted avg 0.69 0.68 0.68 197\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "iL040G8XXPEA" }, "source": [ "Analysis" ] }, { "cell_type": "code", "metadata": { "id": "47kL69LFYOl3" }, "source": [ "import itertools\n", "\n", "# plot confusion matrix\n", "# code borrowed from scikit-learn.org\n", "def plot_confusion_matrix(cm, classes,\n", " normalize=False,\n", " title='Confusion matrix',\n", " cmap=plt.cm.Blues):\n", " \"\"\"\n", " This function prints and plots the confusion matrix.\n", " Normalization can be applied by setting `normalize=True`.\n", " \"\"\"\n", " if normalize:\n", " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", " print(\"Normalized confusion matrix\")\n", " else:\n", " print('Confusion matrix, without normalization')\n", "\n", " print(cm)\n", "\n", " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", " plt.title(title)\n", " plt.colorbar()\n", " tick_marks = np.arange(len(classes))\n", " plt.xticks(tick_marks, classes, rotation=45)\n", " plt.yticks(tick_marks, classes)\n", "\n", " fmt = '.2f' if normalize else 'd'\n", " thresh = cm.max() / 2.\n", " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", " plt.text(j, i, format(cm[i, j], fmt),\n", " horizontalalignment=\"center\",\n", " color=\"white\" if cm[i, j] > thresh else \"black\")\n", "\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "JMx8Rp6AYOl6" }, "source": [ "from sklearn.metrics import classification_report, confusion_matrix\n", "\n", "cm_test = confusion_matrix(true_values, predicted_values)\n", "\n", "np.set_printoptions(precision=2)\n", "\n", "plt.figure(figsize=(6,6))\n", "plot_confusion_matrix(cm_test, classes=label_values, title='Confusion Matrix - Test Dataset')\n", "plt.figure(figsize=(6,6))\n", "plot_confusion_matrix(cm_test, classes=label_values, title='Confusion Matrix - Test Dataset', normalize=True)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "hqRuaxZ1W5KP" }, "source": [ "# END" ] }, { "cell_type": "markdown", "metadata": { "id": "tsxIxgfSwwS7" }, "source": [ "# BERT LARGE" ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "AiWDZVXMswX4", "colab": { "base_uri": "https://localhost:8080/", "height": 688 }, "outputId": "9d1ec937-bdea-40e7-9f68-c9bab7b5804d" }, "source": [ "!pip install transformers==2.3.0" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Collecting transformers==2.3.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/50/10/aeefced99c8a59d828a92cc11d213e2743212d3641c87c82d61b035a7d5c/transformers-2.3.0-py3-none-any.whl (447kB)\n", "\u001b[K |████████████████████████████████| 450kB 4.4MB/s \n", "\u001b[?25hCollecting sacremoses\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n", "\u001b[K |████████████████████████████████| 890kB 11.0MB/s \n", "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (4.41.1)\n", "Collecting sentencepiece\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n", "\u001b[K |████████████████████████████████| 1.1MB 14.1MB/s \n", "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (2.23.0)\n", "Collecting boto3\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e2/dd/8ed2e36ab9e8eb006e0abd298e0d6c4ff89bf00caafd3cf21a58b700e061/boto3-1.16.0-py2.py3-none-any.whl (129kB)\n", "\u001b[K |████████████████████████████████| 133kB 27.7MB/s \n", "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (2019.12.20)\n", "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers==2.3.0) (1.18.5)\n", "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (1.15.0)\n", "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (7.1.2)\n", "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.3.0) (0.16.0)\n", "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (3.0.4)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (2020.6.20)\n", "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (2.10)\n", "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.3.0) (1.24.3)\n", "Collecting s3transfer<0.4.0,>=0.3.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)\n", "\u001b[K |████████████████████████████████| 71kB 8.9MB/s \n", "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", "Collecting botocore<1.20.0,>=1.19.0\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7a/38/317bec1175e74b817a51ca806f3e5822f340bb3b344e8693425122499f4f/botocore-1.19.0-py2.py3-none-any.whl (6.7MB)\n", "\u001b[K |████████████████████████████████| 6.7MB 25.4MB/s \n", "\u001b[?25hRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.20.0,>=1.19.0->boto3->transformers==2.3.0) (2.8.1)\n", "Building wheels for collected packages: sacremoses\n", " Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=73e8d5c792aef8cae5fa6adc8a094357388e19ea1ec16e2b54eb8808b677b18a\n", " Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n", "Successfully built sacremoses\n", "\u001b[31mERROR: botocore 1.19.0 has requirement urllib3<1.26,>=1.25.4; python_version != \"3.4\", but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", "Installing collected packages: sacremoses, sentencepiece, jmespath, botocore, s3transfer, boto3, transformers\n", "Successfully installed boto3-1.16.0 botocore-1.19.0 jmespath-0.10.0 s3transfer-0.3.3 sacremoses-0.0.43 sentencepiece-0.1.91 transformers-2.3.0\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "AY09bJPPswYI" }, "source": [ "We then load and inspect the dataset we had previously prepared. In order to train faster, we use the sampled version, which contains 10% of the original prepared dataset." ] }, { "cell_type": "code", "metadata": { "id": "kTxKB5lbs-Qn", "colab": { "base_uri": "https://localhost:8080/", "height": 184 }, "outputId": "e7e4295b-cf12-494e-cfa2-bac91f3d3db2" }, "source": [ "pip install urllib3==1.25.4" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Collecting urllib3==1.25.4\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/91/0d/7777358f672a14b7ae0dfcd29f949f409f913e0578190d6bfa68eb55864b/urllib3-1.25.4-py2.py3-none-any.whl (125kB)\n", "\r\u001b[K |██▋ | 10kB 15.5MB/s eta 0:00:01\r\u001b[K |█████▏ | 20kB 3.5MB/s eta 0:00:01\r\u001b[K |███████▉ | 30kB 4.6MB/s eta 0:00:01\r\u001b[K |██████████▍ | 40kB 4.9MB/s eta 0:00:01\r\u001b[K |█████████████ | 51kB 4.1MB/s eta 0:00:01\r\u001b[K |███████████████▋ | 61kB 4.6MB/s eta 0:00:01\r\u001b[K |██████████████████▎ | 71kB 5.0MB/s eta 0:00:01\r\u001b[K |████████████████████▉ | 81kB 5.5MB/s eta 0:00:01\r\u001b[K |███████████████████████▌ | 92kB 5.8MB/s eta 0:00:01\r\u001b[K |██████████████████████████ | 102kB 5.6MB/s eta 0:00:01\r\u001b[K |████████████████████████████▊ | 112kB 5.6MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▎| 122kB 5.6MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 133kB 5.6MB/s \n", "\u001b[31mERROR: kaggle 1.5.8 has requirement urllib3<1.25,>=1.21.1, but you'll have urllib3 1.25.4 which is incompatible.\u001b[0m\n", "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n", "\u001b[?25hInstalling collected packages: urllib3\n", " Found existing installation: urllib3 1.24.3\n", " Uninstalling urllib3-1.24.3:\n", " Successfully uninstalled urllib3-1.24.3\n", "Successfully installed urllib3-1.25.4\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "5bPpaVt0swYL" }, "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('/content/result_31.csv', encoding='latin-1')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Jk-EC2iJswYV", "colab": { "base_uri": "https://localhost:8080/", "height": 33 }, "outputId": "4c01bbc4-4419-4238-ff54-abf2b4cb4f48" }, "source": [ "df.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1970, 4)" ] }, "metadata": { "tags": [] }, "execution_count": 4 } ] }, { "cell_type": "code", "metadata": { "id": "j8QBE8xuswYm", "colab": { "base_uri": "https://localhost:8080/", "height": 392 }, "outputId": "a17cc99f-708e-41b5-91f1-b56ad98ddc3f" }, "source": [ "df" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labels
01.005030e+05threeSnake oil Salesmen are not gone. They have jus...3
11.009760e+05threeFact check: Gargling water with salt or vinega...3
21.011710e+05zero.. this is great advice. Also, stop fighting f...0
31.017920e+05three14 people in Iran died from alcohol poisoning,...3
41.037400e+05oneTo be fair, China is not the only country alle...1
...............
19651.230000e+18one@TheSecondRevol1 @VincentCrypt46 I have been s...1
19661.230000e+18oneIf anyone is aware of the 5G theory related to...1
19671.230000e+18one@ClydeLewis @ClydeLewis have you looked into 5...1
19681.230000e+18oneHow many have the “Virus” where there’s...1
19691.230000e+18oneSo why am I still so thoroughly stunned by the...1
\n", "

1970 rows × 4 columns

\n", "
" ], "text/plain": [ " ID ... Product_Labels\n", "0 1.005030e+05 ... 3\n", "1 1.009760e+05 ... 3\n", "2 1.011710e+05 ... 0\n", "3 1.017920e+05 ... 3\n", "4 1.037400e+05 ... 1\n", "... ... ... ...\n", "1965 1.230000e+18 ... 1\n", "1966 1.230000e+18 ... 1\n", "1967 1.230000e+18 ... 1\n", "1968 1.230000e+18 ... 1\n", "1969 1.230000e+18 ... 1\n", "\n", "[1970 rows x 4 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "markdown", "metadata": { "id": "n_287udVswYw" }, "source": [ "Clean(tweet)" ] }, { "cell_type": "code", "metadata": { "cellView": "both", "id": "4bpSwtLzswYy" }, "source": [ "\n", "import re\n", "import string\n", "def clean(tweet):\n", " \n", " # Special characters\n", " tweet = re.sub(r\"\\x89Û_\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÒ\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÓ\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÏWhen\", \"When\", tweet)\n", " tweet = re.sub(r\"\\x89ÛÏ\", \"\", tweet)\n", " tweet = re.sub(r\"China\\x89Ûªs\", \"China's\", tweet)\n", " tweet = re.sub(r\"let\\x89Ûªs\", \"let's\", tweet)\n", " tweet = re.sub(r\"\\x89Û÷\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Ûª\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û\\x9d\", \"\", tweet)\n", " tweet = re.sub(r\"å_\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û¢\", \"\", tweet)\n", " tweet = re.sub(r\"\\x89Û¢åÊ\", \"\", tweet)\n", " tweet = re.sub(r\"fromåÊwounds\", \"from wounds\", tweet)\n", " tweet = re.sub(r\"åÊ\", \"\", tweet)\n", " tweet = re.sub(r\"åÈ\", \"\", tweet)\n", " tweet = re.sub(r\"JapÌ_n\", \"Japan\", tweet) \n", " tweet = re.sub(r\"Ì©\", \"e\", tweet)\n", " tweet = re.sub(r\"å¨\", \"\", tweet)\n", " tweet = re.sub(r\"Surṳ\", \"Suruc\", tweet)\n", " tweet = re.sub(r\"åÇ\", \"\", tweet)\n", " tweet = re.sub(r\"å£3million\", \"3 million\", tweet)\n", " tweet = re.sub(r\"åÀ\", \"\", tweet)\n", " \n", " # Contractions\n", " tweet = re.sub(r\"he's\", \"he is\", tweet)\n", " tweet = re.sub(r\"there's\", \"there is\", tweet)\n", " tweet = re.sub(r\"We're\", \"We are\", tweet)\n", " tweet = re.sub(r\"That's\", \"That is\", tweet)\n", " tweet = re.sub(r\"won't\", \"will not\", tweet)\n", " tweet = re.sub(r\"they're\", \"they are\", tweet)\n", " tweet = re.sub(r\"Can't\", \"Cannot\", tweet)\n", " tweet = re.sub(r\"wasn't\", \"was not\", tweet)\n", " tweet = re.sub(r\"don\\x89Ûªt\", \"do not\", tweet)\n", " tweet = re.sub(r\"aren't\", \"are not\", tweet)\n", " tweet = re.sub(r\"isn't\", \"is not\", tweet)\n", " tweet = re.sub(r\"What's\", \"What is\", tweet)\n", " tweet = re.sub(r\"haven't\", \"have not\", tweet)\n", " tweet = re.sub(r\"hasn't\", \"has not\", tweet)\n", " tweet = re.sub(r\"There's\", \"There is\", tweet)\n", " tweet = re.sub(r\"He's\", \"He is\", tweet)\n", " tweet = re.sub(r\"It's\", \"It is\", tweet)\n", " tweet = re.sub(r\"You're\", \"You are\", tweet)\n", " tweet = re.sub(r\"I'M\", \"I am\", tweet)\n", " tweet = re.sub(r\"shouldn't\", \"should not\", tweet)\n", " tweet = re.sub(r\"wouldn't\", \"would not\", tweet)\n", " tweet = re.sub(r\"i'm\", \"I am\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªm\", \"I am\", tweet)\n", " tweet = re.sub(r\"I'm\", \"I am\", tweet)\n", " tweet = re.sub(r\"Isn't\", \"is not\", tweet)\n", " tweet = re.sub(r\"Here's\", \"Here is\", tweet)\n", " tweet = re.sub(r\"you've\", \"you have\", tweet)\n", " tweet = re.sub(r\"you\\x89Ûªve\", \"you have\", tweet)\n", " tweet = re.sub(r\"we're\", \"we are\", tweet)\n", " tweet = re.sub(r\"what's\", \"what is\", tweet)\n", " tweet = re.sub(r\"couldn't\", \"could not\", tweet)\n", " tweet = re.sub(r\"we've\", \"we have\", tweet)\n", " tweet = re.sub(r\"it\\x89Ûªs\", \"it is\", tweet)\n", " tweet = re.sub(r\"doesn\\x89Ûªt\", \"does not\", tweet)\n", " tweet = re.sub(r\"It\\x89Ûªs\", \"It is\", tweet)\n", " tweet = re.sub(r\"Here\\x89Ûªs\", \"Here is\", tweet)\n", " tweet = re.sub(r\"who's\", \"who is\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªve\", \"I have\", tweet)\n", " tweet = re.sub(r\"y'all\", \"you all\", tweet)\n", " tweet = re.sub(r\"can\\x89Ûªt\", \"cannot\", tweet)\n", " tweet = re.sub(r\"would've\", \"would have\", tweet)\n", " tweet = re.sub(r\"it'll\", \"it will\", tweet)\n", " tweet = re.sub(r\"we'll\", \"we will\", tweet)\n", " tweet = re.sub(r\"wouldn\\x89Ûªt\", \"would not\", tweet)\n", " tweet = re.sub(r\"We've\", \"We have\", tweet)\n", " tweet = re.sub(r\"he'll\", \"he will\", tweet)\n", " tweet = re.sub(r\"Y'all\", \"You all\", tweet)\n", " tweet = re.sub(r\"Weren't\", \"Were not\", tweet)\n", " tweet = re.sub(r\"Didn't\", \"Did not\", tweet)\n", " tweet = re.sub(r\"they'll\", \"they will\", tweet)\n", " tweet = re.sub(r\"they'd\", \"they would\", tweet)\n", " tweet = re.sub(r\"DON'T\", \"DO NOT\", tweet)\n", " tweet = re.sub(r\"That\\x89Ûªs\", \"That is\", tweet)\n", " tweet = re.sub(r\"they've\", \"they have\", tweet)\n", " tweet = re.sub(r\"i'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"should've\", \"should have\", tweet)\n", " tweet = re.sub(r\"You\\x89Ûªre\", \"You are\", tweet)\n", " tweet = re.sub(r\"where's\", \"where is\", tweet)\n", " tweet = re.sub(r\"Don\\x89Ûªt\", \"Do not\", tweet)\n", " tweet = re.sub(r\"we'd\", \"we would\", tweet)\n", " tweet = re.sub(r\"i'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"weren't\", \"were not\", tweet)\n", " tweet = re.sub(r\"They're\", \"They are\", tweet)\n", " tweet = re.sub(r\"Can\\x89Ûªt\", \"Cannot\", tweet)\n", " tweet = re.sub(r\"you\\x89Ûªll\", \"you will\", tweet)\n", " tweet = re.sub(r\"I\\x89Ûªd\", \"I would\", tweet)\n", " tweet = re.sub(r\"let's\", \"let us\", tweet)\n", " tweet = re.sub(r\"it's\", \"it is\", tweet)\n", " tweet = re.sub(r\"can't\", \"cannot\", tweet)\n", " tweet = re.sub(r\"don't\", \"do not\", tweet)\n", " tweet = re.sub(r\"you're\", \"you are\", tweet)\n", " tweet = re.sub(r\"i've\", \"I have\", tweet)\n", " tweet = re.sub(r\"that's\", \"that is\", tweet)\n", " tweet = re.sub(r\"i'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"doesn't\", \"does not\", tweet)\n", " tweet = re.sub(r\"i'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"didn't\", \"did not\", tweet)\n", " tweet = re.sub(r\"ain't\", \"am not\", tweet)\n", " tweet = re.sub(r\"you'll\", \"you will\", tweet)\n", " tweet = re.sub(r\"I've\", \"I have\", tweet)\n", " tweet = re.sub(r\"Don't\", \"do not\", tweet)\n", " tweet = re.sub(r\"I'll\", \"I will\", tweet)\n", " tweet = re.sub(r\"I'd\", \"I would\", tweet)\n", " tweet = re.sub(r\"Let's\", \"Let us\", tweet)\n", " tweet = re.sub(r\"you'd\", \"You would\", tweet)\n", " tweet = re.sub(r\"It's\", \"It is\", tweet)\n", " tweet = re.sub(r\"Ain't\", \"am not\", tweet)\n", " tweet = re.sub(r\"Haven't\", \"Have not\", tweet)\n", " tweet = re.sub(r\"Could've\", \"Could have\", tweet)\n", " tweet = re.sub(r\"youve\", \"you have\", tweet) \n", " tweet = re.sub(r\"donå«t\", \"do not\", tweet) \n", " \n", " # Character entity references\n", " tweet = re.sub(r\">\", \">\", tweet)\n", " tweet = re.sub(r\"<\", \"<\", tweet)\n", " tweet = re.sub(r\"&\", \"&\", tweet)\n", " \n", " # Typos, slang and informal abbreviations\n", " tweet = re.sub(r\"w/e\", \"whatever\", tweet)\n", " tweet = re.sub(r\"w/\", \"with\", tweet)\n", " tweet = re.sub(r\"USAgov\", \"USA government\", tweet)\n", " tweet = re.sub(r\"recentlu\", \"recently\", tweet)\n", " tweet = re.sub(r\"Ph0tos\", \"Photos\", tweet)\n", " tweet = re.sub(r\"amirite\", \"am I right\", tweet)\n", " tweet = re.sub(r\"exp0sed\", \"exposed\", tweet)\n", " tweet = re.sub(r\"<3\", \"love\", tweet)\n", " tweet = re.sub(r\"amageddon\", \"armageddon\", tweet)\n", " tweet = re.sub(r\"Trfc\", \"Traffic\", tweet)\n", " tweet = re.sub(r\"8/5/2015\", \"2015-08-05\", tweet)\n", " tweet = re.sub(r\"WindStorm\", \"Wind Storm\", tweet)\n", " tweet = re.sub(r\"8/6/2015\", \"2015-08-06\", tweet)\n", " tweet = re.sub(r\"10:38PM\", \"10:38 PM\", tweet)\n", " tweet = re.sub(r\"10:30pm\", \"10:30 PM\", tweet)\n", " tweet = re.sub(r\"16yr\", \"16 year\", tweet)\n", " tweet = re.sub(r\"lmao\", \"laughing my ass off\", tweet) \n", " tweet = re.sub(r\"TRAUMATISED\", \"traumatized\", tweet)\n", " \n", " # re.sub(r\"humanconsumption\", \"human consumption\", tweet)\n", " # tweet = re.sub(r\"Typhoon-Devastated\", \"Typhoon Devastated\", tweet)\n", " # tweet = re.sub(r\"Meat-Loving\", \"Meat Loving\", tweet)\n", " # tweet = re.sub(r\"facialabuse\", \"facial abuse\", tweet)\n", " # tweet = re.sub(r\"LakeCounty\", \"Lake County\", tweet)\n", " # tweet = re.sub(r\"BeingAuthor\", \"Being Author\", tweet)\n", " # tweet = re.sub(r\"withheavenly\", \"with heavenly\", tweet)\n", " # tweet = re.sub(r\"thankU\", \"thank you\", tweet)\n", " # tweet = re.sub(r\"iTunesMusic\", \"iTunes Music\", tweet)\n", " # tweet = re.sub(r\"OffensiveContent\", \"Offensive Content\", tweet)\n", " # tweet = re.sub(r\"WorstSummerJob\", \"Worst Summer Job\", tweet)\n", " # tweet = re.sub(r\"HarryBeCareful\", \"Harry Be Careful\", tweet)\n", " # tweet = re.sub(r\"NASASolarSystem\", \"NASA Solar System\", tweet)\n", " # tweet = re.sub(r\"animalrescue\", \"animal rescue\", tweet)\n", " # tweet = re.sub(r\"KurtSchlichter\", \"Kurt Schlichter\", tweet)\n", " # tweet = re.sub(r\"aRmageddon\", \"armageddon\", tweet)\n", " # tweet = re.sub(r\"Throwingknifes\", \"Throwing knives\", tweet)\n", " # tweet = re.sub(r\"GodsLove\", \"God's Love\", tweet)\n", " # tweet = re.sub(r\"bookboost\", \"book boost\", tweet)\n", " # tweet = re.sub(r\"ibooklove\", \"I book love\", tweet)\n", " # tweet = re.sub(r\"NestleIndia\", \"Nestle India\", tweet)\n", " # tweet = re.sub(r\"realDonaldTrump\", \"Donald Trump\", tweet)\n", " # tweet = re.sub(r\"DavidVonderhaar\", \"David Vonderhaar\", tweet)\n", " # tweet = re.sub(r\"CecilTheLion\", \"Cecil The Lion\", tweet)\n", " # tweet = re.sub(r\"weathernetwork\", \"weather network\", tweet)\n", " # tweet = re.sub(r\"withBioterrorism # Hashtags and usernames\n", " # tweet = re.sub(r\"IranDeal\", \"Iran Deal\", tweet)\n", " # tweet = re.sub(r\"ArianaGrande\", \"Ariana Grande\", tweet)\n", " # tweet = re.sub(r\"camilacabello97\", \"camila cabello\", tweet) \n", " # tweet = re.sub(r\"RondaRousey\", \"Ronda Rousey\", tweet) \n", " # tweet = re.sub(r\"MTVHottest\", \"MTV Hottest\", tweet)\n", " # tweet = re.sub(r\"TrapMusic\", \"Trap Music\", tweet)\n", " # tweet = re.sub(r\"ProphetMuhammad\", \"Prophet Muhammad\", tweet)\n", " # tweet = re.sub(r\"PantherAttack\", \"Panther Attack\", tweet)\n", " # tweet = re.sub(r\"StrategicPatience\", \"Strategic Patience\", tweet)\n", " # tweet = re.sub(r\"socialnews\", \"social news\", tweet)\n", " # tweet = re.sub(r\"NASAHurricane\", \"NASA Hurricane\", tweet)\n", " # tweet = re.sub(r\"onlinecommunities\", \"online communities\", tweet)\n", " # tweet = &use\", \"with Bioterrorism & use\", tweet)\n", " # tweet = re.sub(r\"Hostage&2\", \"Hostage & 2\", tweet)\n", " # tweet = re.sub(r\"GOPDebate\", \"GOP Debate\", tweet)\n", " # tweet = re.sub(r\"RickPerry\", \"Rick Perry\", tweet)\n", " # tweet = re.sub(r\"frontpage\", \"front page\", tweet)\n", " # tweet = re.sub(r\"NewsInTweets\", \"News In Tweets\", tweet)\n", " # tweet = re.sub(r\"ViralSpell\", \"Viral Spell\", tweet)\n", " # tweet = re.sub(r\"til_now\", \"until now\", tweet)\n", " # tweet = re.sub(r\"volcanoinRussia\", \"volcano in Russia\", tweet)\n", " # tweet = re.sub(r\"ZippedNews\", \"Zipped News\", tweet)\n", " # tweet = re.sub(r\"MicheleBachman\", \"Michele Bachman\", tweet)\n", " # tweet = re.sub(r\"53inch\", \"53 inch\", tweet)\n", " # tweet = re.sub(r\"KerrickTrial\", \"Kerrick Trial\", tweet)\n", " # tweet = re.sub(r\"abstorm\", \"Alberta Storm\", tweet)\n", " # tweet = re.sub(r\"Beyhive\", \"Beyonce hive\", tweet)\n", " # tweet = re.sub(r\"IDFire\", \"Idaho Fire\", tweet)\n", " # tweet = re.sub(r\"DETECTADO\", \"Detected\", tweet)\n", " # tweet = re.sub(r\"RockyFire\", \"Rocky Fire\", tweet)\n", " # tweet = re.sub(r\"Listen/Buy\", \"Listen / Buy\", tweet)\n", " # tweet = re.sub(r\"NickCannon\", \"Nick Cannon\", tweet)\n", " # tweet = re.sub(r\"FaroeIslands\", \"Faroe Islands\", tweet)\n", " # tweet = re.sub(r\"yycstorm\", \"Calgary Storm\", tweet)\n", " # tweet = re.sub(r\"IDPs:\", \"Internally Displaced People :\", tweet)\n", " # tweet = re.sub(r\"ArtistsUnited\", \"Artists United\", tweet)\n", " # tweet = re.sub(r\"ClaytonBryant\", \"Clayton Bryant\", tweet)\n", " # tweet = re.sub(r\"jimmyfallon\", \"jimmy fallon\", tweet)\n", " # tweet = re.sub(r\"justinbieber\", \"justin bieber\", tweet) \n", " # tweet = re.sub(r\"UTC2015\", \"UTC 2015\", tweet)\n", " # tweet = re.sub(r\"Time2015\", \"Time 2015\", tweet)\n", " # tweet = re.sub(r\"djicemoon\", \"dj icemoon\", tweet)\n", " # tweet = re.sub(r\"LivingSafely\", \"Living Safely\", tweet)\n", " # tweet = re.sub(r\"FIFA16\", \"Fifa 2016\", tweet)\n", " # tweet = re.sub(r\"thisiswhywecanthavenicethings\", \"this is why we cannot have nice things\", tweet)\n", " # tweet = re.sub(r\"bbcnews\", \"bbc news\", tweet)\n", " # tweet = re.sub(r\"UndergroundRailraod\", \"Underground Railraod\", tweet)\n", " # tweet = re.sub(r\"c4news\", \"c4 news\", tweet)\n", " # tweet = re.sub(r\"OBLITERATION\", \"obliteration\", tweet)\n", " # tweet = re.sub(r\"MUDSLIDE\", \"mudslide\", tweet)\n", " # tweet = re.sub(r\"NoSurrender\", \"No Surrender\", tweet)\n", " # tweet = re.sub(r\"NotExplained\", \"Not Explained\", tweet)\n", " # tweet = re.sub(r\"greatbritishbakeoff\", \"great british bake off\", tweet)\n", " # tweet = re.sub(r\"LondonFire\", \"London Fire\", tweet)\n", " # tweet = re.sub(r\"KOTAWeather\", \"KOTA Weather\", tweet)\n", " # tweet = re.sub(r\"LuchaUnderground\", \"Lucha Underground\", tweet)\n", " # tweet = re.sub(r\"KOIN6News\", \"KOIN 6 News\", tweet)\n", " # tweet = re.sub(r\"LiveOnK2\", \"Live On K2\", tweet)\n", " # tweet = re.sub(r\"9NewsGoldCoast\", \"9 News Gold Coast\", tweet)\n", " # tweet = re.sub(r\"nikeplus\", \"nike plus\", tweet)\n", " # tweet = re.sub(r\"david_cameron\", \"David Cameron\", tweet)\n", " # tweet = re.sub(r\"peterjukes\", \"Peter Jukes\", tweet)\n", " # tweet = re.sub(r\"JamesMelville\", \"James Melville\", tweet)\n", " # tweet = re.sub(r\"megynkelly\", \"Megyn Kelly\", tweet)\n", " # tweet = re.sub(r\"cnewslive\", \"C News Live\", tweet)\n", " # tweet = re.sub(r\"JamaicaObserver\", \"Jamaica Observer\", tweet)\n", " # tweet = re.sub(r\"TweetLikeItsSeptember11th2001\", \"Tweet like it is september 11th 2001\", tweet)\n", " # tweet = re.sub(r\"cbplawyers\", \"cbp lawyers\", tweet)\n", " # tweet = re.sub(r\"fewmoretweets\", \"few more tweets\", tweet)\n", " # tweet = re.sub(r\"BlackLivesMatter\", \"Black Lives Matter\", tweet)\n", " # tweet = re.sub(r\"cjoyner\", \"Chris Joyner\", tweet)\n", " # tweet = re.sub(r\"ENGvAUS\", \"England vs Australia\", tweet)\n", " # tweet = re.sub(r\"ScottWalker\", \"Scott Walker\", tweet)\n", " # tweet = re.sub(r\"MikeParrActor\", \"Michael Parr\", tweet)\n", " # tweet = re.sub(r\"4PlayThursdays\", \"Foreplay Thursdays\", tweet)\n", " # tweet = re.sub(r\"TGF2015\", \"Tontitown Grape Festival\", tweet)\n", " # tweet = re.sub(r\"realmandyrain\", \"Mandy Rain\", tweet)\n", " # tweet = re.sub(r\"GraysonDolan\", \"Grayson Dolan\", tweet)\n", " # tweet = re.sub(r\"ApolloBrown\", \"Apollo Brown\", tweet)\n", " # tweet = re.sub(r\"saddlebrooke\", \"Saddlebrooke\", tweet)\n", " # tweet = re.sub(r\"TontitownGrape\", \"Tontitown Grape\", tweet)\n", " # tweet = re.sub(r\"AbbsWinston\", \"Abbs Winston\", tweet)\n", " # tweet = re.sub(r\"ShaunKing\", \"Shaun King\", tweet)\n", " # tweet = re.sub(r\"MeekMill\", \"Meek Mill\", tweet)\n", " # tweet = re.sub(r\"TornadoGiveaway\", \"Tornado Giveaway\", tweet)\n", " # tweet = re.sub(r\"GRupdates\", \"GR updates\", tweet)\n", " # tweet = re.sub(r\"SouthDowns\", \"South Downs\", tweet)\n", " # tweet = re.sub(r\"braininjury\", \"brain injury\", tweet)\n", " # tweet = re.sub(r\"auspol\", \"Australian politics\", tweet)\n", " # tweet = re.sub(r\"PlannedParenthood\", \"Planned Parenthood\", tweet)\n", " # tweet = re.sub(r\"calgaryweather\", \"Calgary Weather\", tweet)\n", " # tweet = re.sub(r\"weallheartonedirection\", \"we all heart one direction\", tweet)\n", " # tweet = re.sub(r\"edsheeran\", \"Ed Sheeran\", tweet)\n", " # tweet = re.sub(r\"TrueHeroes\", \"True Heroes\", tweet)\n", " # tweet = re.sub(r\"S3XLEAK\", \"sex leak\", tweet)\n", " # tweet = re.sub(r\"ComplexMag\", \"Complex Magazine\", tweet)\n", " # tweet = re.sub(r\"TheAdvocateMag\", \"The Advocate Magazine\", tweet)\n", " # tweet = re.sub(r\"CityofCalgary\", \"City of Calgary\", tweet)\n", " # tweet = re.sub(r\"EbolaOutbreak\", \"Ebola Outbreak\", tweet)\n", " # tweet = re.sub(r\"SummerFate\", \"Summer Fate\", tweet)\n", " # tweet = re.sub(r\"RAmag\", \"Royal Academy Magazine\", tweet)\n", " # tweet = re.sub(r\"offers2go\", \"offers to go\", tweet)\n", " # tweet = re.sub(r\"foodscare\", \"food scare\", tweet)\n", " # tweet = re.sub(r\"MNPDNashville\", \"Metropolitan Nashville Police Department\", tweet)\n", " # tweet = re.sub(r\"TfLBusAlerts\", \"TfL Bus Alerts\", tweet)\n", " # tweet = re.sub(r\"GamerGate\", \"Gamer Gate\", tweet)\n", " # tweet = re.sub(r\"IHHen\", \"Humanitarian Relief\", tweet)\n", " # tweet = re.sub(r\"spinningbot\", \"spinning bot\", tweet)\n", " # tweet = re.sub(r\"ModiMinistry\", \"Modi Ministry\", tweet)\n", " # tweet = re.sub(r\"TAXIWAYS\", \"taxi ways\", tweet)\n", " # tweet = re.sub(r\"Calum5SOS\", \"Calum Hood\", tweet)\n", " # tweet = re.sub(r\"po_st\", \"po.st\", tweet)\n", " # tweet = re.sub(r\"scoopit\", \"scoop.it\", tweet)\n", " # tweet = re.sub(r\"UltimaLucha\", \"Ultima Lucha\", tweet)\n", " # tweet = re.sub(r\"JonathanFerrell\", \"Jonathan Ferrell\", tweet)\n", " # tweet = re.sub(r\"aria_ahrary\", \"Aria Ahrary\", tweet)\n", " # tweet = re.sub(r\"rapidcity\", \"Rapid City\", tweet)\n", " # tweet = re.sub(r\"OutBid\", \"outbid\", tweet)\n", " # tweet = re.sub(r\"lavenderpoetrycafe\", \"lavender poetry cafe\", tweet)\n", " # tweet = re.sub(r\"EudryLantiqua\", \"Eudry Lantiqua\", tweet)\n", " # tweet = re.sub(r\"15PM\", \"15 PM\", tweet)\n", " # tweet = re.sub(r\"OriginalFunko\", \"Funko\", tweet)\n", " # tweet = re.sub(r\"rightwaystan\", \"Richard Tan\", tweet)\n", " # tweet = re.sub(r\"CindyNoonan\", \"Cindy Noonan\", tweet)\n", " # tweet = re.sub(r\"RT_America\", \"RT America\", tweet)\n", " # tweet = re.sub(r\"narendramodi\", \"Narendra Modi\", tweet)\n", " # tweet = re.sub(r\"BakeOffFriends\", \"Bake Off Friends\", tweet)\n", " # tweet = re.sub(r\"TeamHendrick\", \"Hendrick Motorsports\", tweet)\n", " # tweet = re.sub(r\"alexbelloli\", \"Alex Belloli\", tweet)\n", " # tweet = re.sub(r\"itsjustinstuart\", \"Justin Stuart\", tweet)\n", " # tweet = re.sub(r\"gunsense\", \"gun sense\", tweet)\n", " # tweet = re.sub(r\"DebateQuestionsWeWantToHear\", \"debate questions we want to hear\", tweet)\n", " # tweet = re.sub(r\"RoyalCarribean\", \"Royal Carribean\", tweet)\n", " # tweet = re.sub(r\"samanthaturne19\", \"Samantha Turner\", tweet)\n", " # tweet = re.sub(r\"JonVoyage\", \"Jon Stewart\", tweet)\n", " # tweet = re.sub(r\"renew911health\", \"renew 911 health\", tweet)\n", " # tweet = re.sub(r\"SuryaRay\", \"Surya Ray\", tweet)\n", " # tweet = re.sub(r\"pattonoswalt\", \"Patton Oswalt\", tweet)\n", " # tweet = re.sub(r\"minhazmerchant\", \"Minhaz Merchant\", tweet)\n", " # tweet = re.sub(r\"TLVFaces\", \"Israel Diaspora Coalition\", tweet)\n", " # tweet = re.sub(r\"pmarca\", \"Marc Andreessen\", tweet)\n", " # tweet = re.sub(r\"pdx911\", \"Portland Police\", tweet)\n", " # tweet = re.sub(r\"jamaicaplain\", \"Jamaica Plain\", tweet)\n", " # tweet = re.sub(r\"Japton\", \"Arkansas\", tweet)\n", " # tweet = re.sub(r\"RouteComplex\", \"Route Complex\", tweet)\n", " # tweet = re.sub(r\"INSubcontinent\", \"Indian Subcontinent\", tweet)\n", " # tweet = re.sub(r\"NJTurnpike\", \"New Jersey Turnpike\", tweet)\n", " # tweet = re.sub(r\"Politifiact\", \"PolitiFact\", tweet)\n", " # tweet = re.sub(r\"Hiroshima70\", \"Hiroshima\", tweet)\n", " # tweet = re.sub(r\"GMMBC\", \"Greater Mt Moriah Baptist Church\", tweet)\n", " # tweet = re.sub(r\"versethe\", \"verse the\", tweet)\n", " # tweet = re.sub(r\"TubeStrike\", \"Tube Strike\", tweet)\n", " # tweet = re.sub(r\"MissionHills\", \"Mission Hills\", tweet)\n", " # tweet = re.sub(r\"ProtectDenaliWolves\", \"Protect Denali Wolves\", tweet)\n", " # tweet = re.sub(r\"NANKANA\", \"Nankana\", tweet)\n", " # tweet = re.sub(r\"SAHIB\", \"Sahib\", tweet)\n", " # tweet = re.sub(r\"PAKPATTAN\", \"Pakpattan\", tweet)\n", " # tweet = re.sub(r\"Newz_Sacramento\", \"News Sacramento\", tweet)\n", " # tweet = re.sub(r\"gofundme\", \"go fund me\", tweet)\n", " # tweet = re.sub(r\"pmharper\", \"Stephen Harper\", tweet)\n", " # tweet = re.sub(r\"IvanBerroa\", \"Ivan Berroa\", tweet)\n", " # tweet = re.sub(r\"LosDelSonido\", \"Los Del Sonido\", tweet)\n", " # tweet = re.sub(r\"bancodeseries\", \"banco de series\", tweet)\n", " # tweet = re.sub(r\"timkaine\", \"Tim Kaine\", tweet)\n", " # tweet = re.sub(r\"IdentityTheft\", \"Identity Theft\", tweet)\n", " # tweet = re.sub(r\"AllLivesMatter\", \"All Lives Matter\", tweet)\n", " # tweet = re.sub(r\"mishacollins\", \"Misha Collins\", tweet)\n", " # tweet = re.sub(r\"BillNeelyNBC\", \"Bill Neely\", tweet)\n", " # tweet = re.sub(r\"BeClearOnCancer\", \"be clear on cancer\", tweet)\n", " # tweet = re.sub(r\"Kowing\", \"Knowing\", tweet)\n", " # tweet = re.sub(r\"ScreamQueens\", \"Scream Queens\", tweet)\n", " # tweet = re.sub(r\"AskCharley\", \"Ask Charley\", tweet)\n", " # tweet = re.sub(r\"BlizzHeroes\", \"Heroes of the Storm\", tweet)\n", " # tweet = re.sub(r\"BradleyBrad47\", \"Bradley Brad\", tweet)\n", " # tweet = re.sub(r\"HannaPH\", \"Typhoon Hanna\", tweet)\n", " # tweet = re.sub(r\"meinlcymbals\", \"MEINL Cymbals\", tweet)\n", " # tweet = re.sub(r\"Ptbo\", \"Peterborough\", tweet)\n", " # tweet = re.sub(r\"cnnbrk\", \"CNN Breaking News\", tweet)\n", " # tweet = re.sub(r\"IndianNews\", \"Indian News\", tweet)\n", " # tweet = re.sub(r\"savebees\", \"save bees\", tweet)\n", " # tweet = re.sub(r\"GreenHarvard\", \"Green Harvard\", tweet)\n", " # tweet = re.sub(r\"StandwithPP\", \"Stand with planned parenthood\", tweet)\n", " # tweet = re.sub(r\"hermancranston\", \"Herman Cranston\", tweet)\n", " # tweet = re.sub(r\"WMUR9\", \"WMUR-TV\", tweet)\n", " # tweet = re.sub(r\"RockBottomRadFM\", \"Rock Bottom Radio\", tweet)\n", " # tweet = re.sub(r\"ameenshaikh3\", \"Ameen Shaikh\", tweet)\n", " # tweet = re.sub(r\"ProSyn\", \"Project Syndicate\", tweet)\n", " # tweet = re.sub(r\"Daesh\", \"ISIS\", tweet)\n", " # tweet = re.sub(r\"s2g\", \"swear to god\", tweet)\n", " # tweet = re.sub(r\"listenlive\", \"listen live\", tweet)\n", " # tweet = re.sub(r\"CDCgov\", \"Centers for Disease Control and Prevention\", tweet)\n", " # tweet = re.sub(r\"FoxNew\", \"Fox News\", tweet)\n", " # tweet = re.sub(r\"CBSBigBrother\", \"Big Brother\", tweet)\n", " # tweet = re.sub(r\"JulieDiCaro\", \"Julie DiCaro\", tweet)\n", " # tweet = re.sub(r\"theadvocatemag\", \"The Advocate Magazine\", tweet)\n", " # tweet = re.sub(r\"RohnertParkDPS\", \"Rohnert Park Police Department\", tweet)\n", " # tweet = re.sub(r\"THISIZBWRIGHT\", \"Bonnie Wright\", tweet)\n", " # tweet = re.sub(r\"Popularmmos\", \"Popular MMOs\", tweet)\n", " # tweet = re.sub(r\"WildHorses\", \"Wild Horses\", tweet)\n", " # tweet = re.sub(r\"FantasticFour\", \"Fantastic Four\", tweet)\n", " # tweet = re.sub(r\"HORNDALE\", \"Horndale\", tweet)\n", " # tweet = re.sub(r\"PINER\", \"Piner\", tweet)\n", " # tweet = re.sub(r\"BathAndNorthEastSomerset\", \"Bath and North East Somerset\", tweet)\n", " # tweet = re.sub(r\"thatswhatfriendsarefor\", \"that is what friends are for\", tweet)\n", " # tweet = re.sub(r\"residualincome\", \"residual income\", tweet)\n", " # tweet = re.sub(r\"YahooNewsDigest\", \"Yahoo News Digest\", tweet)\n", " # tweet = re.sub(r\"MalaysiaAirlines\", \"Malaysia Airlines\", tweet)\n", " # tweet = re.sub(r\"AmazonDeals\", \"Amazon Deals\", tweet)\n", " # tweet = re.sub(r\"MissCharleyWebb\", \"Charley Webb\", tweet)\n", " # tweet = re.sub(r\"shoalstraffic\", \"shoals traffic\", tweet)\n", " # tweet = re.sub(r\"GeorgeFoster72\", \"George Foster\", tweet)\n", " # tweet = re.sub(r\"pop2015\", \"pop 2015\", tweet)\n", " # tweet = re.sub(r\"_PokemonCards_\", \"Pokemon Cards\", tweet)\n", " # tweet = re.sub(r\"DianneG\", \"Dianne Gallagher\", tweet)\n", " # tweet = re.sub(r\"KashmirConflict\", \"Kashmir Conflict\", tweet)\n", " # tweet = re.sub(r\"BritishBakeOff\", \"British Bake Off\", tweet)\n", " # tweet = re.sub(r\"FreeKashmir\", \"Free Kashmir\", tweet)\n", " # tweet = re.sub(r\"mattmosley\", \"Matt Mosley\", tweet)\n", " # tweet = re.sub(r\"BishopFred\", \"Bishop Fred\", tweet)\n", " # tweet = re.sub(r\"EndConflict\", \"End Conflict\", tweet)\n", " # tweet = re.sub(r\"EndOccupation\", \"End Occupation\", tweet)\n", " # tweet = re.sub(r\"UNHEALED\", \"unhealed\", tweet)\n", " # tweet = re.sub(r\"CharlesDagnall\", \"Charles Dagnall\", tweet)\n", " # tweet = re.sub(r\"Latestnews\", \"Latest news\", tweet)\n", " # tweet = re.sub(r\"KindleCountdown\", \"Kindle Countdown\", tweet)\n", " # tweet = re.sub(r\"NoMoreHandouts\", \"No More Handouts\", tweet)\n", " # tweet = re.sub(r\"datingtips\", \"dating tips\", tweet)\n", " # tweet = re.sub(r\"charlesadler\", \"Charles Adler\", tweet)\n", " # tweet = re.sub(r\"twia\", \"Texas Windstorm Insurance Association\", tweet)\n", " # tweet = re.sub(r\"txlege\", \"Texas Legislature\", tweet)\n", " # tweet = re.sub(r\"WindstormInsurer\", \"Windstorm Insurer\", tweet)\n", " # tweet = re.sub(r\"Newss\", \"News\", tweet)\n", " # tweet = re.sub(r\"hempoil\", \"hemp oil\", tweet)\n", " # tweet = re.sub(r\"CommoditiesAre\", \"Commodities are\", tweet)\n", " # tweet = re.sub(r\"tubestrike\", \"tube strike\", tweet)\n", " # tweet = re.sub(r\"JoeNBC\", \"Joe Scarborough\", tweet)\n", " # tweet = re.sub(r\"LiteraryCakes\", \"Literary Cakes\", tweet)\n", " # tweet = re.sub(r\"TI5\", \"The International 5\", tweet)\n", " # tweet = re.sub(r\"thehill\", \"the hill\", tweet)\n", " # tweet = re.sub(r\"3others\", \"3 others\", tweet)\n", " # tweet = re.sub(r\"stighefootball\", \"Sam Tighe\", tweet)\n", " # tweet = re.sub(r\"whatstheimportantvideo\", \"what is the important video\", tweet)\n", " # tweet = re.sub(r\"ClaudioMeloni\", \"Claudio Meloni\", tweet)\n", " # tweet = re.sub(r\"DukeSkywalker\", \"Duke Skywalker\", tweet)\n", " # tweet = re.sub(r\"carsonmwr\", \"Fort Carson\", tweet)\n", " # tweet = re.sub(r\"offdishduty\", \"off dish duty\", tweet)\n", " # tweet = re.sub(r\"andword\", \"and word\", tweet)\n", " # tweet = re.sub(r\"rhodeisland\", \"Rhode Island\", tweet)\n", " # tweet = re.sub(r\"easternoregon\", \"Eastern Oregon\", tweet)\n", " # tweet = re.sub(r\"WAwildfire\", \"Washington Wildfire\", tweet)\n", " # tweet = re.sub(r\"fingerrockfire\", \"Finger Rock Fire\", tweet)\n", " # tweet = re.sub(r\"57am\", \"57 am\", tweet)\n", " # tweet = re.sub(r\"fingerrockfire\", \"Finger Rock Fire\", tweet)\n", " # tweet = re.sub(r\"JacobHoggard\", \"Jacob Hoggard\", tweet)\n", " # tweet = re.sub(r\"newnewnew\", \"new new new\", tweet)\n", " # tweet = re.sub(r\"under50\", \"under 50\", tweet)\n", " # tweet = re.sub(r\"getitbeforeitsgone\", \"get it before it is gone\", tweet)\n", " # tweet = re.sub(r\"freshoutofthebox\", \"fresh out of the box\", tweet)\n", " # tweet = re.sub(r\"amwriting\", \"am writing\", tweet)\n", " # tweet = re.sub(r\"Bokoharm\", \"Boko Haram\", tweet)\n", " # tweet = re.sub(r\"Nowlike\", \"Now like\", tweet)\n", " # tweet = re.sub(r\"seasonfrom\", \"season from\", tweet)\n", " # tweet = re.sub(r\"epicente\", \"epicenter\", tweet)\n", " # tweet = re.sub(r\"epicenterr\", \"epicenter\", tweet)\n", " # tweet = re.sub(r\"sicklife\", \"sick life\", tweet)\n", " # tweet = re.sub(r\"yycweather\", \"Calgary Weather\", tweet)\n", " # tweet = re.sub(r\"calgarysun\", \"Calgary Sun\", tweet)\n", " # tweet = re.sub(r\"approachng\", \"approaching\", tweet)\n", " # tweet = re.sub(r\"evng\", \"evening\", tweet)\n", " # tweet = re.sub(r\"Sumthng\", \"something\", tweet)\n", " # tweet = re.sub(r\"EllenPompeo\", \"Ellen Pompeo\", tweet)\n", " # tweet = re.sub(r\"shondarhimes\", \"Shonda Rhimes\", tweet)\n", " # tweet = re.sub(r\"ABCNetwork\", \"ABC Network\", tweet)\n", " # tweet = re.sub(r\"SushmaSwaraj\", \"Sushma Swaraj\", tweet)\n", " # tweet = re.sub(r\"pray4japan\", \"Pray for Japan\", tweet)\n", " # tweet = re.sub(r\"hope4japan\", \"Hope for Japan\", tweet)\n", " # tweet = re.sub(r\"Illusionimagess\", \"Illusion images\", tweet)\n", " # tweet = re.sub(r\"SummerUnderTheStars\", \"Summer Under The Stars\", tweet)\n", " # tweet = re.sub(r\"ShallWeDance\", \"Shall We Dance\", tweet)\n", " # tweet = re.sub(r\"TCMParty\", \"TCM Party\", tweet)\n", " # tweet = re.sub(r\"marijuananews\", \"marijuana news\", tweet)\n", " # tweet = re.sub(r\"onbeingwithKristaTippett\", \"on being with Krista Tippett\", tweet)\n", " # tweet = re.sub(r\"Beingtweets\", \"Being tweets\", tweet)\n", " # tweet = re.sub(r\"newauthors\", \"new authors\", tweet)\n", " # tweet = re.sub(r\"remedyyyy\", \"remedy\", tweet)\n", " # tweet = re.sub(r\"44PM\", \"44 PM\", tweet)\n", " # tweet = re.sub(r\"HeadlinesApp\", \"Headlines App\", tweet)\n", " # tweet = re.sub(r\"40PM\", \"40 PM\", tweet)\n", " # tweet = re.sub(r\"myswc\", \"Severe Weather Center\", tweet)\n", " # tweet = re.sub(r\"ithats\", \"that is\", tweet)\n", " # tweet = re.sub(r\"icouldsitinthismomentforever\", \"I could sit in this moment forever\", tweet)\n", " # tweet = re.sub(r\"FatLoss\", \"Fat Loss\", tweet)\n", " # tweet = re.sub(r\"02PM\", \"02 PM\", tweet)\n", " # tweet = re.sub(r\"MetroFmTalk\", \"Metro Fm Talk\", tweet)\n", " # tweet = re.sub(r\"Bstrd\", \"bastard\", tweet)\n", " # tweet = re.sub(r\"bldy\", \"bloody\", tweet)\n", " # tweet = re.sub(r\"MetrofmTalk\", \"Metro Fm Talk\", tweet)\n", " # tweet = re.sub(r\"terrorismturn\", \"terrorism turn\", tweet)\n", " # tweet = re.sub(r\"BBCNewsAsia\", \"BBC News Asia\", tweet)\n", " # tweet = re.sub(r\"BehindTheScenes\", \"Behind The Scenes\", tweet)\n", " # tweet = re.sub(r\"GeorgeTakei\", \"George Takei\", tweet)\n", " # tweet = re.sub(r\"WomensWeeklyMag\", \"Womens Weekly Magazine\", tweet)\n", " # tweet = re.sub(r\"SurvivorsGuidetoEarth\", \"Survivors Guide to Earth\", tweet)\n", " # tweet = re.sub(r\"incubusband\", \"incubus band\", tweet)\n", " # tweet = re.sub(r\"Babypicturethis\", \"Baby picture this\", tweet)\n", " # tweet = re.sub(r\"BombEffects\", \"Bomb Effects\", tweet)\n", " # tweet = re.sub(r\"win10\", \"Windows 10\", tweet)\n", " # tweet = re.sub(r\"idkidk\", \"I do not know I do not know\", tweet)\n", " # tweet = re.sub(r\"TheWalkingDead\", \"The Walking Dead\", tweet)\n", " # tweet = re.sub(r\"amyschumer\", \"Amy Schumer\", tweet)\n", " # tweet = re.sub(r\"crewlist\", \"crew list\", tweet)\n", " # tweet = re.sub(r\"Erdogans\", \"Erdogan\", tweet)\n", " # tweet = re.sub(r\"BBCLive\", \"BBC Live\", tweet)\n", " # tweet = re.sub(r\"TonyAbbottMHR\", \"Tony Abbott\", tweet)\n", " # tweet = re.sub(r\"paulmyerscough\", \"Paul Myerscough\", tweet)\n", " # tweet = re.sub(r\"georgegallagher\", \"George Gallagher\", tweet)\n", " # tweet = re.sub(r\"JimmieJohnson\", \"Jimmie Johnson\", tweet)\n", " # tweet = re.sub(r\"pctool\", \"pc tool\", tweet)\n", " # tweet = re.sub(r\"DoingHashtagsRight\", \"Doing Hashtags Right\", tweet)\n", " # tweet = re.sub(r\"ThrowbackThursday\", \"Throwback Thursday\", tweet)\n", " # tweet = re.sub(r\"SnowBackSunday\", \"Snowback Sunday\", tweet)\n", " # tweet = re.sub(r\"LakeEffect\", \"Lake Effect\", tweet)\n", " # tweet = re.sub(r\"RTphotographyUK\", \"Richard Thomas Photography UK\", tweet)\n", " # tweet = re.sub(r\"BigBang_CBS\", \"Big Bang CBS\", tweet)\n", " # tweet = re.sub(r\"writerslife\", \"writers life\", tweet)\n", " # tweet = re.sub(r\"NaturalBirth\", \"Natural Birth\", tweet)\n", " # tweet = re.sub(r\"UnusualWords\", \"Unusual Words\", tweet)\n", " # tweet = re.sub(r\"wizkhalifa\", \"Wiz Khalifa\", tweet)\n", " # tweet = re.sub(r\"acreativedc\", \"a creative DC\", tweet)\n", " # tweet = re.sub(r\"vscodc\", \"vsco DC\", tweet)\n", " # tweet = re.sub(r\"VSCOcam\", \"vsco camera\", tweet)\n", " # tweet = re.sub(r\"TheBEACHDC\", \"The beach DC\", tweet)\n", " # tweet = re.sub(r\"buildingmuseum\", \"building museum\", tweet)\n", " # tweet = re.sub(r\"WorldOil\", \"World Oil\", tweet)\n", " # tweet = re.sub(r\"redwedding\", \"red wedding\", tweet)\n", " # tweet = re.sub(r\"AmazingRaceCanada\", \"Amazing Race Canada\", tweet)\n", " # tweet = re.sub(r\"WakeUpAmerica\", \"Wake Up America\", tweet)\n", " # tweet = re.sub(r\"\\\\Allahuakbar\\\\\", \"Allahu Akbar\", tweet)\n", " # tweet = re.sub(r\"bleased\", \"blessed\", tweet)\n", " # tweet = re.sub(r\"nigeriantribune\", \"Nigerian Tribune\", tweet)\n", " # tweet = re.sub(r\"HIDEO_KOJIMA_EN\", \"Hideo Kojima\", tweet)\n", " # tweet = re.sub(r\"FusionFestival\", \"Fusion Festival\", tweet)\n", " # tweet = re.sub(r\"50Mixed\", \"50 Mixed\", tweet)\n", " # tweet = re.sub(r\"NoAgenda\", \"No Agenda\", tweet)\n", " # tweet = re.sub(r\"WhiteGenocide\", \"White Genocide\", tweet)\n", " # tweet = re.sub(r\"dirtylying\", \"dirty lying\", tweet)\n", " # tweet = re.sub(r\"SyrianRefugees\", \"Syrian Refugees\", tweet)\n", " # tweet = re.sub(r\"changetheworld\", \"change the world\", tweet)\n", " # tweet = re.sub(r\"Ebolacase\", \"Ebola case\", tweet)\n", " # tweet = re.sub(r\"mcgtech\", \"mcg technologies\", tweet)\n", " # tweet = re.sub(r\"withweapons\", \"with weapons\", tweet)\n", " # tweet = re.sub(r\"advancedwarfare\", \"advanced warfare\", tweet)\n", " # tweet = re.sub(r\"letsFootball\", \"let us Football\", tweet)\n", " # tweet = re.sub(r\"LateNiteMix\", \"late night mix\", tweet)\n", " # tweet = re.sub(r\"PhilCollinsFeed\", \"Phil Collins\", tweet)\n", " # tweet = re.sub(r\"RudyHavenstein\", \"Rudy Havenstein\", tweet)\n", " # tweet = re.sub(r\"22PM\", \"22 PM\", tweet)\n", " # tweet = re.sub(r\"54am\", \"54 AM\", tweet)\n", " # tweet = re.sub(r\"38am\", \"38 AM\", tweet)\n", " # tweet = re.sub(r\"OldFolkExplainStuff\", \"Old Folk Explain Stuff\", tweet)\n", " # tweet = re.sub(r\"BlacklivesMatter\", \"Black Lives Matter\", tweet)\n", " # tweet = re.sub(r\"InsaneLimits\", \"Insane Limits\", tweet)\n", " # tweet = re.sub(r\"youcantsitwithus\", \"you cannot sit with us\", tweet)\n", " # tweet = re.sub(r\"2k15\", \"2015\", tweet)\n", " # tweet = re.sub(r\"TheIran\", \"Iran\", tweet)\n", " # tweet = re.sub(r\"JimmyFallon\", \"Jimmy Fallon\", tweet)\n", " # tweet = re.sub(r\"AlbertBrooks\", \"Albert Brooks\", tweet)\n", " # tweet = re.sub(r\"defense_news\", \"defense news\", tweet)\n", " # tweet = re.sub(r\"nuclearrcSA\", \"Nuclear Risk Control Self Assessment\", tweet)\n", " # tweet = re.sub(r\"Auspol\", \"Australia Politics\", tweet)\n", " # tweet = re.sub(r\"NuclearPower\", \"Nuclear Power\", tweet)\n", " # tweet = re.sub(r\"WhiteTerrorism\", \"White Terrorism\", tweet)\n", " # tweet = re.sub(r\"truthfrequencyradio\", \"Truth Frequency Radio\", tweet)\n", " # tweet = re.sub(r\"ErasureIsNotEquality\", \"Erasure is not equality\", tweet)\n", " # tweet = re.sub(r\"ProBonoNews\", \"Pro Bono News\", tweet)\n", " # tweet = re.sub(r\"JakartaPost\", \"Jakarta Post\", tweet)\n", " # tweet = re.sub(r\"toopainful\", \"too painful\", tweet)\n", " # tweet = re.sub(r\"melindahaunton\", \"Melinda Haunton\", tweet)\n", " # tweet = re.sub(r\"NoNukes\", \"No Nukes\", tweet)\n", " # tweet = re.sub(r\"curryspcworld\", \"Currys PC World\", tweet)\n", " # tweet = re.sub(r\"ineedcake\", \"I need cake\", tweet)\n", " # tweet = re.sub(r\"blackforestgateau\", \"black forest gateau\", tweet)\n", " # tweet = re.sub(r\"BBCOne\", \"BBC One\", tweet)\n", " # tweet = re.sub(r\"AlexxPage\", \"Alex Page\", tweet)\n", " # tweet = re.sub(r\"jonathanserrie\", \"Jonathan Serrie\", tweet)\n", " # tweet = re.sub(r\"SocialJerkBlog\", \"Social Jerk Blog\", tweet)\n", " # tweet = re.sub(r\"ChelseaVPeretti\", \"Chelsea Peretti\", tweet)\n", " # tweet = re.sub(r\"irongiant\", \"iron giant\", tweet)\n", " # tweet = re.sub(r\"RonFunches\", \"Ron Funches\", tweet)\n", " # tweet = re.sub(r\"TimCook\", \"Tim Cook\", tweet)\n", " # tweet = re.sub(r\"sebastianstanisaliveandwell\", \"Sebastian Stan is alive and well\", tweet)\n", " # tweet = re.sub(r\"Madsummer\", \"Mad summer\", tweet)\n", " # tweet = re.sub(r\"NowYouKnow\", \"Now you know\", tweet)\n", " # tweet = re.sub(r\"concertphotography\", \"concert photography\", tweet)\n", " # tweet = re.sub(r\"TomLandry\", \"Tom Landry\", tweet)\n", " # tweet = re.sub(r\"showgirldayoff\", \"show girl day off\", tweet)\n", " # tweet = re.sub(r\"Yougslavia\", \"Yugoslavia\", tweet)\n", " # tweet = re.sub(r\"QuantumDataInformatics\", \"Quantum Data Informatics\", tweet)\n", " # tweet = re.sub(r\"FromTheDesk\", \"From The Desk\", tweet)\n", " # tweet = re.sub(r\"TheaterTrial\", \"Theater Trial\", tweet)\n", " # tweet = re.sub(r\"CatoInstitute\", \"Cato Institute\", tweet)\n", " # tweet = re.sub(r\"EmekaGift\", \"Emeka Gift\", tweet)\n", " # tweet = re.sub(r\"LetsBe_Rational\", \"Let us be rational\", tweet)\n", " # tweet = re.sub(r\"Cynicalreality\", \"Cynical reality\", tweet)\n", " # tweet = re.sub(r\"FredOlsenCruise\", \"Fred Olsen Cruise\", tweet)\n", " # tweet = re.sub(r\"NotSorry\", \"not sorry\", tweet)\n", " # tweet = re.sub(r\"UseYourWords\", \"use your words\", tweet)\n", " # tweet = re.sub(r\"WordoftheDay\", \"word of the day\", tweet)\n", " # tweet = re.sub(r\"Dictionarycom\", \"Dictionary.com\", tweet)\n", " # tweet = re.sub(r\"TheBrooklynLife\", \"The Brooklyn Life\", tweet)\n", " # tweet = re.sub(r\"jokethey\", \"joke they\", tweet)\n", " # tweet = re.sub(r\"nflweek1picks\", \"NFL week 1 picks\", tweet)\n", " # tweet = re.sub(r\"uiseful\", \"useful\", tweet)\n", " # tweet = re.sub(r\"JusticeDotOrg\", \"The American Association for Justice\", tweet)\n", " # tweet = re.sub(r\"autoaccidents\", \"auto accidents\", tweet)\n", " # tweet = re.sub(r\"SteveGursten\", \"Steve Gursten\", tweet)\n", " # tweet = re.sub(r\"MichiganAutoLaw\", \"Michigan Auto Law\", tweet)\n", " # tweet = re.sub(r\"birdgang\", \"bird gang\", tweet)\n", " # tweet = re.sub(r\"nflnetwork\", \"NFL Network\", tweet)\n", " # tweet = re.sub(r\"NYDNSports\", \"NY Daily News Sports\", tweet)\n", " # tweet = re.sub(r\"RVacchianoNYDN\", \"Ralph Vacchiano NY Daily News\", tweet)\n", " # tweet = re.sub(r\"EdmontonEsks\", \"Edmonton Eskimos\", tweet)\n", " # tweet = re.sub(r\"david_brelsford\", \"David Brelsford\", tweet)\n", " # tweet = re.sub(r\"TOI_India\", \"The Times of India\", tweet)\n", " # tweet = re.sub(r\"hegot\", \"he got\", tweet)\n", " # tweet = re.sub(r\"SkinsOn9\", \"Skins on 9\", tweet)\n", " # tweet = re.sub(r\"sothathappened\", \"so that happened\", tweet)\n", " # tweet = re.sub(r\"LCOutOfDoors\", \"LC Out Of Doors\", tweet)\n", " # tweet = re.sub(r\"NationFirst\", \"Nation First\", tweet)\n", " # tweet = re.sub(r\"IndiaToday\", \"India Today\", tweet)\n", " # tweet = re.sub(r\"HLPS\", \"helps\", tweet)\n", " # tweet = re.sub(r\"HOSTAGESTHROSW\", \"hostages throw\", tweet)\n", " # tweet = re.sub(r\"SNCTIONS\", \"sanctions\", tweet)\n", " # tweet = re.sub(r\"BidTime\", \"Bid Time\", tweet)\n", " # tweet = re.sub(r\"crunchysensible\", \"crunchy sensible\", tweet)\n", " # tweet = re.sub(r\"RandomActsOfRomance\", \"Random acts of romance\", tweet)\n", " # tweet = re.sub(r\"MomentsAtHill\", \"Moments at hill\", tweet)\n", " # tweet = re.sub(r\"eatshit\", \"eat shit\", tweet)\n", " # tweet = re.sub(r\"liveleakfun\", \"live leak fun\", tweet)\n", " # tweet = re.sub(r\"SahelNews\", \"Sahel News\", tweet)\n", " # tweet = re.sub(r\"abc7newsbayarea\", \"ABC 7 News Bay Area\", tweet)\n", " # tweet = re.sub(r\"facilitiesmanagement\", \"facilities management\", tweet)\n", " # tweet = re.sub(r\"facilitydude\", \"facility dude\", tweet)\n", " # tweet = re.sub(r\"CampLogistics\", \"Camp logistics\", tweet)\n", " # tweet = re.sub(r\"alaskapublic\", \"Alaska public\", tweet)\n", " # tweet = re.sub(r\"MarketResearch\", \"Market Research\", tweet)\n", " # tweet = re.sub(r\"AccuracyEsports\", \"Accuracy Esports\", tweet)\n", " # tweet = re.sub(r\"TheBodyShopAust\", \"The Body Shop Australia\", tweet)\n", " # tweet = re.sub(r\"yychail\", \"Calgary hail\", tweet)\n", " # tweet = re.sub(r\"yyctraffic\", \"Calgary traffic\", tweet)\n", " # tweet = re.sub(r\"eliotschool\", \"eliot school\", tweet)\n", " # tweet = re.sub(r\"TheBrokenCity\", \"The Broken City\", tweet)\n", " # tweet = re.sub(r\"OldsFireDept\", \"Olds Fire Department\", tweet)\n", " # tweet = re.sub(r\"RiverComplex\", \"River Complex\", tweet)\n", " # tweet = re.sub(r\"fieldworksmells\", \"field work smells\", tweet)\n", " # tweet = re.sub(r\"IranElection\", \"Iran Election\", tweet)\n", " # tweet = re.sub(r\"glowng\", \"glowing\", tweet)\n", " # tweet = re.sub(r\"kindlng\", \"kindling\", tweet)\n", " # tweet = re.sub(r\"riggd\", \"rigged\", tweet)\n", " # tweet = re.sub(r\"slownewsday\", \"slow news day\", tweet)\n", " # tweet = re.sub(r\"MyanmarFlood\", \"Myanmar Flood\", tweet)\n", " # tweet = re.sub(r\"abc7chicago\", \"ABC 7 Chicago\", tweet)\n", " # tweet = re.sub(r\"copolitics\", \"Colorado Politics\", tweet)\n", " # tweet = re.sub(r\"AdilGhumro\", \"Adil Ghumro\", tweet)\n", " # tweet = re.sub(r\"netbots\", \"net bots\", tweet)\n", " # tweet = re.sub(r\"byebyeroad\", \"bye bye road\", tweet)\n", " # tweet = re.sub(r\"massiveflooding\", \"massive flooding\", tweet)\n", " # tweet = re.sub(r\"EndofUS\", \"End of United States\", tweet)\n", " # tweet = re.sub(r\"35PM\", \"35 PM\", tweet)\n", " # tweet = re.sub(r\"greektheatrela\", \"Greek Theatre Los Angeles\", tweet)\n", " # tweet = re.sub(r\"76mins\", \"76 minutes\", tweet)\n", " # tweet = re.sub(r\"publicsafetyfirst\", \"public safety first\", tweet)\n", " # tweet = re.sub(r\"livesmatter\", \"lives matter\", tweet)\n", " # tweet = re.sub(r\"myhometown\", \"my hometown\", tweet)\n", " # tweet = re.sub(r\"tankerfire\", \"tanker fire\", tweet)\n", " # tweet = re.sub(r\"MEMORIALDAY\", \"memorial day\", tweet)\n", " # tweet = re.sub(r\"MEMORIAL_DAY\", \"memorial day\", tweet)\n", " # tweet = re.sub(r\"instaxbooty\", \"instagram booty\", tweet)\n", " # tweet = re.sub(r\"Jerusalem_Post\", \"Jerusalem Post\", tweet)\n", " # tweet = re.sub(r\"WayneRooney_INA\", \"Wayne Rooney\", tweet)\n", " # tweet = re.sub(r\"VirtualReality\", \"Virtual Reality\", tweet)\n", " # tweet = re.sub(r\"OculusRift\", \"Oculus Rift\", tweet)\n", " # tweet = re.sub(r\"OwenJones84\", \"Owen Jones\", tweet)\n", " # tweet = re.sub(r\"jeremycorbyn\", \"Jeremy Corbyn\", tweet)\n", " # tweet = re.sub(r\"paulrogers002\", \"Paul Rogers\", tweet)\n", " # tweet = re.sub(r\"mortalkombatx\", \"Mortal Kombat X\", tweet)\n", " # tweet = re.sub(r\"mortalkombat\", \"Mortal Kombat\", tweet)\n", " # tweet = re.sub(r\"FilipeCoelho92\", \"Filipe Coelho\", tweet)\n", " # tweet = re.sub(r\"OnlyQuakeNews\", \"Only Quake News\", tweet)\n", " # tweet = re.sub(r\"kostumes\", \"costumes\", tweet)\n", " # tweet = re.sub(r\"YEEESSSS\", \"yes\", tweet)\n", " # tweet = re.sub(r\"ToshikazuKatayama\", \"Toshikazu Katayama\", tweet)\n", " # tweet = re.sub(r\"IntlDevelopment\", \"Intl Development\", tweet)\n", " # tweet = re.sub(r\"ExtremeWeather\", \"Extreme Weather\", tweet)\n", " # tweet = re.sub(r\"WereNotGruberVoters\", \"We are not gruber voters\", tweet)\n", " # tweet = re.sub(r\"NewsThousands\", \"News Thousands\", tweet)\n", " # tweet = re.sub(r\"EdmundAdamus\", \"Edmund Adamus\", tweet)\n", " # tweet = re.sub(r\"EyewitnessWV\", \"Eye witness WV\", tweet)\n", " # tweet = re.sub(r\"PhiladelphiaMuseu\", \"Philadelphia Museum\", tweet)\n", " # tweet = re.sub(r\"DublinComicCon\", \"Dublin Comic Con\", tweet)\n", " # tweet = re.sub(r\"NicholasBrendon\", \"Nicholas Brendon\", tweet)\n", " # tweet = re.sub(r\"Alltheway80s\", \"All the way 80s\", tweet)\n", " # tweet = re.sub(r\"FromTheField\", \"From the field\", tweet)\n", " # tweet = re.sub(r\"NorthIowa\", \"North Iowa\", tweet)\n", " # tweet = re.sub(r\"WillowFire\", \"Willow Fire\", tweet)\n", " # tweet = re.sub(r\"MadRiverComplex\", \"Mad River Complex\", tweet)\n", " # tweet = re.sub(r\"feelingmanly\", \"feeling manly\", tweet)\n", " # tweet = re.sub(r\"stillnotoverit\", \"still not over it\", tweet)\n", " # tweet = re.sub(r\"FortitudeValley\", \"Fortitude Valley\", tweet)\n", " # tweet = re.sub(r\"CoastpowerlineTramTr\", \"Coast powerline\", tweet)\n", " # tweet = re.sub(r\"ServicesGold\", \"Services Gold\", tweet)\n", " # tweet = re.sub(r\"NewsbrokenEmergency\", \"News broken emergency\", tweet)\n", " # tweet = re.sub(r\"Evaucation\", \"evacuation\", tweet)\n", " # tweet = re.sub(r\"leaveevacuateexitbe\", \"leave evacuate exit be\", tweet)\n", " # tweet = re.sub(r\"P_EOPLE\", \"PEOPLE\", tweet)\n", " # tweet = re.sub(r\"Tubestrike\", \"tube strike\", tweet)\n", " # tweet = re.sub(r\"CLASS_SICK\", \"CLASS SICK\", tweet)\n", " # tweet = re.sub(r\"localplumber\", \"local plumber\", tweet)\n", " # tweet = re.sub(r\"awesomejobsiri\", \"awesome job siri\", tweet)\n", " # tweet = re.sub(r\"PayForItHow\", \"Pay for it how\", tweet)\n", " # tweet = re.sub(r\"ThisIsAfrica\", \"This is Africa\", tweet)\n", " # tweet = re.sub(r\"crimeairnetwork\", \"crime air network\", tweet)\n", " # tweet = re.sub(r\"KimAcheson\", \"Kim Acheson\", tweet)\n", " # tweet = re.sub(r\"cityofcalgary\", \"City of Calgary\", tweet)\n", " # tweet = re.sub(r\"prosyndicate\", \"pro syndicate\", tweet)\n", " # tweet = re.sub(r\"660NEWS\", \"660 NEWS\", tweet)\n", " # tweet = re.sub(r\"BusInsMagazine\", \"Business Insurance Magazine\", tweet)\n", " # tweet = re.sub(r\"wfocus\", \"focus\", tweet)\n", " # tweet = re.sub(r\"ShastaDam\", \"Shasta Dam\", tweet)\n", " # tweet = re.sub(r\"go2MarkFranco\", \"Mark Franco\", tweet)\n", " # tweet = re.sub(r\"StephGHinojosa\", \"Steph Hinojosa\", tweet)\n", " # tweet = re.sub(r\"Nashgrier\", \"Nash Grier\", tweet)\n", " # tweet = re.sub(r\"NashNewVideo\", \"Nash new video\", tweet)\n", " # tweet = re.sub(r\"IWouldntGetElectedBecause\", \"I would not get elected because\", tweet)\n", " # tweet = re.sub(r\"SHGames\", \"Sledgehammer Games\", tweet)\n", " # tweet = re.sub(r\"bedhair\", \"bed hair\", tweet)\n", " # tweet = re.sub(r\"JoelHeyman\", \"Joel Heyman\", tweet)\n", " # tweet = re.sub(r\"viaYouTube\", \"via YouTube\", tweet)\n", " \n", " # Urls\n", " tweet = re.sub(r\"https?:\\/\\/t.co\\/[A-Za-z0-9]+\", \"\", tweet)\n", " \n", " # Words with punctuations and special characters\n", " punctuations = '@#!?+&*[]-%.:/();$=><|{}^' + \"'`\"\n", " for p in punctuations:\n", " tweet = tweet.replace(p, f' {p} ')\n", " \n", " # ... and ..\n", " tweet = tweet.replace('...', ' ... ')\n", " if '...' not in tweet:\n", " tweet = tweet.replace('..', ' ... ') \n", " \n", " # # Acronyms\n", " # tweet = re.sub(r\"MH370\", \"Malaysia Airlines Flight 370\", tweet)\n", " # tweet = re.sub(r\"m̼sica\", \"music\", tweet)\n", " # tweet = re.sub(r\"okwx\", \"Oklahoma City Weather\", tweet)\n", " # tweet = re.sub(r\"arwx\", \"Arkansas Weather\", tweet) \n", " # tweet = re.sub(r\"gawx\", \"Georgia Weather\", tweet) \n", " # tweet = re.sub(r\"scwx\", \"South Carolina Weather\", tweet) \n", " # tweet = re.sub(r\"cawx\", \"California Weather\", tweet)\n", " # tweet = re.sub(r\"tnwx\", \"Tennessee Weather\", tweet)\n", " # tweet = re.sub(r\"azwx\", \"Arizona Weather\", tweet) \n", " # tweet = re.sub(r\"alwx\", \"Alabama Weather\", tweet)\n", " # tweet = re.sub(r\"wordpressdotcom\", \"wordpress\", tweet) \n", " # tweet = re.sub(r\"usNWSgov\", \"United States National Weather Service\", tweet)\n", " # tweet = re.sub(r\"Suruc\", \"Sanliurfa\", tweet) \n", " \n", " # Grouping same words without embeddings\n", " tweet = re.sub(r\"Bestnaijamade\", \"bestnaijamade\", tweet)\n", " tweet = re.sub(r\"SOUDELOR\", \"Soudelor\", tweet)\n", "\n", " tweet = re.sub(r'https?://\\S+|www\\.\\S+',r'',tweet)\n", " tweet = re.sub(r'<.*?>',r'',tweet)\n", " tweet = re.sub(\"[\"\n", " u\"\\U0001F600-\\U0001F64F\" # emoticons\n", " u\"\\U0001F300-\\U0001F5FF\" # symbols & pictographs\n", " u\"\\U0001F680-\\U0001F6FF\" # transport & map symbols\n", " u\"\\U0001F1E0-\\U0001F1FF\" # flags (iOS)\n", " u\"\\U00002702-\\U000027B0\"\n", " u\"\\U000024C2-\\U0001F251\"\n", " \"]+\",r'', tweet, flags=re.UNICODE)\n", " table=str.maketrans('','',string.punctuation)\n", " return tweet.translate(table).lower()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "DAmmT-wyswY8" }, "source": [ "df['clean_text'] = df['text'].apply(lambda x: clean(x))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "2um44dMIswZF" }, "source": [ "df['Product_Label']=df['Product_Labels']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "gkHrgNHoswZO" }, "source": [ "df['Complaint']=df['clean_text']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "yTLWUMZWswZW" }, "source": [ "df['Product']=df['Primary']" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "iBSZyMByswZc", "colab": { "base_uri": "https://localhost:8080/", "height": 392 }, "outputId": "2498e194-092c-4684-9d1b-7c1fd029bb69" }, "source": [ "df" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labelsclean_textProduct_LabelComplaintProduct
01.005030e+05threeSnake oil Salesmen are not gone. They have jus...3snake oil salesmen are not gone they have ju...3snake oil salesmen are not gone they have ju...three
11.009760e+05threeFact check: Gargling water with salt or vinega...3fact check gargling water with salt or vineg...3fact check gargling water with salt or vineg...three
21.011710e+05zero.. this is great advice. Also, stop fighting f...0this is great advice also stop fighting...0this is great advice also stop fighting...zero
31.017920e+05three14 people in Iran died from alcohol poisoning,...314 people in iran died from alcohol poisoning ...314 people in iran died from alcohol poisoning ...three
41.037400e+05oneTo be fair, China is not the only country alle...1to be fair china is not the only country alleg...1to be fair china is not the only country alleg...one
...........................
19651.230000e+18one@TheSecondRevol1 @VincentCrypt46 I have been s...1thesecondrevol1 vincentcrypt46 i have been...1thesecondrevol1 vincentcrypt46 i have been...one
19661.230000e+18oneIf anyone is aware of the 5G theory related to...1if anyone is aware of the 5g theory related to...1if anyone is aware of the 5g theory related to...one
19671.230000e+18one@ClydeLewis @ClydeLewis have you looked into 5...1clydelewis clydelewis have you looked into...1clydelewis clydelewis have you looked into...one
19681.230000e+18oneHow many have the “Virus” where there’s...1how many have the “virus” where there’s...1how many have the “virus” where there’s...one
19691.230000e+18oneSo why am I still so thoroughly stunned by the...1so why am i still so thoroughly stunned by the...1so why am i still so thoroughly stunned by the...one
\n", "

1970 rows × 8 columns

\n", "
" ], "text/plain": [ " ID ... Product\n", "0 1.005030e+05 ... three\n", "1 1.009760e+05 ... three\n", "2 1.011710e+05 ... zero\n", "3 1.017920e+05 ... three\n", "4 1.037400e+05 ... one\n", "... ... ... ...\n", "1965 1.230000e+18 ... one\n", "1966 1.230000e+18 ... one\n", "1967 1.230000e+18 ... one\n", "1968 1.230000e+18 ... one\n", "1969 1.230000e+18 ... one\n", "\n", "[1970 rows x 8 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "code", "metadata": { "id": "yjtcKo9BswZh", "colab": { "base_uri": "https://localhost:8080/", "height": 334 }, "outputId": "426497f2-7f8e-44ba-fc7a-b172ecf32075" }, "source": [ "df.head(10)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDPrimarytextProduct_Labelsclean_textProduct_LabelComplaintProduct
0100503.0threeSnake oil Salesmen are not gone. They have jus...3snake oil salesmen are not gone they have ju...3snake oil salesmen are not gone they have ju...three
1100976.0threeFact check: Gargling water with salt or vinega...3fact check gargling water with salt or vineg...3fact check gargling water with salt or vineg...three
2101171.0zero.. this is great advice. Also, stop fighting f...0this is great advice also stop fighting...0this is great advice also stop fighting...zero
3101792.0three14 people in Iran died from alcohol poisoning,...314 people in iran died from alcohol poisoning ...314 people in iran died from alcohol poisoning ...three
4103740.0oneTo be fair, China is not the only country alle...1to be fair china is not the only country alleg...1to be fair china is not the only country alleg...one
5108359.0threeCocaine does NOT cure #coronavirus: French gov...3cocaine does not cure coronavirus french g...3cocaine does not cure coronavirus french g...three
6108430.0three2008 Research paper demonstrating various esse...32008 research paper demonstrating various esse...32008 research paper demonstrating various esse...three
7110352.0three@SwerveChris @FakeNewsRehab @RudyGiuliani http...3swervechris fakenewsrehab rudygiuliani ...3swervechris fakenewsrehab rudygiuliani ...three
8110704.0three@wevarts @oliverdarcy @RudyGiuliani @charlieki...3wevarts oliverdarcy rudygiuliani charl...3wevarts oliverdarcy rudygiuliani charl...three
9110956.0zeroWasn't it great when our only health concerns ...0wasn t it great when our only health concerns...0wasn t it great when our only health concerns...zero
\n", "
" ], "text/plain": [ " ID Primary ... Complaint Product\n", "0 100503.0 three ... snake oil salesmen are not gone they have ju... three\n", "1 100976.0 three ... fact check gargling water with salt or vineg... three\n", "2 101171.0 zero ... this is great advice also stop fighting... zero\n", "3 101792.0 three ... 14 people in iran died from alcohol poisoning ... three\n", "4 103740.0 one ... to be fair china is not the only country alleg... one\n", "5 108359.0 three ... cocaine does not cure coronavirus french g... three\n", "6 108430.0 three ... 2008 research paper demonstrating various esse... three\n", "7 110352.0 three ... swervechris fakenewsrehab rudygiuliani ... three\n", "8 110704.0 three ... wevarts oliverdarcy rudygiuliani charl... three\n", "9 110956.0 zero ... wasn t it great when our only health concerns... zero\n", "\n", "[10 rows x 8 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 12 } ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "4E1fzftxswZm", "colab": { "base_uri": "https://localhost:8080/", "height": 162 }, "outputId": "9d43ffb8-a884-4d37-9676-2423f482b207" }, "source": [ "label_counts = pd.DataFrame(df['Product'].value_counts())\n", "label_counts" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Product
zero768
two468
one462
three272
\n", "
" ], "text/plain": [ " Product\n", "zero 768\n", "two 468\n", "one 462\n", "three 272" ] }, "metadata": { "tags": [] }, "execution_count": 13 } ] }, { "cell_type": "markdown", "metadata": { "id": "2sKrwbrpswZq" }, "source": [ "Here we create an array with the label names in the order they were numerically encoded. We use them later when plotting model performance data." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "0nSXK223swZr", "colab": { "base_uri": "https://localhost:8080/", "height": 33 }, "outputId": "391c380c-533e-4afb-a2e4-718aa707f209" }, "source": [ "label_values = list(label_counts.index)\n", "order = list(pd.DataFrame(df['Product_Label'].value_counts()).index)\n", "label_values = [l for _,l in sorted(zip(order, label_values))]\n", "\n", "label_values" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['zero', 'one', 'two', 'three']" ] }, "metadata": { "tags": [] }, "execution_count": 14 } ] }, { "cell_type": "markdown", "metadata": { "id": "SBgQ_U1lswZv" }, "source": [ "We need to create 2 arrays: one with the textual data, which is our feature data, and one with the numerically encoded labels, representing our target data." ] }, { "cell_type": "code", "metadata": { "id": "ubPd1LLgswZw" }, "source": [ "texts = df['Complaint'].values\n", "labels = df['Product_Label'].values" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "p22S4mqRswZz" }, "source": [ "BERT is a ‘heavy-weight’´model. This makes the training a very resource-intensive process, specially when we are fine-tuning for all model layers. To mitigate this, we can control the sequence length of our input text, which is given by the number of tokens in our input text, plus 2 special tokens to mark the beginning and ending of a text sequence." ] }, { "cell_type": "code", "metadata": { "id": "xN_9gxE1swZ0", "colab": { "base_uri": "https://localhost:8080/", "height": 50 }, "outputId": "8ec64441-df27-4da5-9480-74f1a2064463" }, "source": [ "text_lengths = [len(texts[i].split()) for i in range(len(texts))]\n", "print(min(text_lengths))\n", "print(max(text_lengths))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "1\n", "90\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "RUxWAmWuswZ3", "colab": { "base_uri": "https://localhost:8080/", "height": 33 }, "outputId": "d242e2b0-eee7-475c-b8b7-fc0109a0c2d5" }, "source": [ "sum([1 for i in range(len(text_lengths)) if text_lengths[i] >= 55])" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "7" ] }, "metadata": { "tags": [] }, "execution_count": 17 } ] }, { "cell_type": "markdown", "metadata": { "id": "Ac5M_helswZ7" }, "source": [ "Here we instantiate a BERT tokenizer and show an example of a tokenized text. " ] }, { "cell_type": "code", "metadata": { "scrolled": false, "id": "242t3rCfswZ7", "colab": { "base_uri": "https://localhost:8080/", "height": 120 }, "outputId": "4e9d106d-3e73-4d94-af1d-df4b580b540a" }, "source": [ "from transformers import BertTokenizer\n", "tokenizer = BertTokenizer.from_pretrained('bert-large-uncased', do_lower_case=True)\n", "\n", "\n", "\n", "print('Original Text: ', texts[0], '\\n')\n", "print('Tokenized Text: ', tokenizer.tokenize(texts[0]), '\\n')\n", "print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(texts[0])))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Original Text: snake oil salesmen are not gone they have just replaced the horse cart with twitter accounts and websites got an fda warning letter for advertising and selling the most powerful essential oil that can treat and defend against covid19 stay safe \n", "\n", "Tokenized Text: ['snake', 'oil', 'sales', '##men', 'are', 'not', 'gone', 'they', 'have', 'just', 'replaced', 'the', 'horse', 'cart', 'with', 'twitter', 'accounts', 'and', 'websites', 'got', 'an', 'fda', 'warning', 'letter', 'for', 'advertising', 'and', 'selling', 'the', 'most', 'powerful', 'essential', 'oil', 'that', 'can', 'treat', 'and', 'defend', 'against', 'co', '##vid', '##19', 'stay', 'safe'] \n", "\n", "Token IDs: [7488, 3514, 4341, 3549, 2024, 2025, 2908, 2027, 2031, 2074, 2999, 1996, 3586, 11122, 2007, 10474, 6115, 1998, 11744, 2288, 2019, 17473, 5432, 3661, 2005, 6475, 1998, 4855, 1996, 2087, 3928, 6827, 3514, 2008, 2064, 7438, 1998, 6985, 2114, 2522, 17258, 16147, 2994, 3647]\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "CpRZdV5lswZ-" }, "source": [ "We then tokenize and encode the entire dataset. In this process, we perform the following:\n", "- tokenize the text as shown above\n", "- encode it to the corresponding numeric values for each token.\n", "- truncate it to the maximum sequence length of 55.\n", "- pad the tokens positions greater than 55.\n", "- include the special token IDs to mark the beginning and end of each sequence." ] }, { "cell_type": "code", "metadata": { "id": "KbhImXcrswZ_", "colab": { "base_uri": "https://localhost:8080/", "height": 940 }, "outputId": "9913e71f-0649-4dd6-c1a4-547d5623520d" }, "source": [ "text_ids = [tokenizer.encode(text, max_length=55, pad_to_max_length=True) for text in texts]\n", "\n", "text_ids[0]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[101,\n", " 7488,\n", " 3514,\n", " 4341,\n", " 3549,\n", " 2024,\n", " 2025,\n", " 2908,\n", " 2027,\n", " 2031,\n", " 2074,\n", " 2999,\n", " 1996,\n", " 3586,\n", " 11122,\n", " 2007,\n", " 10474,\n", " 6115,\n", " 1998,\n", " 11744,\n", " 2288,\n", " 2019,\n", " 17473,\n", " 5432,\n", " 3661,\n", " 2005,\n", " 6475,\n", " 1998,\n", " 4855,\n", " 1996,\n", " 2087,\n", " 3928,\n", " 6827,\n", " 3514,\n", " 2008,\n", " 2064,\n", " 7438,\n", " 1998,\n", " 6985,\n", " 2114,\n", " 2522,\n", " 17258,\n", " 16147,\n", " 2994,\n", " 3647,\n", " 102,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0]" ] }, "metadata": { "tags": [] }, "execution_count": 19 } ] }, { "cell_type": "code", "metadata": { "id": "SciI37HgswaC", "colab": { "base_uri": "https://localhost:8080/", "height": 50 }, "outputId": "2c609feb-4139-44c3-a258-36c879d21cd0" }, "source": [ "text_ids_lengths = [len(text_ids[i]) for i in range(len(text_ids))]\n", "print(min(text_ids_lengths))\n", "print(max(text_ids_lengths))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "55\n", "55\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "w53rUNWQswaE" }, "source": [ "To fine-tune our model, we need two inputs: one array of token IDs (created above) and one array of a corresponding binary mask, called attention mask in the BERT model specification. Each attention mask has the same length of the corresponding input sequence and has a 0 if the corresponding token is a pad token, or a 1 otherwise. " ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "3jM33Yu3swaF", "colab": { "base_uri": "https://localhost:8080/", "height": 940 }, "outputId": "b2b2a39b-ad80-47ec-d406-14a640d3426c" }, "source": [ "att_masks = []\n", "for ids in text_ids:\n", " masks = [int(id > 0) for id in ids]\n", " att_masks.append(masks)\n", " \n", "att_masks[0]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 1,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0,\n", " 0]" ] }, "metadata": { "tags": [] }, "execution_count": 21 } ] }, { "cell_type": "markdown", "metadata": { "id": "dQKTh7KFswaI" }, "source": [ "Here we split the input and output arrays created before into train, validation, and test sets. We use 80% of the data for training, 10% for training validation, and 10% for final testing." ] }, { "cell_type": "code", "metadata": { "id": "D_tuwpjGswaJ" }, "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "train_x, test_val_x, train_y, test_val_y = train_test_split(text_ids, labels, random_state=11, test_size=0.2, stratify=labels)\n", "train_m, test_val_m = train_test_split(att_masks, random_state=11, test_size=0.2)\n", "\n", "test_x, val_x, test_y, val_y = train_test_split(test_val_x, test_val_y, random_state=11, test_size=0.5, stratify=test_val_y)\n", "test_m, val_m = train_test_split(test_val_m, random_state=11, test_size=0.5)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "ezUCLaePswaL" }, "source": [ "We are working with the PyTorch artifacts in the transformers library, therefore we need our model input and output data as PyTorch tensors." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "CeVB-yB-swaM", "colab": { "base_uri": "https://localhost:8080/", "height": 168 }, "outputId": "fe9b559b-65b8-4ea6-b872-f66cc56f74c6" }, "source": [ "import torch\n", "\n", "train_x = torch.tensor(train_x)\n", "test_x = torch.tensor(test_x)\n", "val_x = torch.tensor(val_x)\n", "train_y = torch.tensor(train_y)\n", "test_y = torch.tensor(test_y)\n", "val_y = torch.tensor(val_y)\n", "train_m = torch.tensor(train_m)\n", "test_m = torch.tensor(test_m)\n", "val_m = torch.tensor(val_m)\n", "\n", "print(train_x.shape)\n", "print(test_x.shape)\n", "print(val_x.shape)\n", "print(train_y.shape)\n", "print(test_y.shape)\n", "print(val_y.shape)\n", "print(train_m.shape)\n", "print(test_m.shape)\n", "print(val_m.shape)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "torch.Size([1576, 55])\n", "torch.Size([197, 55])\n", "torch.Size([197, 55])\n", "torch.Size([1576])\n", "torch.Size([197])\n", "torch.Size([197])\n", "torch.Size([1576, 55])\n", "torch.Size([197, 55])\n", "torch.Size([197, 55])\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "3GKJS6X5swaO" }, "source": [ "To feed data into the model for training, we use Pytorch’s Dataset, DataLoader, and Sampler. For feeding training data, which drives model weights updates, we use the RandomSampler. For feeding the validation data we can use the SequentialSampler." ] }, { "cell_type": "code", "metadata": { "id": "O45f2jUqswaP", "colab": { "base_uri": "https://localhost:8080/", "height": 352 }, "outputId": "c68750b2-f280-4c79-adae-1cb7b5acc4af" }, "source": [ " # Checking if GPU is available or not\n", " !nvidia-smi" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Tue Oct 20 06:32:24 2020 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 455.23.05 Driver Version: 418.67 CUDA Version: 10.1 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n", "| N/A 53C P8 10W / 70W | 0MiB / 15079MiB | 0% Default |\n", "| | | ERR! |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "| No running processes found |\n", "+-----------------------------------------------------------------------------+\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "pJHvcnzTswaS" }, "source": [ "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n", "\n", "batch_size = 8\n", "\n", "train_data = TensorDataset(train_x, train_m, train_y)\n", "train_sampler = RandomSampler(train_data)\n", "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n", "\n", "val_data = TensorDataset(val_x, val_m, val_y)\n", "val_sampler = SequentialSampler(val_data)\n", "val_dataloader = DataLoader(val_data, sampler=val_sampler, batch_size=batch_size)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "TjXEQ3xyswaW" }, "source": [ "Here we instantiate our model class. We use a compact version, that is trained through model distillation from a base BERT model and modified to include a classification layer at the output. This compact version has 6 transformer layers instead of 12 as in the original BERT model. Please see [here]( https://github.com/huggingface/transformers/tree/master/examples/distillation) for more details." ] }, { "cell_type": "code", "metadata": { "id": "9HEkY8aqswaX" }, "source": [ "from transformers import BertForSequenceClassification, BertConfig, AdamW\n", "\n", "num_labels = len(set(labels))\n", "\n", "model = BertForSequenceClassification.from_pretrained('bert-large-uncased', num_labels=num_labels,\n", " output_attentions=False, output_hidden_states=False)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ZadgdKuEswaa", "colab": { "base_uri": "https://localhost:8080/", "height": 33 }, "outputId": "4fb4d6cb-0a48-43bb-8218-83574878cae3" }, "source": [ "num_labels" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "4" ] }, "metadata": { "tags": [] }, "execution_count": 27 } ] }, { "cell_type": "markdown", "metadata": { "id": "NBdPmfVKswac" }, "source": [ "BERT is a very large model. Unless you are freezing model weights in all layers but the classification layer, it is recommended to train it on a GPU." ] }, { "cell_type": "code", "metadata": { "id": "u7sEOkBxswad", "colab": { "base_uri": "https://localhost:8080/", "height": 33 }, "outputId": "ca579899-97af-4d93-933c-02925600afdb" }, "source": [ "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "print(device)\n", "model = model.to(device)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "cuda\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "8rS9gITLswag" }, "source": [ "Here we print the model architecture and all model learnable parameters." ] }, { "cell_type": "code", "metadata": { "id": "xSho9YLmswag", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "0d4e54d8-a43d-4cc9-c31e-4286924eecb3" }, "source": [ "def count_parameters(model):\n", " return sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "\n", "print('Number of trainable parameters:', count_parameters(model), '\\n', model)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Number of trainable parameters: 335145988 \n", " BertForSequenceClassification(\n", " (bert): BertModel(\n", " (embeddings): BertEmbeddings(\n", " (word_embeddings): Embedding(30522, 1024, padding_idx=0)\n", " (position_embeddings): Embedding(512, 1024)\n", " (token_type_embeddings): Embedding(2, 1024)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (encoder): BertEncoder(\n", " (layer): ModuleList(\n", " (0): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (1): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (2): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (3): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (4): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (5): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (6): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (7): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (8): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (9): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (10): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (11): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (12): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (13): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (14): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (15): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (16): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (17): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (18): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (19): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (20): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (21): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (22): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (23): BertLayer(\n", " (attention): BertAttention(\n", " (self): BertSelfAttention(\n", " (query): Linear(in_features=1024, out_features=1024, bias=True)\n", " (key): Linear(in_features=1024, out_features=1024, bias=True)\n", " (value): Linear(in_features=1024, out_features=1024, bias=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " (output): BertSelfOutput(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " (intermediate): BertIntermediate(\n", " (dense): Linear(in_features=1024, out_features=4096, bias=True)\n", " )\n", " (output): BertOutput(\n", " (dense): Linear(in_features=4096, out_features=1024, bias=True)\n", " (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " )\n", " )\n", " )\n", " )\n", " (pooler): BertPooler(\n", " (dense): Linear(in_features=1024, out_features=1024, bias=True)\n", " (activation): Tanh()\n", " )\n", " )\n", " (dropout): Dropout(p=0.1, inplace=False)\n", " (classifier): Linear(in_features=1024, out_features=4, bias=True)\n", ")\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "K6IwlgViswaj", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "ddf7e1e2-14dd-4257-f04d-e3b57f64fe8e" }, "source": [ "[n for n, p in model.named_parameters()]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['bert.embeddings.word_embeddings.weight',\n", " 'bert.embeddings.position_embeddings.weight',\n", " 'bert.embeddings.token_type_embeddings.weight',\n", " 'bert.embeddings.LayerNorm.weight',\n", " 'bert.embeddings.LayerNorm.bias',\n", " 'bert.encoder.layer.0.attention.self.query.weight',\n", " 'bert.encoder.layer.0.attention.self.query.bias',\n", " 'bert.encoder.layer.0.attention.self.key.weight',\n", " 'bert.encoder.layer.0.attention.self.key.bias',\n", " 'bert.encoder.layer.0.attention.self.value.weight',\n", " 'bert.encoder.layer.0.attention.self.value.bias',\n", " 'bert.encoder.layer.0.attention.output.dense.weight',\n", " 'bert.encoder.layer.0.attention.output.dense.bias',\n", " 'bert.encoder.layer.0.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.0.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.0.intermediate.dense.weight',\n", " 'bert.encoder.layer.0.intermediate.dense.bias',\n", " 'bert.encoder.layer.0.output.dense.weight',\n", " 'bert.encoder.layer.0.output.dense.bias',\n", " 'bert.encoder.layer.0.output.LayerNorm.weight',\n", " 'bert.encoder.layer.0.output.LayerNorm.bias',\n", " 'bert.encoder.layer.1.attention.self.query.weight',\n", " 'bert.encoder.layer.1.attention.self.query.bias',\n", " 'bert.encoder.layer.1.attention.self.key.weight',\n", " 'bert.encoder.layer.1.attention.self.key.bias',\n", " 'bert.encoder.layer.1.attention.self.value.weight',\n", " 'bert.encoder.layer.1.attention.self.value.bias',\n", " 'bert.encoder.layer.1.attention.output.dense.weight',\n", " 'bert.encoder.layer.1.attention.output.dense.bias',\n", " 'bert.encoder.layer.1.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.1.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.1.intermediate.dense.weight',\n", " 'bert.encoder.layer.1.intermediate.dense.bias',\n", " 'bert.encoder.layer.1.output.dense.weight',\n", " 'bert.encoder.layer.1.output.dense.bias',\n", " 'bert.encoder.layer.1.output.LayerNorm.weight',\n", " 'bert.encoder.layer.1.output.LayerNorm.bias',\n", " 'bert.encoder.layer.2.attention.self.query.weight',\n", " 'bert.encoder.layer.2.attention.self.query.bias',\n", " 'bert.encoder.layer.2.attention.self.key.weight',\n", " 'bert.encoder.layer.2.attention.self.key.bias',\n", " 'bert.encoder.layer.2.attention.self.value.weight',\n", " 'bert.encoder.layer.2.attention.self.value.bias',\n", " 'bert.encoder.layer.2.attention.output.dense.weight',\n", " 'bert.encoder.layer.2.attention.output.dense.bias',\n", " 'bert.encoder.layer.2.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.2.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.2.intermediate.dense.weight',\n", " 'bert.encoder.layer.2.intermediate.dense.bias',\n", " 'bert.encoder.layer.2.output.dense.weight',\n", " 'bert.encoder.layer.2.output.dense.bias',\n", " 'bert.encoder.layer.2.output.LayerNorm.weight',\n", " 'bert.encoder.layer.2.output.LayerNorm.bias',\n", " 'bert.encoder.layer.3.attention.self.query.weight',\n", " 'bert.encoder.layer.3.attention.self.query.bias',\n", " 'bert.encoder.layer.3.attention.self.key.weight',\n", " 'bert.encoder.layer.3.attention.self.key.bias',\n", " 'bert.encoder.layer.3.attention.self.value.weight',\n", " 'bert.encoder.layer.3.attention.self.value.bias',\n", " 'bert.encoder.layer.3.attention.output.dense.weight',\n", " 'bert.encoder.layer.3.attention.output.dense.bias',\n", " 'bert.encoder.layer.3.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.3.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.3.intermediate.dense.weight',\n", " 'bert.encoder.layer.3.intermediate.dense.bias',\n", " 'bert.encoder.layer.3.output.dense.weight',\n", " 'bert.encoder.layer.3.output.dense.bias',\n", " 'bert.encoder.layer.3.output.LayerNorm.weight',\n", " 'bert.encoder.layer.3.output.LayerNorm.bias',\n", " 'bert.encoder.layer.4.attention.self.query.weight',\n", " 'bert.encoder.layer.4.attention.self.query.bias',\n", " 'bert.encoder.layer.4.attention.self.key.weight',\n", " 'bert.encoder.layer.4.attention.self.key.bias',\n", " 'bert.encoder.layer.4.attention.self.value.weight',\n", " 'bert.encoder.layer.4.attention.self.value.bias',\n", " 'bert.encoder.layer.4.attention.output.dense.weight',\n", " 'bert.encoder.layer.4.attention.output.dense.bias',\n", " 'bert.encoder.layer.4.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.4.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.4.intermediate.dense.weight',\n", " 'bert.encoder.layer.4.intermediate.dense.bias',\n", " 'bert.encoder.layer.4.output.dense.weight',\n", " 'bert.encoder.layer.4.output.dense.bias',\n", " 'bert.encoder.layer.4.output.LayerNorm.weight',\n", " 'bert.encoder.layer.4.output.LayerNorm.bias',\n", " 'bert.encoder.layer.5.attention.self.query.weight',\n", " 'bert.encoder.layer.5.attention.self.query.bias',\n", " 'bert.encoder.layer.5.attention.self.key.weight',\n", " 'bert.encoder.layer.5.attention.self.key.bias',\n", " 'bert.encoder.layer.5.attention.self.value.weight',\n", " 'bert.encoder.layer.5.attention.self.value.bias',\n", " 'bert.encoder.layer.5.attention.output.dense.weight',\n", " 'bert.encoder.layer.5.attention.output.dense.bias',\n", " 'bert.encoder.layer.5.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.5.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.5.intermediate.dense.weight',\n", " 'bert.encoder.layer.5.intermediate.dense.bias',\n", " 'bert.encoder.layer.5.output.dense.weight',\n", " 'bert.encoder.layer.5.output.dense.bias',\n", " 'bert.encoder.layer.5.output.LayerNorm.weight',\n", " 'bert.encoder.layer.5.output.LayerNorm.bias',\n", " 'bert.encoder.layer.6.attention.self.query.weight',\n", " 'bert.encoder.layer.6.attention.self.query.bias',\n", " 'bert.encoder.layer.6.attention.self.key.weight',\n", " 'bert.encoder.layer.6.attention.self.key.bias',\n", " 'bert.encoder.layer.6.attention.self.value.weight',\n", " 'bert.encoder.layer.6.attention.self.value.bias',\n", " 'bert.encoder.layer.6.attention.output.dense.weight',\n", " 'bert.encoder.layer.6.attention.output.dense.bias',\n", " 'bert.encoder.layer.6.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.6.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.6.intermediate.dense.weight',\n", " 'bert.encoder.layer.6.intermediate.dense.bias',\n", " 'bert.encoder.layer.6.output.dense.weight',\n", " 'bert.encoder.layer.6.output.dense.bias',\n", " 'bert.encoder.layer.6.output.LayerNorm.weight',\n", " 'bert.encoder.layer.6.output.LayerNorm.bias',\n", " 'bert.encoder.layer.7.attention.self.query.weight',\n", " 'bert.encoder.layer.7.attention.self.query.bias',\n", " 'bert.encoder.layer.7.attention.self.key.weight',\n", " 'bert.encoder.layer.7.attention.self.key.bias',\n", " 'bert.encoder.layer.7.attention.self.value.weight',\n", " 'bert.encoder.layer.7.attention.self.value.bias',\n", " 'bert.encoder.layer.7.attention.output.dense.weight',\n", " 'bert.encoder.layer.7.attention.output.dense.bias',\n", " 'bert.encoder.layer.7.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.7.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.7.intermediate.dense.weight',\n", " 'bert.encoder.layer.7.intermediate.dense.bias',\n", " 'bert.encoder.layer.7.output.dense.weight',\n", " 'bert.encoder.layer.7.output.dense.bias',\n", " 'bert.encoder.layer.7.output.LayerNorm.weight',\n", " 'bert.encoder.layer.7.output.LayerNorm.bias',\n", " 'bert.encoder.layer.8.attention.self.query.weight',\n", " 'bert.encoder.layer.8.attention.self.query.bias',\n", " 'bert.encoder.layer.8.attention.self.key.weight',\n", " 'bert.encoder.layer.8.attention.self.key.bias',\n", " 'bert.encoder.layer.8.attention.self.value.weight',\n", " 'bert.encoder.layer.8.attention.self.value.bias',\n", " 'bert.encoder.layer.8.attention.output.dense.weight',\n", " 'bert.encoder.layer.8.attention.output.dense.bias',\n", " 'bert.encoder.layer.8.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.8.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.8.intermediate.dense.weight',\n", " 'bert.encoder.layer.8.intermediate.dense.bias',\n", " 'bert.encoder.layer.8.output.dense.weight',\n", " 'bert.encoder.layer.8.output.dense.bias',\n", " 'bert.encoder.layer.8.output.LayerNorm.weight',\n", " 'bert.encoder.layer.8.output.LayerNorm.bias',\n", " 'bert.encoder.layer.9.attention.self.query.weight',\n", " 'bert.encoder.layer.9.attention.self.query.bias',\n", " 'bert.encoder.layer.9.attention.self.key.weight',\n", " 'bert.encoder.layer.9.attention.self.key.bias',\n", " 'bert.encoder.layer.9.attention.self.value.weight',\n", " 'bert.encoder.layer.9.attention.self.value.bias',\n", " 'bert.encoder.layer.9.attention.output.dense.weight',\n", " 'bert.encoder.layer.9.attention.output.dense.bias',\n", " 'bert.encoder.layer.9.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.9.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.9.intermediate.dense.weight',\n", " 'bert.encoder.layer.9.intermediate.dense.bias',\n", " 'bert.encoder.layer.9.output.dense.weight',\n", " 'bert.encoder.layer.9.output.dense.bias',\n", " 'bert.encoder.layer.9.output.LayerNorm.weight',\n", " 'bert.encoder.layer.9.output.LayerNorm.bias',\n", " 'bert.encoder.layer.10.attention.self.query.weight',\n", " 'bert.encoder.layer.10.attention.self.query.bias',\n", " 'bert.encoder.layer.10.attention.self.key.weight',\n", " 'bert.encoder.layer.10.attention.self.key.bias',\n", " 'bert.encoder.layer.10.attention.self.value.weight',\n", " 'bert.encoder.layer.10.attention.self.value.bias',\n", " 'bert.encoder.layer.10.attention.output.dense.weight',\n", " 'bert.encoder.layer.10.attention.output.dense.bias',\n", " 'bert.encoder.layer.10.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.10.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.10.intermediate.dense.weight',\n", " 'bert.encoder.layer.10.intermediate.dense.bias',\n", " 'bert.encoder.layer.10.output.dense.weight',\n", " 'bert.encoder.layer.10.output.dense.bias',\n", " 'bert.encoder.layer.10.output.LayerNorm.weight',\n", " 'bert.encoder.layer.10.output.LayerNorm.bias',\n", " 'bert.encoder.layer.11.attention.self.query.weight',\n", " 'bert.encoder.layer.11.attention.self.query.bias',\n", " 'bert.encoder.layer.11.attention.self.key.weight',\n", " 'bert.encoder.layer.11.attention.self.key.bias',\n", " 'bert.encoder.layer.11.attention.self.value.weight',\n", " 'bert.encoder.layer.11.attention.self.value.bias',\n", " 'bert.encoder.layer.11.attention.output.dense.weight',\n", " 'bert.encoder.layer.11.attention.output.dense.bias',\n", " 'bert.encoder.layer.11.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.11.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.11.intermediate.dense.weight',\n", " 'bert.encoder.layer.11.intermediate.dense.bias',\n", " 'bert.encoder.layer.11.output.dense.weight',\n", " 'bert.encoder.layer.11.output.dense.bias',\n", " 'bert.encoder.layer.11.output.LayerNorm.weight',\n", " 'bert.encoder.layer.11.output.LayerNorm.bias',\n", " 'bert.encoder.layer.12.attention.self.query.weight',\n", " 'bert.encoder.layer.12.attention.self.query.bias',\n", " 'bert.encoder.layer.12.attention.self.key.weight',\n", " 'bert.encoder.layer.12.attention.self.key.bias',\n", " 'bert.encoder.layer.12.attention.self.value.weight',\n", " 'bert.encoder.layer.12.attention.self.value.bias',\n", " 'bert.encoder.layer.12.attention.output.dense.weight',\n", " 'bert.encoder.layer.12.attention.output.dense.bias',\n", " 'bert.encoder.layer.12.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.12.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.12.intermediate.dense.weight',\n", " 'bert.encoder.layer.12.intermediate.dense.bias',\n", " 'bert.encoder.layer.12.output.dense.weight',\n", " 'bert.encoder.layer.12.output.dense.bias',\n", " 'bert.encoder.layer.12.output.LayerNorm.weight',\n", " 'bert.encoder.layer.12.output.LayerNorm.bias',\n", " 'bert.encoder.layer.13.attention.self.query.weight',\n", " 'bert.encoder.layer.13.attention.self.query.bias',\n", " 'bert.encoder.layer.13.attention.self.key.weight',\n", " 'bert.encoder.layer.13.attention.self.key.bias',\n", " 'bert.encoder.layer.13.attention.self.value.weight',\n", " 'bert.encoder.layer.13.attention.self.value.bias',\n", " 'bert.encoder.layer.13.attention.output.dense.weight',\n", " 'bert.encoder.layer.13.attention.output.dense.bias',\n", " 'bert.encoder.layer.13.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.13.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.13.intermediate.dense.weight',\n", " 'bert.encoder.layer.13.intermediate.dense.bias',\n", " 'bert.encoder.layer.13.output.dense.weight',\n", " 'bert.encoder.layer.13.output.dense.bias',\n", " 'bert.encoder.layer.13.output.LayerNorm.weight',\n", " 'bert.encoder.layer.13.output.LayerNorm.bias',\n", " 'bert.encoder.layer.14.attention.self.query.weight',\n", " 'bert.encoder.layer.14.attention.self.query.bias',\n", " 'bert.encoder.layer.14.attention.self.key.weight',\n", " 'bert.encoder.layer.14.attention.self.key.bias',\n", " 'bert.encoder.layer.14.attention.self.value.weight',\n", " 'bert.encoder.layer.14.attention.self.value.bias',\n", " 'bert.encoder.layer.14.attention.output.dense.weight',\n", " 'bert.encoder.layer.14.attention.output.dense.bias',\n", " 'bert.encoder.layer.14.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.14.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.14.intermediate.dense.weight',\n", " 'bert.encoder.layer.14.intermediate.dense.bias',\n", " 'bert.encoder.layer.14.output.dense.weight',\n", " 'bert.encoder.layer.14.output.dense.bias',\n", " 'bert.encoder.layer.14.output.LayerNorm.weight',\n", " 'bert.encoder.layer.14.output.LayerNorm.bias',\n", " 'bert.encoder.layer.15.attention.self.query.weight',\n", " 'bert.encoder.layer.15.attention.self.query.bias',\n", " 'bert.encoder.layer.15.attention.self.key.weight',\n", " 'bert.encoder.layer.15.attention.self.key.bias',\n", " 'bert.encoder.layer.15.attention.self.value.weight',\n", " 'bert.encoder.layer.15.attention.self.value.bias',\n", " 'bert.encoder.layer.15.attention.output.dense.weight',\n", " 'bert.encoder.layer.15.attention.output.dense.bias',\n", " 'bert.encoder.layer.15.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.15.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.15.intermediate.dense.weight',\n", " 'bert.encoder.layer.15.intermediate.dense.bias',\n", " 'bert.encoder.layer.15.output.dense.weight',\n", " 'bert.encoder.layer.15.output.dense.bias',\n", " 'bert.encoder.layer.15.output.LayerNorm.weight',\n", " 'bert.encoder.layer.15.output.LayerNorm.bias',\n", " 'bert.encoder.layer.16.attention.self.query.weight',\n", " 'bert.encoder.layer.16.attention.self.query.bias',\n", " 'bert.encoder.layer.16.attention.self.key.weight',\n", " 'bert.encoder.layer.16.attention.self.key.bias',\n", " 'bert.encoder.layer.16.attention.self.value.weight',\n", " 'bert.encoder.layer.16.attention.self.value.bias',\n", " 'bert.encoder.layer.16.attention.output.dense.weight',\n", " 'bert.encoder.layer.16.attention.output.dense.bias',\n", " 'bert.encoder.layer.16.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.16.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.16.intermediate.dense.weight',\n", " 'bert.encoder.layer.16.intermediate.dense.bias',\n", " 'bert.encoder.layer.16.output.dense.weight',\n", " 'bert.encoder.layer.16.output.dense.bias',\n", " 'bert.encoder.layer.16.output.LayerNorm.weight',\n", " 'bert.encoder.layer.16.output.LayerNorm.bias',\n", " 'bert.encoder.layer.17.attention.self.query.weight',\n", " 'bert.encoder.layer.17.attention.self.query.bias',\n", " 'bert.encoder.layer.17.attention.self.key.weight',\n", " 'bert.encoder.layer.17.attention.self.key.bias',\n", " 'bert.encoder.layer.17.attention.self.value.weight',\n", " 'bert.encoder.layer.17.attention.self.value.bias',\n", " 'bert.encoder.layer.17.attention.output.dense.weight',\n", " 'bert.encoder.layer.17.attention.output.dense.bias',\n", " 'bert.encoder.layer.17.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.17.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.17.intermediate.dense.weight',\n", " 'bert.encoder.layer.17.intermediate.dense.bias',\n", " 'bert.encoder.layer.17.output.dense.weight',\n", " 'bert.encoder.layer.17.output.dense.bias',\n", " 'bert.encoder.layer.17.output.LayerNorm.weight',\n", " 'bert.encoder.layer.17.output.LayerNorm.bias',\n", " 'bert.encoder.layer.18.attention.self.query.weight',\n", " 'bert.encoder.layer.18.attention.self.query.bias',\n", " 'bert.encoder.layer.18.attention.self.key.weight',\n", " 'bert.encoder.layer.18.attention.self.key.bias',\n", " 'bert.encoder.layer.18.attention.self.value.weight',\n", " 'bert.encoder.layer.18.attention.self.value.bias',\n", " 'bert.encoder.layer.18.attention.output.dense.weight',\n", " 'bert.encoder.layer.18.attention.output.dense.bias',\n", " 'bert.encoder.layer.18.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.18.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.18.intermediate.dense.weight',\n", " 'bert.encoder.layer.18.intermediate.dense.bias',\n", " 'bert.encoder.layer.18.output.dense.weight',\n", " 'bert.encoder.layer.18.output.dense.bias',\n", " 'bert.encoder.layer.18.output.LayerNorm.weight',\n", " 'bert.encoder.layer.18.output.LayerNorm.bias',\n", " 'bert.encoder.layer.19.attention.self.query.weight',\n", " 'bert.encoder.layer.19.attention.self.query.bias',\n", " 'bert.encoder.layer.19.attention.self.key.weight',\n", " 'bert.encoder.layer.19.attention.self.key.bias',\n", " 'bert.encoder.layer.19.attention.self.value.weight',\n", " 'bert.encoder.layer.19.attention.self.value.bias',\n", " 'bert.encoder.layer.19.attention.output.dense.weight',\n", " 'bert.encoder.layer.19.attention.output.dense.bias',\n", " 'bert.encoder.layer.19.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.19.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.19.intermediate.dense.weight',\n", " 'bert.encoder.layer.19.intermediate.dense.bias',\n", " 'bert.encoder.layer.19.output.dense.weight',\n", " 'bert.encoder.layer.19.output.dense.bias',\n", " 'bert.encoder.layer.19.output.LayerNorm.weight',\n", " 'bert.encoder.layer.19.output.LayerNorm.bias',\n", " 'bert.encoder.layer.20.attention.self.query.weight',\n", " 'bert.encoder.layer.20.attention.self.query.bias',\n", " 'bert.encoder.layer.20.attention.self.key.weight',\n", " 'bert.encoder.layer.20.attention.self.key.bias',\n", " 'bert.encoder.layer.20.attention.self.value.weight',\n", " 'bert.encoder.layer.20.attention.self.value.bias',\n", " 'bert.encoder.layer.20.attention.output.dense.weight',\n", " 'bert.encoder.layer.20.attention.output.dense.bias',\n", " 'bert.encoder.layer.20.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.20.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.20.intermediate.dense.weight',\n", " 'bert.encoder.layer.20.intermediate.dense.bias',\n", " 'bert.encoder.layer.20.output.dense.weight',\n", " 'bert.encoder.layer.20.output.dense.bias',\n", " 'bert.encoder.layer.20.output.LayerNorm.weight',\n", " 'bert.encoder.layer.20.output.LayerNorm.bias',\n", " 'bert.encoder.layer.21.attention.self.query.weight',\n", " 'bert.encoder.layer.21.attention.self.query.bias',\n", " 'bert.encoder.layer.21.attention.self.key.weight',\n", " 'bert.encoder.layer.21.attention.self.key.bias',\n", " 'bert.encoder.layer.21.attention.self.value.weight',\n", " 'bert.encoder.layer.21.attention.self.value.bias',\n", " 'bert.encoder.layer.21.attention.output.dense.weight',\n", " 'bert.encoder.layer.21.attention.output.dense.bias',\n", " 'bert.encoder.layer.21.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.21.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.21.intermediate.dense.weight',\n", " 'bert.encoder.layer.21.intermediate.dense.bias',\n", " 'bert.encoder.layer.21.output.dense.weight',\n", " 'bert.encoder.layer.21.output.dense.bias',\n", " 'bert.encoder.layer.21.output.LayerNorm.weight',\n", " 'bert.encoder.layer.21.output.LayerNorm.bias',\n", " 'bert.encoder.layer.22.attention.self.query.weight',\n", " 'bert.encoder.layer.22.attention.self.query.bias',\n", " 'bert.encoder.layer.22.attention.self.key.weight',\n", " 'bert.encoder.layer.22.attention.self.key.bias',\n", " 'bert.encoder.layer.22.attention.self.value.weight',\n", " 'bert.encoder.layer.22.attention.self.value.bias',\n", " 'bert.encoder.layer.22.attention.output.dense.weight',\n", " 'bert.encoder.layer.22.attention.output.dense.bias',\n", " 'bert.encoder.layer.22.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.22.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.22.intermediate.dense.weight',\n", " 'bert.encoder.layer.22.intermediate.dense.bias',\n", " 'bert.encoder.layer.22.output.dense.weight',\n", " 'bert.encoder.layer.22.output.dense.bias',\n", " 'bert.encoder.layer.22.output.LayerNorm.weight',\n", " 'bert.encoder.layer.22.output.LayerNorm.bias',\n", " 'bert.encoder.layer.23.attention.self.query.weight',\n", " 'bert.encoder.layer.23.attention.self.query.bias',\n", " 'bert.encoder.layer.23.attention.self.key.weight',\n", " 'bert.encoder.layer.23.attention.self.key.bias',\n", " 'bert.encoder.layer.23.attention.self.value.weight',\n", " 'bert.encoder.layer.23.attention.self.value.bias',\n", " 'bert.encoder.layer.23.attention.output.dense.weight',\n", " 'bert.encoder.layer.23.attention.output.dense.bias',\n", " 'bert.encoder.layer.23.attention.output.LayerNorm.weight',\n", " 'bert.encoder.layer.23.attention.output.LayerNorm.bias',\n", " 'bert.encoder.layer.23.intermediate.dense.weight',\n", " 'bert.encoder.layer.23.intermediate.dense.bias',\n", " 'bert.encoder.layer.23.output.dense.weight',\n", " 'bert.encoder.layer.23.output.dense.bias',\n", " 'bert.encoder.layer.23.output.LayerNorm.weight',\n", " 'bert.encoder.layer.23.output.LayerNorm.bias',\n", " 'bert.pooler.dense.weight',\n", " 'bert.pooler.dense.bias',\n", " 'classifier.weight',\n", " 'classifier.bias']" ] }, "metadata": { "tags": [] }, "execution_count": 30 } ] }, { "cell_type": "markdown", "metadata": { "id": "qjNS0qL0swam" }, "source": [ "In the following 5 cells we define our PyTorch optimizer and corresponding parameters, learning rate scheduler, and the training loop for the fine-tuning procedure. We train for 10 epochs." ] }, { "cell_type": "code", "metadata": { "id": "PSrR3_3Wswam" }, "source": [ "learning_rate = 1e-5\n", "adam_epsilon = 1e-8\n", "\n", "no_decay = ['bias', 'LayerNorm.weight']\n", "optimizer_grouped_parameters = [\n", " {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.2},\n", " {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n", " 'weight_decay_rate': 0.0}\n", "]\n", "\n", "optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=adam_epsilon)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "yX1Mbyyhswap" }, "source": [ "from transformers import get_linear_schedule_with_warmup\n", "\n", "num_epochs = 10\n", "total_steps = len(train_dataloader) * num_epochs\n", "\n", "scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=total_steps)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "bBxkeoFRswas" }, "source": [ "import time\n", "\n", "def epoch_time(start_time, end_time):\n", " elapsed_time = end_time - start_time\n", " elapsed_mins = int(elapsed_time / 60)\n", " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n", " return elapsed_mins, elapsed_secs" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "d28cvJbSswav" }, "source": [ "import numpy as np\n", "import random\n", "\n", "seed_val = 11\n", "\n", "random.seed(seed_val)\n", "np.random.seed(seed_val)\n", "torch.manual_seed(seed_val)\n", "torch.cuda.manual_seed_all(seed_val)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "6MUnLD-Rsway", "colab": { "base_uri": "https://localhost:8080/", "height": 772 }, "outputId": "003d1c53-37b6-43ea-89be-85449f70f582" }, "source": [ "train_losses = []\n", "val_losses = []\n", "num_mb_train = len(train_dataloader)\n", "num_mb_val = len(val_dataloader)\n", "\n", "if num_mb_val == 0:\n", " num_mb_val = 1\n", "\n", "for n in range(num_epochs):\n", " train_loss = 0\n", " val_loss = 0\n", " start_time = time.time()\n", " \n", " for k, (mb_x, mb_m, mb_y) in enumerate(train_dataloader):\n", " optimizer.zero_grad()\n", " model.train()\n", " \n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " mb_y = mb_y.to(device)\n", " \n", " outputs = model(mb_x, attention_mask=mb_m, labels=mb_y)\n", " \n", " loss = outputs[0]\n", " #loss = model_loss(outputs[1], mb_y)\n", " loss.backward()\n", " torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n", " optimizer.step()\n", " scheduler.step()\n", " \n", " train_loss += loss.data / num_mb_train\n", " \n", " print (\"\\nTrain loss after itaration %i: %f\" % (n+1, train_loss))\n", " train_losses.append(train_loss.cpu())\n", " \n", " with torch.no_grad():\n", " model.eval()\n", " \n", " for k, (mb_x, mb_m, mb_y) in enumerate(val_dataloader):\n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " mb_y = mb_y.to(device)\n", " \n", " outputs = model(mb_x, attention_mask=mb_m, labels=mb_y)\n", " \n", " loss = outputs[0]\n", " #loss = model_loss(outputs[1], mb_y)\n", " \n", " val_loss += loss.data / num_mb_val\n", " \n", " print (\"Validation loss after itaration %i: %f\" % (n+1, val_loss))\n", " val_losses.append(val_loss.cpu())\n", " \n", " end_time = time.time()\n", " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", " print(f'Time: {epoch_mins}m {epoch_secs}s')" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.6/dist-packages/transformers/optimization.py:146: UserWarning: This overload of add_ is deprecated:\n", "\tadd_(Number alpha, Tensor other)\n", "Consider using one of the following signatures instead:\n", "\tadd_(Tensor other, *, Number alpha) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)\n", " exp_avg.mul_(beta1).add_(1.0 - beta1, grad)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\n", "Train loss after itaration 1: 1.307759\n", "Validation loss after itaration 1: 1.098660\n", "Time: 1m 25s\n", "\n", "Train loss after itaration 2: 0.906865\n", "Validation loss after itaration 2: 0.871359\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 3: 0.676544\n", "Validation loss after itaration 3: 0.809871\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 4: 0.524132\n", "Validation loss after itaration 4: 0.832292\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 5: 0.419839\n", "Validation loss after itaration 5: 1.095985\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 6: 0.319559\n", "Validation loss after itaration 6: 1.017313\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 7: 0.263214\n", "Validation loss after itaration 7: 1.234148\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 8: 0.185777\n", "Validation loss after itaration 8: 1.273066\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 9: 0.139775\n", "Validation loss after itaration 9: 1.418784\n", "Time: 1m 26s\n", "\n", "Train loss after itaration 10: 0.114112\n", "Validation loss after itaration 10: 1.448836\n", "Time: 1m 25s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "7vlt-7cnswa0" }, "source": [ "After training, we can save the model and necessary configuration parameters, to recreate it later and use it to score the test data. Here we also save the losses computed from both training and validation data." ] }, { "cell_type": "code", "metadata": { "id": "bjQYOzLMswa1" }, "source": [ "import pickle\n", "import os\n", "\n", "out_dir = './model'\n", "\n", "if not os.path.exists(out_dir):\n", " os.makedirs(out_dir)\n", " \n", "model_to_save = model.module if hasattr(model, 'module') else model\n", "model_to_save.save_pretrained(out_dir)\n", "tokenizer.save_pretrained(out_dir)\n", "\n", "with open(out_dir + '/train_losses.pkl', 'wb') as f:\n", " pickle.dump(train_losses, f)\n", " \n", "with open(out_dir + '/val_losses.pkl', 'wb') as f:\n", " pickle.dump(val_losses, f)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "JRAYa5Wpswa3" }, "source": [ "out_dir = './model'\n", "\n", "model = BertForSequenceClassification.from_pretrained(out_dir)\n", "\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "model = model.to(device)\n", "\n", "with open(out_dir + '/train_losses.pkl', 'rb') as f:\n", " train_losses = pickle.load(f)\n", " \n", "with open(out_dir + '/val_losses.pkl', 'rb') as f:\n", " val_losses = pickle.load(f)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "mPJFGzmlswa7", "colab": { "base_uri": "https://localhost:8080/", "height": 282 }, "outputId": "b4486792-ca5c-4eea-8516-7e69de58a59c" }, "source": [ "from matplotlib import pyplot as plt\n", "%matplotlib inline\n", "\n", "plt.figure()\n", "plt.plot(train_losses)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[]" ] }, "metadata": { "tags": [] }, "execution_count": 38 }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXRW9b3v8fc3EyEMmcOQBMKQgIAoEEBUZlAcinag1WrrbB0QxLanw7k9t6ftveece3qdqdUqSm2d7WAtrUVFGZQhYVJAkhCGJEwBMgCZk9/5I6lGZAjJE/YzfF5rsWA/e+fZn7UXfNaP357MOYeIiAS+MK8DiIiIb6jQRUSChApdRCRIqNBFRIKECl1EJEhEeLXjpKQkl5GR4dXuRUQCUm5u7iHnXPLJ1nlW6BkZGeTk5Hi1exGRgGRmu0+1TlMuIiJBQoUuIhIkVOgiIkFChS4iEiRU6CIiQUKFLiISJFToIiJBIuAKfdeh4/z7X7ZQ39jkdRQREb8ScIVeeOgYz67axeu5xV5HERHxKwFX6FOHpHBBehyPvVtAXYNG6SIi/xRwhW5mPDAzi5Lyal7OKfI6joiI3wi4QgeYlJlEdv94Fr5bQE19o9dxRET8QkAW+j9H6fsra3hp7R6v44iI+IWALHSAiwcncdHABBa+t4PqOo3SRUQCttABFszIovRoLb9fc8qnSYqIhIyALvTxAxO5dHAST7y3g+O1DV7HERHxVEAXOsCCmVkcPl7Hbz/UKF1EQlvAF/qY/vFMGZLMk8t3cLSm3us4IiKeCfhCB3hgZhblVfU8t2qX11FERDwTFIU+Mi2OGef14jcrCqmo1ihdREJTUBQ6wIKZmVTWNLBo5U6vo4iIeCJoCn1431iuGNGbRSt3Ul5V53UcEZFzLmgKHeD+GVkcq2vgNysKvY4iInLOnbHQzWyRmR00s49Psf4GM9tsZh+Z2QdmdoHvY7bNkN49uHpkX55dtYvDx2q9iiEi4om2jNCfA2adZv1OYLJz7nzg58BTPsjVbvOnZ1JT38hTyzVKF5HQcsZCd84tB46cZv0HzrmylsXVQJqPsrXL4JTuXHNhKos/3EXpUY3SRSR0+HoO/Tbgb6daaWZ3mlmOmeWUlpb6eNefmTc9k/pGxxPv7ei0fYiI+BufFbqZTaW50H9wqm2cc08557Kdc9nJycm+2vUXDEjqxldGpfK7Nbs5UFnTafsREfEnPil0MxsJPA1c45w77Ivv7Kh50zNpanIsXFbgdRQRkXOiw4VuZv2APwDfcs7ldTySb6QnxDAnO52X1hZRUl7tdRwRkU7XlssWXwQ+BIaYWbGZ3WZmd5nZXS2b/BuQCPzKzDaaWU4n5j0rc6cNBtAoXURCQsSZNnDOXX+G9bcDt/sskQ+lxnXlunHpvLBmD3dPHkR6QozXkUREOk1Q3Sl6MvdMGUxYmPHYu/leRxER6VRBX+i9Y6O5cXx/Xl9fwq5Dx72OIyLSaYK+0AHumjKQyHDj0Xc0SheR4BUShZ7SI5pvT8jgTxtLKDh4zOs4IiKdIiQKHeA7kwYSHRnOIxqli0iQCplCT+zehZsvzuDNzXvZvv+o13FERHwuZAod4M5JA+kWFcHDb/vN/U8iIj4TUoUeFxPFrZcO4G8f72fL3gqv44iI+FRIFTrAbZcOoGd0BA+/rbl0EQkuIVfosV0juWPiQJZuPcDm4nKv44iI+EzIFTrAzZdkEBcTyUNLNZcuIsEjJAu9R3Qkd04ayLLtpeTuLjvzD4iIBICQLHSAmyZkkNgtSle8iEjQCNlC79YlgrsmD2JF/iHW7jzlK1NFRAJGyBY6wI0X9Se5RxceXLrd6ygiIh0W0oXeNSqce6YMYnXhET7YccjrOCIiHRLShQ5w/bh+9O4ZzYP/yMM553UcEZF2C/lCj44M595pg8nZXcaKfI3SRSRwhXyhA3w9O43UuK48uFSjdBEJXCp0oEtEOPdNG8zGonKWbT/odRwRkXZRobf46pg00hM0SheRwKVCbxEZHsa8aZl8XFLJP7Ye8DqOiMhZU6G38uVRqQxI6sZDS/NoatIoXUQCiwq9lYjwMOZPz+ST/Uf5+5b9XscRETkrZyx0M1tkZgfN7ONTrDcze9TMCsxss5mN9n3Mc+dLF/RlcEp3HlqaR6NG6SISQNoyQn8OmHWa9VcAmS2/7gSe6Hgs74SHGffPyCT/4DHe3LzX6zgiIm12xkJ3zi0HTvf0qmuA37pmq4E4M+vjq4BeuHJEH4b27sEjb+fT0NjkdRwRkTbxxRx6KlDUarm45bMvMLM7zSzHzHJKS0t9sOvOERZm3D8ji8JDx/nzRo3SRSQwnNOTos65p5xz2c657OTk5HO567N2+fBeDO/bk0ffzadeo3QRCQC+KPQSIL3VclrLZwHNzHhgZha7D1fxh/XFXscRETkjXxT6G8C3W652uQiocM7t88H3em7a0BQuSIvl0XcKqGvQKF1E/FtbLlt8EfgQGGJmxWZ2m5ndZWZ3tWyyBCgECoDfAPd0WtpzzMxYMDOLkvJqXskpOvMPiIh4KOJMGzjnrj/Degfc67NEfmZyVjJj+sezcFkBXxuTRnRkuNeRREROSneKnsE/59L3VdTw8jqN0kXEf6nQ2+DiQYmMH5DAwmUF1NQ3eh1HROSkVOht8M+59INHa/nd6t1exxEROSkVehtdNDCRSwYn8uv3d1BV1+B1HBGRL1Chn4UHZmZx6Fgdv/1Qo3QR8T8q9LMwpn8Ck7OSefL9HRyr1ShdRPyLCv0sLZiZRVlVPc+t2ul1FBGRz1Ghn6UL0+OYcV4KTy0vpLKm3us4IiKfUqG3w/0zsqisaeCZFRqli4j/UKG3w4jUWGYN783TKwrZtq/S6zgiIoAKvd3+9+xh9IiO5OZn11JSXu11HBERFXp79YntynO3jqWqtpGbF62lokrz6SLiLRV6Bwzt3ZMnvz2G3YeruOP5HD0WQEQ8pULvoIsHJfHLr1/A2p1H+O6rm2hqcl5HEpEQdcbH58qZzb6gL/srqvm/Sz6hd89ofnL1MK8jiUgIUqH7yB0TB7K3vIZnVu6kT2w0t08c6HUkEQkxKnQfMTN+cvUwDlTW8Iu/bqN3bDRXj+zrdSwRCSGaQ/eh8DDjoW9cyNiMeB54eROrCw97HUlEQogK3ceiI8P5zbez6ZcYw52/zSHvwFGvI4lIiFChd4K4mCieu2Us0ZHh3LRoLfsraryOJCIhQIXeSdLiY3j2lrEcrWng5mfX6kFeItLpVOidaHjfWJ64cTQFB49x1/O51DU0eR1JRIKYCr2TTcxM5v99bSQf7DjM91/TjUci0nl02eI58JXRaeyrqOG/39pO79hofnTFeV5HEpEg1KYRupnNMrPtZlZgZj88yfp+ZrbMzDaY2WYzu9L3UQPbPVMGceNF/Xjy/UIWf7DL6zgiEoTOOEI3s3BgITATKAbWmdkbzrmtrTb7X8ArzrknzGwYsATI6IS8AcvM+PfZIzhQWctP/7KFXj27MGtEH69jiUgQacsIfRxQ4JwrdM7VAS8B15ywjQN6tvw5Ftjru4jBIzzMePS6UVyYHsf8lzaSs+uI15FEJIi0pdBTgaJWy8Utn7X2U+BGMyumeXR+38m+yMzuNLMcM8spLS1tR9zA1zUqnGduGkvfuK7ctjiHgoPHvI4kIkHCV1e5XA8855xLA64EnjezL3y3c+4p51y2cy47OTnZR7sOPAndolh8yzgiw42bFq3lYKVuPBKRjmtLoZcA6a2W01o+a+024BUA59yHQDSQ5IuAwapfYgyLbh5LWVUdtzy3jmO1DV5HEpEA15ZCXwdkmtkAM4sCrgPeOGGbPcB0ADM7j+ZCD805lbMwMi2OhTeM5pP9R7n7d7nUN+rGIxFpvzMWunOuAZgLvAVso/lqli1m9jMzm92y2XeBO8xsE/AicLNzTnfQtMHUISn8x5fPZ0X+IX7w+mZ02ESkvdp0Y5FzbgnNJztbf/Zvrf68FbjEt9FCx9fHprO3opqH386nb2xXvnf5EK8jiUgA0p2ifmL+9Ez2V9Tw+LIC+sRFc8P4/l5HEpEAo0L3E2bGL64dwYHKGn7yp49J6RHNzGG9vI4lIgFED+fyIxHhYTz+zdGMSI3lvhfXs35PmdeRRCSAqND9TLcuESy6eSwpPaK5fXEOOw8d9zqSiAQIFbofSurehcW3jgPgpkVrKT1a63EiEQkEKnQ/NSCpG8/clM3BozXctngdx3XjkYicgQrdj43qF8/j14/m45IK5r6wngbdeCQip6FC93MzhvXi59eOYNn2Uv71jx/rxiMROSVdthgAbhjfn33ln12jfv+MLK8jiYgfUqEHiO9elsW+ihoefjufPrHRfGNsP68jiYifUaEHCDPjP796PgeP1vDjPzbfeDR1aIrXsUTEj2gOPYBEhofxxI1jGNq7B/f8fj2bi8u9jiQifkSFHmC6d4ng2ZvHktAtilufW8eew1VeRxIRP6FCD0ApPaNZfOs46hsdNz27lsPHdOORiKjQA9bglO48c1M2JeXV3PD0GnbpEQEiIU+FHsCyMxL4zbez2VdRw9WPreQvm/Z6HUlEPKRCD3CTs5JZMn8iWb26c9+LG/jxHz+ipr7R61gi4gEVehBIjevKy9+ZwHcmD+SFNXu4duEqdpQe8zqWiJxjKvQgERkexo+uOI9nbx7LgcoavvTYSv60ocTrWCJyDqnQg8zUoSksmT+R4X17cv/LG/nh65uprtMUjEgoUKEHoT6xXXnxjou4Z8ogXlpXxLULV1Fw8KjXsUSkk6nQg1REeBj/Mmsoi28dx6FjtXzpsVW8nlvsdSwR6UQq9CD3z6tgRqbF8t1XN/G9VzdRVaeXZYgEIxV6COjVM5rf3z6eedMG8/r6Yq55fBV5BzQFIxJs2lToZjbLzLabWYGZ/fAU23zdzLaa2RYze8G3MaWjIsLDeOCyITx/63jKquqZ/fhKXllXpBdmiASRMxa6mYUDC4ErgGHA9WY27IRtMoEfAZc454YD93dCVvGBSzOTWDL/Ukb3i+dfXt/MA69s0vtKRYJEW0bo44AC51yhc64OeAm45oRt7gAWOufKAJxzB30bU3wppUc0z982ngUzsvjzxhK+9PhKtu2r9DqWiHRQWwo9FShqtVzc8llrWUCWma0ys9VmNutkX2Rmd5pZjpnllJaWti+x+ER4mDF/Ria/v/0ijtY0cO3CVbywZo+mYEQCmK9OikYAmcAU4HrgN2YWd+JGzrmnnHPZzrns5ORkH+1aOmLCoET+Nn8i4wYk8OM/fsS8lzZytKbe61gi0g5tKfQSIL3VclrLZ60VA2845+qdczuBPJoLXgJAUvcuLL5lHN+/fAh/3byXLz22ko9LKryOJSJnqS2Fvg7INLMBZhYFXAe8ccI2f6J5dI6ZJdE8BVPow5zSycLCjHunDubFOy6iur6RrzzxAc+v3q0pGJEAcsZCd841AHOBt4BtwCvOuS1m9jMzm92y2VvAYTPbCiwDvu+cO9xZoaXzjB+YyJJ5E5kwMJGf/Olj5r6wgUpNwYgEBPNqBJadne1ycnI82becWVOT48nlhfzyH9tJjevKwm+O5vy0WK9jiYQ8M8t1zmWfbJ3uFJWTCgsz7p4yiJfvvIj6xia++sQHPLdqp6ZgRPyYCl1OKzsjgSXzJjIxM4mf/mUrd/9uPRXVmoIR8UcqdDmj+G5RPH1TNv965Xm8ve0AVz26go1F5V7HEpETqNClTcyMOyYN5JW7JuAczPn1Bzy9olBTMCJ+RIUuZ2V0v3iWzJvIlCEp/OKv27jjt7mUV9V5HUtEUKFLO8TGRPLUt8bwb1cP4/28g1z16Epyd5d5HUsk5KnQpV3MjFsvHcBrd11MWBh848kP+e+3PtEJUxEPqdClQy5Ij+PN+yZy9cg+LFy2g0v/610efSdfz4MR8YBuLBKf2bK3goeW5vP2tgPExUTynUmDuOni/sRERXgdTSRonO7GIhW6+Nzm4nIeXJrHe9tLSewWxd1TBnHjRf2Jjgz3OppIwFOhiydyd5fx0NI8VhYcIqVHF+6dOpjrxqXTJULFLtJeKnTx1OrCwzy4NI+1O4/QJzaaudMGM2dMOlEROoUjcrZU6OI55xyrCg7z/5duZ8OectLiuzJveiZfGZVKRLiKXaStVOjiN5xzvJdXykNL89hcXEFGYgzzZ2Qy+4JUwsPM63gifk9PWxS/YWZMHZLCn++9hKe+NYboyHAWvLyJyx9ezpub99LUpEcJiLSXCl08YWZcNrw3S+ZN5Fc3jMaAuS9s4MpHV/D3j/frGTEi7aBCF0+FhRlXnt+Hv98/iUeuu5C6hibu+l0uVz+2kne2HVCxi5wFFbr4hfAw45oLU/nHgkn8cs4FHK1p4LbFOXz5Vx+wPK9UxS7SBjopKn6pvrGJ13KLeeydfPZW1DA2I54HZg5hwqBEr6OJeEpXuUjAqm1o5JV1RTy+rIADlbVcPCiRB2ZmkZ2R4HU0EU+o0CXg1dQ38sKaPfzqvR0cOlbLpKxkHpiZxYXpcV5HEzmnVOgSNKrqGnj+w938+v0dlFXVM31oCgtmZjEiNdbraCLnhApdgs6x2gaeW7WTp5YXUlnTwKzhvVkwM4shvXt4HU2kU6nQJWhV1tTzzIqdLFq5k2N1DVx1fh/un5HJ4BQVuwSnDt8pamazzGy7mRWY2Q9Ps91XzcyZ2Ul3JuJrPaMjWTAzixU/mMo9Uwbx7icHmfnQcu59YT2f7K/0Op7IOXXGEbqZhQN5wEygGFgHXO+c23rCdj2AvwJRwFzn3GmH3xqhS2c4cryOZ1YWsviD3RyrbeDy4b24b1qm5tglaHR0hD4OKHDOFTrn6oCXgGtOst3Pgf8CatqdVKSDErpF8f3Lh7LyB1OZPz2TD3Yc5urHVnL74nVsKir3Op5Ip2pLoacCRa2Wi1s++5SZjQbSnXN/9WE2kXaLi4liwcwsVv1wGt+7LIuc3WVcs3AVNy1aS+7uI17HE+kUHb7138zCgAeB77Zh2zvNLMfMckpLSzu6a5Ez6hkdydxpmaz8wTR+MGsoH5VU8NUnPuSGp1ezpvCw1/FEfKotc+gTgJ865y5vWf4RgHPuP1qWY4EdwLGWH+kNHAFmn24eXXPo4oWqugZeWLOHX79fyKFjtYwfkMC86ZlcPCgRMz2PXfxfhy5bNLMImk+KTgdKaD4p+k3n3JZTbP8e8D2dFBV/VlPfyItr9/Dr93dwoLKWMf3jmTc9k0mZSSp28WsdOinqnGsA5gJvAduAV5xzW8zsZ2Y227dRRc6N6MhwbrlkAO9/fyo/v3YE+8qruWnRWq5duEqP7ZWApRuLRIC6hiZeX1/Mr94roOhINcP79uS+aZlcNqwXYXo1nvgR3Skq0kb1jU38aUMJC5cVsOtwFUN79+C+aZlcMaK3il38ggpd5Cw1NDbx5uZ9PPZuPjtKjzM4pTv3TRvM1SP76mXW4ikVukg7NTY5lnzUXOx5B44xMKkb904dzDUX9iUiXC/8knNPhS7SQU1Njn9s3c8j7xSwbV8l/RJiuHfqIL48Ko2oCBW7nDsqdBEfcc7x9raDPPpOPh+VVJAa15W7pwxiTnYaXSLCvY4nIUCFLuJjzjneyyvl0Xfy2bCnnN49o7l7yiC+MTad6EgVu3QeFbpIJ3HOsargMI+8k8e6XWUk9+jCdyYN5Ibx/ekapWIX31Ohi3Qy5xyrC4/w6Dv5fFh4mIRuUUwfmsKkrGQuHZxEfLcoryNKkDhdoUec6zAiwcjMmDAokQmDElm36wjPrdrFW1v282puMWYwMjWWSVnJTMpK5sL0OCJ1hYx0Ao3QRTpJQ2MTm4orWJFfyvK8UjYWldPkoEeXCCYMSmRiVjKTM5PplxjjdVQJIJpyEfEDFVX1fLDjEMvzS1med4iS8moAMhJjmJjZPHqfMCiR7l30H2c5NRW6iJ9xzlF46DjL80pZkX+ID3ccprq+kYgwY3T/eCZnJTMpM5nhfXvqkQPyOSp0ET9X29BI7u4ylucdYnleKVv3Nb/gOqFbFJcOTmJSVjITM5Po1TPa46TiNRW6SIApPVrLyoLmqZkV+aUcOlYHwNDePZiY2VzwYzMSdM17CFKhiwSwpibHtv2VrMhvHr3n7CqjrrGJLhFhjB+YyKTMJCZnJTM4pbtezhECVOgiQaSqroE1hUd4P6+U5fmlFJYeB6BPbDQTM5OYmKlr34OZCl0kiBWXVbEiv3lqZmX+ISprGpqvfU+LY0pWMl8elUpGUjevY4qPqNBFQsSprn0fl5HA17LTuOr8PnTTZZEBTYUuEqL2V9Tw+vpiXsstZueh48REhXPl+X2YMyaNcQMSNOcegFToIiHOOUfu7jJezSnmzc17OV7XSP/EGOaMSeMro9PoG9fV64jSRip0EflUVV0Df/toP6/mFrG68AhmcOngJOZkp3PZsF66FNLPqdBF5KT2HK7itfXFvJ5bTEl5NT2jI5h9YV/mjElnZFqspmT8kApdRE6rqcnxYeFhXskp4u8f76e2oYmsXt2ZMyada0elktyji9cRpYUKXUTarKK6njc37+XVnGI2FpUTEWZMGZLC17PTmDo0RY/+9ZgKXUTaJf/AUV7LLeYPG0ooPVpLUvcorr0wlTnZ6Qzp3cPreCGpw4VuZrOAR4Bw4Gnn3H+esP4B4HagASgFbnXO7T7dd6rQRQJHQ2MT7+eV8mpOMe98coD6RsfItFjmjElj9gWpxMZEeh0xZHSo0M0sHMgDZgLFwDrgeufc1lbbTAXWOOeqzOxuYIpz7hun+14VukhgOnyslj9v3MsrOUV8sv8oURFhXDasF3Oy07l0cBLhetxvp+roK+jGAQXOucKWL3sJuAb4tNCdc8tabb8auLH9cUXEnyV278Ktlw7glksy2LK3kldzivjzpr28uXkffWKj+croVL42Jp0BetzAOdeWQk8FilotFwPjT7P9bcDfTrbCzO4E7gTo169fGyOKiD8yM0akxjIiNZYfX3Ueb289yKu5RTzx3g4WLtvB2Ix45oxJ58qRffQWpnOkLVMuXwNmOedub1n+FjDeOTf3JNveCMwFJjvnak/3vZpyEQlO+ytq+MOGYl7LKaaw5XED04amMG5AAqP7xTO0dw8idKVMu3V0yqUESG+1nNby2Yk7mQH8K20ocxEJXr1jo7lnymDunjyI9XuaHzfw7icHeXPzPgBiosK5IC2OMf3jGd0/jlHp8XrUr4+0pdDXAZlmNoDmIr8O+GbrDcxsFPAkzSP5gz5PKSIBx8wY0z+BMf0TcM5RUl5N7u4y1u8uY/2ecp54fweNTc0zBIOSuzG6Xzxj+jf/GpTcXe9SbYe2XrZ4JfAwzZctLnLO/R8z+xmQ45x7w8zeBs4H9rX8yB7n3OzTfaemXERCW1VdA5uKKli/p7nkc/eUUV5VD0DP6AhGtRT86H7xXNgvTvPwLXRjkYj4PeccOw8dbx7F7ylj/e5y8g4exTkIMxjSuyej+8V9OorvlxATks+aUaGLSECqqK5nY1F5yzRNGRv2lHOstgGAxG5RjO7/2Sh+ZFpsSDwpsqMnRUVEPBHbNZLJWclMzkoGoLHJkX/wKLm7y8jd3VzwS7ceACAizBieGvu5UXyf2NB6zrtG6CIS0A4fq2X9nnLW72ku+U1F5dQ2NAHNL84e3TKCH9O/+ZLJQB/Fa4QuIkErsXsXZg7rxcxhvQCob2xi277Kz43i/7p536fbp/ToQlp8V9LiYz79PT2h+fe+cdF0iQjcwtcIXUSC3r6KatbvLmdH6TGKy6ooLqumuKyaveXVNDR9vgN79ezSXPInlH5afFf6xnUlKsLbm6I0QheRkNYntitXjfzifHpDYxMHjtZSfOSzki8uq6KorIqc3WX8ZfO+T6+VBzCD3j2jTxjht4zy42PoExft6fPiVegiErIiwsNIjetKalzXkz6gqqGxif2VNZ+WfdGnxV/F2p1H+PPGaloP8MM+LfyWsk/4rPTT42PoHdu5ha9CFxE5hYjwsJZyjjnp+vrGJvZX1FBU9vkRfnFZNasLD7N/Y8kXCr9PbFduvjiDOyYN9H1en3+jiEiIiAwPIz0hhvSEkxd+XUNz4X82b19FUVk1KT075x2tKnQRkU4SFRFGv8QY+iWevPB9Tc+wFBEJEip0EZEgoUIXEQkSKnQRkSChQhcRCRIqdBGRIKFCFxEJEip0EZEg4dnTFs2sFNjdzh9PAg75ME6g0/H4PB2Pz+hYfF4wHI/+zrnkk63wrNA7wsxyTvX4yFCk4/F5Oh6f0bH4vGA/HppyEREJEip0EZEgEaiF/pTXAfyMjsfn6Xh8Rsfi84L6eATkHLqIiHxRoI7QRUTkBCp0EZEgEXCFbmazzGy7mRWY2Q+9zuMlM0s3s2VmttXMtpjZfK8zec3Mws1sg5m96XUWr5lZnJm9ZmafmNk2M5vgdSavmNmCln8jH5vZi2YW7XWmzhBQhW5m4cBC4ApgGHC9mQ3zNpWnGoDvOueGARcB94b48QCYD2zzOoSfeAT4u3NuKHABIXpczCwVmAdkO+dGAOHAdd6m6hwBVejAOKDAOVfonKsDXgKu8TiTZ5xz+5xz61v+fJTmf7Cp3qbyjpmlAVcBT3udxWtmFgtMAp4BcM7VOefKvU3lqQigq5lFADHAXo/zdIpAK/RUoKjVcjEhXGCtmVkGMApY420STz0M/AvQ5HUQPzAAKAWebZmCetrMunkdygvOuRLgl8AeYB9Q4Zz7h7epOkegFbqchJl1B14H7nfOVXqdxwtmdjVw0DmX63UWPxEBjAaecM6NAo4DIXnOycziaf6f/ACgL9DNzG70NlXnCLRCLwHSWy2ntXwWsswskuYy/71z7g9e5/HQJcBsM9tF81TcNDP7nbeRPFUMFDvn/vk/ttdoLvhQNAPY6Zwrdc7VA38ALvY4U6cItEJfB2Sa2QAzi6L5xMYbHmfyjJkZzXOk25xzD3qdx0vOuR8559Kccxk0/7141zkXlKOwtnDO7QeKzGxIy0fTga0eRvLSHuAiM4tp+TcznSA9QRzhdYCz4ZxrMLO5wFs0n6le5Jzb4nEsL10CfAv4yMw2tnz2Y+fcEhUoZlUAAABWSURBVA8zif+4D/h9y+CnELjF4zyecM6tMbPXgPU0Xxm2gSB9BIBu/RcRCRKBNuUiIiKnoEIXEQkSKnQRkSChQhcRCRIqdBGRIKFCFxEJEip0EZEg8T+w3UgNIeCT1wAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "3qNAwu3Qswa_" }, "source": [ "After instantiating a trained model, we can then score the test data and compute its accuracy. We then print the classification report and plot a confusion matrix.\n" ] }, { "cell_type": "code", "metadata": { "id": "3x8VvAbiswa_" }, "source": [ "batch_size = 8\n", "\n", "test_data = TensorDataset(test_x, test_m)\n", "test_sampler = SequentialSampler(test_data)\n", "test_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=batch_size)\n", "\n", "outputs = []\n", "with torch.no_grad():\n", " model.eval()\n", " for k, (mb_x, mb_m) in enumerate(test_dataloader):\n", " mb_x = mb_x.to(device)\n", " mb_m = mb_m.to(device)\n", " output = model(mb_x, attention_mask=mb_m)\n", " outputs.append(output[0].to('cpu'))\n", "\n", "outputs = torch.cat(outputs)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "jh36teiaswbC" }, "source": [ "_, predicted_values = torch.max(outputs, 1)\n", "predicted_values = predicted_values.numpy()\n", "true_values = test_y.numpy()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "nSAayTEoswbF", "colab": { "base_uri": "https://localhost:8080/", "height": 33 }, "outputId": "37c91226-1b64-4074-ad5f-fa3713e57009" }, "source": [ "test_accuracy = np.sum(predicted_values == true_values) / len(true_values)\n", "print (\"Test Accuracy:\", test_accuracy)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Test Accuracy: 0.649746192893401\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "9tNMCh7mswbI" }, "source": [ "import warnings\n", "warnings.filterwarnings('always')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "pajXvGrxswbL", "colab": { "base_uri": "https://localhost:8080/", "height": 201 }, "outputId": "70e8b147-d15f-4818-83f7-ad9a48c9a4ab" }, "source": [ "from sklearn.metrics import classification_report\n", "\n", "print(classification_report(true_values, predicted_values, target_names=[str(l) for l in label_values]))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " zero 0.67 0.65 0.66 77\n", " one 0.86 0.91 0.88 46\n", " two 0.51 0.51 0.51 47\n", " three 0.46 0.44 0.45 27\n", "\n", " accuracy 0.65 197\n", " macro avg 0.62 0.63 0.63 197\n", "weighted avg 0.65 0.65 0.65 197\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "3Dxvrg_LswbO" }, "source": [ "Analysis" ] }, { "cell_type": "code", "metadata": { "id": "TlNGk8luswbP" }, "source": [ "import itertools\n", "\n", "# plot confusion matrix\n", "# code borrowed from scikit-learn.org\n", "def plot_confusion_matrix(cm, classes,\n", " normalize=False,\n", " title='Confusion matrix',\n", " cmap=plt.cm.Blues):\n", " \"\"\"\n", " This function prints and plots the confusion matrix.\n", " Normalization can be applied by setting `normalize=True`.\n", " \"\"\"\n", " if normalize:\n", " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", " print(\"Normalized confusion matrix\")\n", " else:\n", " print('Confusion matrix, without normalization')\n", "\n", " print(cm)\n", "\n", " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", " plt.title(title)\n", " plt.colorbar()\n", " tick_marks = np.arange(len(classes))\n", " plt.xticks(tick_marks, classes, rotation=45)\n", " plt.yticks(tick_marks, classes)\n", "\n", " fmt = '.2f' if normalize else 'd'\n", " thresh = cm.max() / 2.\n", " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", " plt.text(j, i, format(cm[i, j], fmt),\n", " horizontalalignment=\"center\",\n", " color=\"white\" if cm[i, j] > thresh else \"black\")\n", "\n", " plt.tight_layout()\n", " plt.ylabel('True label')\n", " plt.xlabel('Predicted label')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Z1PFY-x3swbR", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "38c01f95-c407-42cf-e1eb-06e25b9de97f" }, "source": [ "from sklearn.metrics import classification_report, confusion_matrix\n", "\n", "cm_test = confusion_matrix(true_values, predicted_values)\n", "\n", "np.set_printoptions(precision=2)\n", "\n", "plt.figure(figsize=(6,6))\n", "plot_confusion_matrix(cm_test, classes=label_values, title='Confusion Matrix - Test Dataset')\n", "plt.figure(figsize=(6,6))\n", "plot_confusion_matrix(cm_test, classes=label_values, title='Confusion Matrix - Test Dataset', normalize=True)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Confusion matrix, without normalization\n", "[[50 5 15 7]\n", " [ 1 42 1 2]\n", " [17 1 24 5]\n", " [ 7 1 7 12]]\n", "Normalized confusion matrix\n", "[[0.65 0.06 0.19 0.09]\n", " [0.02 0.91 0.02 0.04]\n", " [0.36 0.02 0.51 0.11]\n", " [0.26 0.04 0.26 0.44]]\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaYAAAGlCAYAAACx5VKtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU5fn+8c8FKIqASjOAUVSiRjSiYEEsWIIaC2DsoijYe4wx5he/6SbRxBo1hiixYIwlGkWNqFjBjsGCLdaoKAKKAlbw/v1xzuK6wu4CM3MK15vXvJg59d6zs3PP/ZznnEcRgZmZWV60yDoAMzOz+pyYzMwsV5yYzMwsV5yYzMwsV5yYzMwsV5yYzMwsV1plHYCZmZWDpNeAWcA8YG5E9JXUAbgG6AG8BuwdEe83th1XTGZmVknbRkTviOibvj4VGBcR3wLGpa8b5cRkZmbVNAi4PH1+OTC4qRWcmMzMrFICuEPSREmHp9NWiYi30+fvAKs0tRGfYzIzK6GW7VePmPtxxbYXH0+bDHxSb9LIiBjZYLEtI+ItSV2AOyU9/5VtRISkJu+D58RkZlZCMfdjWq+zd8W298mkCz+pd95owfuMeCv9/11JNwKbAlMldY2ItyV1Bd5tal9uyjMzKyWBWlTu0dTepBUktat7DgwEngFuBoaliw0DbmpqW66YzMysElYBbpQESW75e0TcLukx4FpJI4DXgSbLOCcmM7MyEpAkiZqIiFeADRcwfQaw/aJsy4nJzKysmtEEl0fFjNrMzErLFZOZWVnVsCmvkpyYzMxKSW7KMzMzqwRXTGZmZeWmPDMzyw3hpjwzM7NKcMVkZlZKKmxTnismMzPLFVdMZmZlVdBzTE5MZmZl5aY8MzOzJeeKycyslIp75wcnJjOzMqrxsBeVVMx0amZmpeWKycysrNyUZ2Zm+VHcc0zFjNrMzErLFZOZWVm1cOcHMzOzJeaKycysjAo87IUTk5lZWfk6JjMzsyXnisnMrJSK213cicnMrKzclGdmZrbkXDGZmZVVQZvyihm1mZmVlismM7MyknyOycpH0vKSxkj6QNJ1S7CdAyTdUcnYsiDp35KGZR2HWbOpReUeNeTEVAKS9pf0uKTZkt5OP0C3rMCm9wRWATpGxF6Lu5GIuCoiBlYgnq+QNEBSSLqxwfQN0+n3NnM7v5A0uqnlImLniLh8McNd2L63Sn9vsyXNSeOeXe+x2mJsMyT1bGT+wZLm1dvHq5L+JmntRdjHZZJ+s6ixLapa7cfyxYmp4CSdBJwL/JYkiawGXAQMqsDmVwdejIi5FdhWtUwD+knqWG/aMODFSu1Aiar8rUTEAxHRNiLaAr3SySvVTYuI/1Vjv8BD6T5XBHYAPgYmSlq/SvuzLNQ151XiUUNOTAUmaUXgV8AxEXFDRMyJiM8jYkxE/ChdprWkcyVNSR/nSmqdzhsg6U1JP5T0blptHZLO+yXwM2Cf9Fv1iIaVhaQe6bfzVunrgyW9ImlW+i38gHrTx9dbbwtJj6VNhI9J2qLevHsl/VrShHQ7d0jq1Mhh+Az4F7Bvun5LYB/gqgbH6jxJb0j6UNJESVul03cC/l+9n/PJenGcLmkC8BGwZjrt0HT+nyX9s972z5A0TqrcX7CkFSVdmv5e3pL0m/TnQ1JPSfelx3C6pGvS6fenqz+Z/jz7NLaPiJgXES9HxNHAfcAv6u3/OknvpPu4X1KvdPrhwAHAKek+xqTTT5X0cvp7e1bSkHrbWmC86bx1Jd0p6T1JL0jau7H9WHPJTXmWiX7AcsCNjSzzU2BzoDewIbApcFq9+d8g+dbcHRgBXChp5Yj4OUkVdk36zf3SxgKRtAJwPrBzRLQDtgAmLWC5DsCt6bIdgbOBWxtUPPsDhwBdgGWBkxvbN3AFcFD6fEfgGWBKg2UeIzkGHYC/A9dJWi4ibm/wc25Yb50DgcOBdsDrDbb3Q2CDNOluRXLshkVENBHrorgMmAv0BDYCBgKHpvN+DdwBrAysCvwJICK2TudvmP4819B8NwBb1Xv9b+BbJL+HJ0iTfUSMTJ+fme5jt3T5l9P1VwR+CYyW1LWxeNP3zZ0kv5MuJF8wLpK0XiP7sZJzYiq2jsD0JpraDgB+FRHvRsQ0kg+MA+vN/zyd/3lE3AbMBtZZzHi+ANaXtHxEvB0RkxewzC7AfyPiyoiYGxFXA88D9T90/hYRL0bEx8C1JAlloSLiQaCDpHVIEtQVC1hmdETMSPd5FtCapn/OyyJicrrO5w229xHJcTwbGA0cFxFvNrG9ZpO0CvA94MS0En4XOIe0MiT5va0OdIuITyJi/EI2tSimkCRuACJiVETMiohPSSqpDdMqfYEi4rqImBIRX6QJ8b8kX4Qai3dX4LWI+Ft6nP8D/BNY7HOaVo+b8iwDM4BOdU1pC9GNr37bfz2dNn8bDRLbR0DbRQ0kIuaQNKEdCbwt6VZJ6zYjnrqYutd7/c5ixHMlcCywLQuoICWdLOm5tClpJsm3+saaCAHeaGxmRDwCvAKIJIEukKTJ+rKjwVYLW66B1YFlSI7lzDTmv5BUFQCnpPt9NN3+8GZutzHdgffSmFtK+n3aNPch8Fq6zEKPmaSDJE2qF+/69ZZfWLyrA5vVrZOudwBJJW9Lom7YiwI25fk6pmJ7CPgUGAxcv5BlppD88ddVL6vx9Wau5poDtKn3+isfHhExFhgraXngN8Bf+WrTUP146lsNuH0xY6pzJfAScEVEfFT/VE+aDE4BtgcmR8QXkt4n+dMFWFjzW6PNcpKOIam8pqTb/90CNxLRa0HTm/AGye+204Iq4oh4BzgsjWNL4C5J90fES4uxrzpDgAfS5/uTdKDZgSQprQgs9JhJWp3k9709SceKeZIm1S2/sHjTn/O+iPjuQmKqZNOoFYQrpgKLiA9IOihcKGmwpDaSlpG0s6Qz08WuBk6T1DntRPAzkqanxTEJ2FrSammTzk/qZkhaRdKg9JzBpyRNgl8sYBu3AWsr6eLeKj05vx5wy2LGBEBEvApsQ3JOraF2JOdqpgGtJP0MaF9v/lSghxah552SrtW/AYaSNOmdIqnRJsdFERFvk5yTOUtSe0ktJK0laZt0/3tJWjVd/H2SD/C64z0VWLOZP0dLSWtI+hMwgKSpF5Jj9ilJVd6G5DxcfQ33sUIaw7R0u4eQVEx1+1lYvLeQvB8OTN+7y0jaRNK3F/VnsYbc+cEykp4vOYmkQ8M0km+gx5L0VIPkw/Nx4CngaZKT2It1XUhE3Alck25rIl9NJi3SOKaQNAdtAxy1gG3MIDmv8EOSD71TgF0jYvrixNRg2+MjYkHV4FiSiuxFkmbDT/hqM13dxcMzJD3R1H7SptPRwBkR8WRE/JekZ9+VSns8VshBJJ0/niX5ML8eqOtMsAnwiKTZwM3ACRHxSjrvF8DladPY3gvZdr903Q+Be0kS9SYR8XQ6/wqSY/VWuv+HG6x/KbBeuo9/RcSzwFkkVfxUYANgQr3lFxhvRMwi6dSxL8l75x3gDJJK9Gv7aeqAWTmosp2IzMwsD1qstHq03vrUim3vkzFHT4yIvhXbYCN8jsnMrKx8d3EzM7Ml54rJzKysCnp3cScmM7MyktyUZ2ZmVgmlrpjUavnQsu2yDqMQNljnm1mHUCgffz4v6xAKo3WrllmHUBhvvfE6782YXrn2Nzfl5Y+WbUfrdRZ2GYfVN/bes7MOoVCenvJB1iEURs/O/nLYXLvv0L+i21NBE5Ob8szMLFdKXTGZmS2thCsmMzOzinDFZGZWRuLLe8EXjBOTmVkpyU15ZmZmleCKycyspIpaMTkxmZmVVFETk5vyzMwsV1wxmZmVVFErJicmM7MyKnB3cTflmZlZrrhiMjMrIfk6JjMzs8pwxWRmVlJFrZicmMzMSqqoiclNeWZmliuumMzMSqqoFZMTk5lZGfk6JjMzs8pwxWRmVlJuyjMzs9zwBbZmZmYV4orJzKykXDGZmZlVgBOTmVlZqYKP5u5SainpP5JuSV+vIekRSS9JukbSsk1tw4nJzKyMlDTlVeqxCE4Anqv3+gzgnIjoCbwPjGhqA05MZmZWEZJWBXYBLklfC9gOuD5d5HJgcFPbceeHKnv+1l8ya86nzPviC+bO+4ItDziTldu34cozhrN6tw68PuU9hp5yKTNnfZx1qLmyyQZr07ZdW1q2aEnLVq0Ye+9DWYeUK2f99AQevu9OVurQib/efD8AV1xwJv++fjQrrtwRgOEn/pRNt9khyzBz55WXXuS4Qw+c//qN11/lxB//H8OPPC7DqKong84P5wKnAO3S1x2BmRExN339JtC9qY04MdXAToefx4yZc+a/PvmQ73Lvoy/wx7/dycmHfJeTDxnIaefflGGE+XT9mDvo2LFT1mHk0neH7MvuB4zgzFOP/cr0PQ46gr2GH5NRVPm3Zs+1ufXeRwCYN28e/TZYix132T3jqKqnwompk6TH670eGREj6+1rV+DdiJgoacCS7MhNeRnYdcB3GD0m+eMYPeYRdtv2OxlHZEXznb79aLfiSlmHUWgP3n8Pq/dYg+7fXD3rUIpiekT0rfcY2WB+f2B3Sa8B/yBpwjsPWElSXRG0KvBWUztyYqqyiGDMRccy4apTGL5HfwC6dGzHO9M/BOCd6R/SpWO7xjaxVJJg3yG7MHCbzbnyskuyDqcwbv77KI4YvA1n/fQEZn0wM+twcm3Mjdex2x57Zx1G1dTd+aFWnR8i4icRsWpE9AD2Be6OiAOAe4A908WGAU02D7kpr8q2P+Qcpkz7gM4rt+WWi4/lhdfe+doyERkElnM33X4PXbt1Z/q0d9ln8Pfo+a116Nd/q6zDyrXd9j2YA476IZK4/PzfM/LMn/PD08/LOqxc+uyzzxg39lZ+dNqvsg6luvJxfe2PgX9I+g3wH+DSplYoRMUkqWXWMSyuKdM+AGDa+7O5+e6n2KRXD96dMYtvdGoPwDc6tWfae7OyDDGXunZLzo926tyFnXcdxKQnHss4ovxbuVMXWrZsSYsWLdh5r6E8//R/sg4pt+4bN5Ze3+lN5y6rZB1KKUXEvRGxa/r8lYjYNCJ6RsReEfFpU+vXNDFJOlLSpPTxqqR7JA2U9JCkJyRdJ6ltuuxrks6Q9ASwl6T9JD0t6RlJZ9Qy7sXVZrlladum9fznO/Rbl8kvT+HW+55m6G6bATB0t8245d6nsgwzdz6aM4fZs2bNf37fPXexzrd7ZRxV/s2YNnX+8wl33UaPb62bYTT5NuaGa9ltSHmb8YAsr2NaYjVtyouIi4GLJS0D3A2MAk4DdoiIOZJ+DJwE1NXXMyJiY0ndgIeBPiQXaN0haXBE/KvhPiQdDhwOwDJtq/0jNapLx3Zcc/ZhALRq2ZJr/v04dz74HBMn/4/RZwxn2OB+/O/t9xh6yqhM48ybadOmMvyA5ENj7ry5DNlzX7bbYceMo8qX3558BE89OoEPZr7H/ttuyIHHnsJTj07g5ecnI8Eq3VfjhF/8Meswc+mjOXMYf9/d/OasC7IOxRZCkcEJDkkXAdOAx4DLSPq2AywLPBQRI9KeHdtExOuSBgHfj4iD0vVHAL0i4qTG9tOiTZdovU7JvxVVyKv3np11CIXy9JQPsg6hMHp2duee5tp9h/48PWliRcqTZbv0jC57Vu7LyVt/HjIxIvpWbIONqHnnB0kHA6sDx5JcIXxnROy3kMXnLGS6mZk1wXcXbwZJfYCTgaER8QVJ81x/ST3T+StIWnsBqz4KbCOpU9oRYj/gvlrFbWZmtVPriulYoANwT5rJHwcOBq6W1Dpd5jTgxforRcTbkk4l6Q8v4NaI8K0SzMwaU8yCqeadHw5ZyKxNFrBsjwavrwaurkJYZmal5KY8MzOzCvCdH8zMSiiL648qxYnJzKykipqY3JRnZma54orJzKykXDGZmZlVgCsmM7OyKmbB5MRkZlZWbsozMzOrAFdMZmZlpOJWTE5MZmYlJKCgeclNeWZmli+umMzMSsm3JDIzs5wpaF5yU56ZmeWLKyYzs5IqalOeKyYzM8sVV0xmZmWk4p5jcmIyMyshAS1aFDMzuSnPzMxyxRWTmVlJuSnPzMxyxb3yzMzMKsAVk5lZGblXnpmZ5Ulyd/FiZiY35ZmZWa64YjIzK6Xi3l3cFZOZmeWKKyYzs5IqaMHkxGRmVlZuyjMzM6uAUldMG317NSY8ckHWYRTCN4aNzjqEQnnn8qFZh1AYc+d9kXUIhbFMywpWOL6OyczM8sTXMZmZmVWIKyYzs5IqaMHkxGRmVlZuyjMzM6sAV0xmZiVV0ILJFZOZmeWLKyYzszJScc8xOTGZmZVQch1T1lEsHjflmZlZrrhiMjMrpeKOx+TEZGZWUgXNS27KMzOzfHHFZGZWUkVtynPFZGZmueKKycysjDwek5mZ5YnHYzIzM6sQV0xmZiVV1IrJicnMrKQKmpfclGdmZvniisnMrKTclGdmZvlR4O7ibsozM7NcccVkZlZCKvDdxV0xmZlZrrhiMjMrqYIWTE5MZmZl1aKgmclNeWZmliuumMzMSqqgBZMrplo54tDhrNatC316r591KLnWQuL+07/HP04eAMDIo/vz2B9258Hf78oFh21Oq5YF/UurIr+3mu/NN97gewO3p2/v9dlkow246ILzsw6paqTkAttKPWrJialGDhx2MDfdcnvWYeTeUTutywtTPpj/+roJr7LJj25mi1NvYbllW3HQgJ4ZRpdPfm81X6tWrfjtGX/g8UnPcPf9DzLy4ot4/rlnsw7LGnBiqpEtt9qaDh06ZB1GrnXr0IaBvbtx5T0vzZ9255NT5j9/4uXpdOvQJovQcs3vreb7Rteu9N5oYwDatWvHOuuuy5S33so4quppoco9ahp3bXdntnC/O7APP7v6P3wRX5/XqqXYZ8s1GffUlK/PNFsMr7/2Gk9NmkTfTTfLOpSqcVOe2RLYcaPuTPvgE5587b0Fzj/rkE158PmpPPTCtBpHZmU0e/Zshu63F7//49m0b98+63CsAffKs1zYbO3O7NxnVQb27k7rZVrSbvll+MtR/TnizxP48R4b0Kndcgy99L6sw7QS+Pzzzxm6757sve/+DBq8R9bhVFUtCx1JywH3A61Jcsv1EfFzSWsA/wA6AhOBAyPis8a2lWnFJOkkSc+kjxMl9ZD0nKS/Spos6Q5Jy6fLriXpdkkTJT0gad0sY7fK+tU1k+h13I1858R/MeKC8dz/7Dsc8ecJHDigJ9tt0I0RF4wnFtDEZ7YoIoJjjjiUddb9Nsed8IOswymbT4HtImJDoDewk6TNgTOAcyKiJ/A+MKKpDWWWmCT1AQ4BNgM2Bw4DVga+BVwYEb2AmcD301VGAsdFRB/gZOCimge9BA4auh8DturHiy+8wFo9VuWyUZdmHVIhnDN8U7qsuBx3/nJHHvjt9zhlyAZZh5Q7fm8130MPTuDqv4/mvnvvYYtNN2aLTTdm7O23ZR1WVYj0Rq4V+teUSMxOXy6TPgLYDrg+nX45MLipbWXZlLclcGNEzAGQdAOwFfBqRExKl5kI9JDUFtgCuK7eSbjWC9qopMOBwwG+udpq1Yt+EV0x+uqsQyiM8c9NZfxzUwHodNDfM44m//zear4t+m/JrE/mZR1GzVS4N10nSY/Xez0yIkbWX0BSS5LP7Z7AhcDLwMyImJsu8ibQvakd5fEc06f1ns8Dliep7GZGRO+mVk4P1EiAPn36uvHHzKwypkdE38YWiIh5QG9JKwE3Aot1yiXLc0wPAIMltZG0AjAknfY1EfEh8KqkvQCU2LB2oZqZFUwFu4ovanfxiJgJ3AP0A1aSVFcErQo0eeFYZokpIp4ALgMeBR4BLiE5MbYwBwAjJD0JTAYGVTtGM7Mikyr3aHpf6pxWSqSd1r4LPEeSoPZMFxsG3NTUtjJtyouIs4GzG0xev978P9Z7/iqwU41CMzOzRdMVuDw9z9QCuDYibpH0LPAPSb8B/gM02Tsnj+eYzMxsCYnajscUEU8BGy1g+ivApouyLScmM7OS8rAXZmZmFeCKycyspGp989VKccVkZma54orJzKyEmtvNO4+cmMzMSqqWvfIqyU15ZmaWK66YzMxKqpj1khOTmVlpFbVX3kITk6Q/kYylsUARcXxVIjIzs6VaYxXT443MMzOzHEtuSZR1FItnoYkpIi6v/1pSm4j4qPohmZnZEluM4SryosleeZL6pXeHfT59vaGkQg1rbmZmxdGc7uLnAjsCMwAi4klg62oGZWZmS66W4zFVUrOuY4qINxpMmleFWMzMzJrVXfwNSVsAIWkZ4ASSUQnNzCzHinqOqTmJ6UjgPKA7MAUYCxxTzaDMzGzJlLJXXp2ImA4cUINYzMzMmtUrb01JYyRNk/SupJskrVmL4MzMbPEp7TJeiUctNafzw9+Ba4GuQDfgOuDqagZlZmZLThV81FJzElObiLgyIuamj9HActUOzMzMlk6N3SuvQ/r035JOBf5Bcu+8fYDbahCbmZktJqm44zE11vlhIkkiqvvJjqg3L4CfVCsoMzNbcgXNS43eK2+NWgZiZmYGzRyPSdL6wHrUO7cUEVdUKygzM1typb3AVtLPgQEkiek2YGdgPODEZGZmFdecXnl7AtsD70TEIcCGwIpVjcrMzJZYUW/i2pymvI8j4gtJcyW1B94FvlnluMzMbAkIlbJXXp3HJa0E/JWkp95s4KGqRmVmZkut5twr7+j06cWSbgfaR8RT1Q3LzMyWSAZNcJXS2AW2Gzc2LyKeqE5IZmZWCWXslXdWI/MC2K7CsVTcB598zm2T3846jEJ45/KhWYdQKL8Y+0LWIRTGiVv6ksjmmvdFZB1CLjR2ge22tQzEzMwqq1lDlOdQsy6wNTOzYhHFbcorakI1M7OScsVkZlZSRR1avTkj2ErSUEk/S1+vJmnT6odmZmZLo+Y05V0E9AP2S1/PAi6sWkRmZlYRLVS5Ry01pylvs4jYWNJ/ACLifUnLVjkuMzNbAsk97orZlteciulzSS1Jrl1CUmfgi6pGZWZmS63mVEznAzcCXSSdTnK38dOqGpWZmS2xonZ+aM698q6SNJFk6AsBgyPiuapHZmZmS6SgLXnNGihwNeAjYEz9aRHxv2oGZmZmS6fmNOXdSnJ+SSRDq68BvAD0qmJcZma2BATlHY8pIjao/zq96/jRC1nczMxyoqi39lnkuNPhLjarQixmZmbNOsd0Ur2XLYCNgSlVi8jMzCqioC15zTrH1K7e87kk55z+WZ1wzMxsaddoYkovrG0XESfXKB4zM6sASeXr/CCpVUTMldS/lgGZmVllFDQvNVoxPUpyPmmSpJuB64A5dTMj4oYqx2ZmZkuh5pxjWg6YAWzHl9czBeDEZGaWY2W8JVGXtEfeM3yZkOpEVaMyM7MlUtYLbFsCbflqQqrjxGRmZlXRWGJ6OyJ+VbNIzMysogpaMDWamAr6I5mZGRmMPFspjd2SaPuaRWFmZpZaaMUUEe/VMhAzM6ssFbThq6g3nzUzs5JqznVMZmZWMEl38ayjWDxOTGZmJeXEZF9z4c9/wOP338WKHTpx7j/vAeCsU45gymsvAzBn1oes0K49Z117V5Zh5tIRhw7n37fdQucuXZg46Zmsw8mdD6e9za1n/5g5M2eARO8d96bvoIPmz3/0hlHcM+pMjrvqIdqsuHKGkebPJhusTdt2bWnZoiUtW7Vi7L0PZR2SNeDEVEUDdt+Hnfc9hPNPO2H+tB+e+Zf5zy8765e0adtuQasu9Q4cdjBHHn0shw4/qOmFl0ItWrZk2xE/5hs9e/HpR7O5/MTv02OjLei0Wk8+nPY2r/5nAu07d8s6zNy6fswddOzYKeswqk4FvZDJnR+qqFefzWnbfsHfViOCB++4mS13GlzjqIphy622pkOHDlmHkVttO3ThGz17AdC6TVs6fnMtZs2YCsC4v/6ObQ/5ka9EXMrVnWOq1KOWnJgy8uwTj7BSx850W33NrEOxgvtg6ptMfeU5uq2zIf99eBztOq5ClzXXzTqs3JJg3yG7MHCbzbnyskuyDscWILOmPEkrAftHxEVZxZCl8bf/y9WSLbHPPp7Djb89nu0P+wktWrTkoWv/wj6/vjTrsHLtptvvoWu37kyf9i77DP4ePb+1Dv36b5V1WJWn4t6SKMuKaSXg6Az3n5l5c+fyyLjb6L/j7lmHYgU2b+7n3Pjb41lvwG6ss8VAZr7zPz6Y+iajjhvEn4dvx6zpU7nsxD2Y/f60rEPNla7dugPQqXMXdt51EJOeeCzjiKqnRTqKbSUeNY27pnv7qt8Da0maJOlvknYHkHSjpFHp8+GSTk+fnyTpmfRxYoZxL7GnHnmA7mv0pOMqPjltiyci+Pd5p9Hxm2ux6ZBDAOjcYx2Ou+pBjhp1N0eNupt2nVbh4HNvoO3KnTOONj8+mjOH2bNmzX9+3z13sc63e2UclTWUZWI6FXg5InoDY4G6Wro7sF76fCvgfkl9gEOAzYDNgcMkbVTjeBfZ2acexU+G7caU11/msIF9uOvGvwMw/vab3IzXhIOG7seArfrx4gsvsFaPVblslJun6nvr2SeYfM9N/O+ph/nbcYP523GDefmx+7IOK/emTZvKoJ22Zfv+fdl5+/5sP3Bnttthx6zDqooid37IS3fxB4ATJa0HPAusLKkr0A84HhgO3BgRcwAk3UCStP7TcEOSDgcOB+jUtXttol+Ik37/5wVOP+7X59Y4kuK5YvTVWYeQa6v26sOPb3m+0WWOGnV3jaIpjtV7rMm4CY9nHYY1IReJKSLeSjtD7ATcD3QA9gZmR8SsRemLHxEjgZEAPXtt6AENzWyp5c4Pi24WUP/q0oeBE0kS0wPAyen/pP8PltRG0grAkHrzzMzsa0SLCj5qKbOKKSJmSJog6Rng3ySJZmBEvCTpdZKq6YF02SckXQY8mq5+SUR8rRnPzMyKL9OmvIjYv8GkS9PpnwMrNFj2bODsGoVmZlZoorZNeZK+CVwBrAIEMDIizpPUAbgG6AG8BuwdEe83ti3f+cHMrIwq2COvmb3y5gI/jIj1SHpPH5N2aDsVGBcR3wLGpa8b5cRkZmZLLCLejogn0uezgOdILv8ZBPL5dYUAABLRSURBVFyeLnY50OS1MrnolWdmZpVX6zs21JHUA9gIeARYJSLeTme9Q9LU1ygnJjMza45OkupfBDYyvTznKyS1Bf4JnBgRH9a/3CciQlKTl/E4MZmZlVAVOj9Mj4i+je5TWoYkKV0VETekk6dK6hoRb6c3Tni3qR35HJOZWUnV8iauSkqjS4Hn0l7UdW4GhqXPhwE3NbUtV0xmZlYJ/YEDgaclTUqn/T+SG3ZfK2kE8DrJXX0a5cRkZlZStez7EBHjWfi4ydsvyracmMzMSkgU91xNUeM2M7OScsVkZlZGgkUZmSFPnJjMzEqqmGnJTXlmZpYzrpjMzEooGVq9mDWTKyYzM8sVV0xmZiVVzHrJicnMrLQK2pLnpjwzM8sXV0xmZqUkX8dkZmb54VsSmZmZVYgrJjOzknJTnpmZ5Uox05Kb8szMLGdcMZmZlVGB7y7uisnMzHLFFZOZWQkVubu4E5OZWUm5Kc/MzKwCXDGZmZVUMeslJyYzs9IqaEuem/LMzCxfSl0xLb9MKzbqvnLWYVgJHbHZ6lmHUBhvvvdx1iEUxudzv6jYtpJeecUsmUqdmMzMlmZuyjMzM6sAV0xmZqUkVNCmPFdMZmaWK66YzMxKqqjnmJyYzMxKqMi98tyUZ2ZmueKKycysjOSmPDMzy5miJiY35ZmZWa64YjIzK6miXsfkxGRmVkICWhQzL7kpz8zM8sUVk5lZSRW1Kc8Vk5mZ5YorJjOzkipqd3EnJjOzknJTnpmZWQW4YjIzK6Eidxd3YjIzKyUPFGhmZlYRrpjMzMrIdxc3M7O8KWheclOemZnliysmM7MSSnrlFbNmcsVkZma54orJzKykilkvOTGZmZVXQTOTm/LMzCxXXDGZmZWU7/xgjXrlpRfZZcBm8x/fWaMLoy7+U9Zh5dYRhw5ntW5d6NN7/axDyT2/txr3q1OOYeAmPdlnp37zp533u/9jzx02Yb+dt+BHRx7ArA9nZhhh9UiVe9SSE1ONrNlzbW699xFuvfcRbh73IMst34Ydd9k967By68BhB3PTLbdnHUYh+L3VuF333J/z/3b9V6ZttuW2/OP2h7j63w+yWo+eXHbRORlFZwvixJSBB++/h9V7rEH3b66edSi5teVWW9OhQ4eswygcv7e+buNN+9N+pZW/Mm3zrbajVavkTMb6G/Vl6jtTsgit6lTBRy05MWVgzI3Xsdsee2cdhpWQ31uL7ubrRrPFgB2yDqM6CpqZqpKYJK0k6ej0+QBJt1RjP0X02WefMW7srey8+x5Zh2Il4/fWoht14R9p1aoVOw9yMs+TalVMKwFHL8oKklpWKZZcuW/cWHp9pzedu6ySdShWMn5vLZox11/F+LvH8utz/ooKeuuexiSFTuX+1VK1uov/HlhL0iTgc2COpOuB9YGJwNCICEmvAdcA3wXOlPQe8EugNfAycEhEzJbUBzgbaAtMBw6OiLerFHtVjbnhWnYb4m9nVnl+bzXfg/fdxZUjz+cvV9/Kcsu3yToca6BaFdOpwMsR0Rv4EbARcCKwHrAm0L/esjMiYmPgLuA0YIf09ePASZKWAf4E7BkRfYBRwOkL27GkwyU9Lunx92ZMq8KPtvg+mjOH8ffdzY67Dso6lNw7aOh+DNiqHy++8AJr9ViVy0ZdmnVIueb31sL99PgRDP/+QF5/5b/sssV63HTNFfzhFz9izpzZHHPQYPbfZUt+99MfZB1m5VWwq3itC8paXWD7aES8CZBWUT2A8em8a9L/NydJXBPSsnpZ4CFgHZJK6850ektgodVSRIwERgJs0LtPVPjnWCJtVliBJ158K+swCuGK0VdnHUKh+L21cKef//UvNYP2OSiDSGqvqA2UtUpMn9Z7Pq/Bfuek/wu4MyL2q7+ipA2AyRHRDzMzK71qNeXNAtot4joPA/0l9QSQtIKktYEXgM6S+qXTl5HUq6LRmpmVUUG7i1elYoqIGZImSHoG+BiY2ox1pkk6GLhaUut08mkR8aKkPYHzJa2YxnwuMLkasZuZlUPte9NVStWa8iJi/4VMP7be8x4N5t0NbLKAdSYBW1c4RDMzyyHfXdzMrKSKenmWE5OZWQllcY+7SvG98szMLFdcMZmZlVVBSyZXTGZmlitOTGZmJVXLm7hKGiXp3fQyobppHSTdKem/6f8rN7aNOk5MZmYlVeN75V0G7NRg2qnAuIj4FjAufd0kJyYzM1tiEXE/8F6DyYOAy9PnlwODm7Mtd34wMyupHPR9WKXeEEXvAM0aLMyJycysjCp/IVMnSY/Xez0yHc2hWdIx+Jo14oMTk5mZNcf0iOi7iOtMldQ1It6W1BV4tzkr+RyTmVlJ5WBo9ZuBYenzYcBNzVnJFZOZWQmJ2t4rT9LVwACSJr83gZ8DvweulTQCeB3YuznbcmIyM7Ml1nCQ13q2X9RtOTGZmZVUDnrlLRafYzIzs1xxxWRmVlYFLZmcmMzMSqqoQ6u7Kc/MzHLFFZOZWUl5aHUzM8uVguYlN+WZmVm+uGIyMyurgpZMrpjMzCxXXDGZmZVQMupFMUsmJyYzszJq/pDoueOmPDMzyxVXTGZmJVXQgsmJycystAqamdyUZ2ZmueKKycyslJZoSPRMOTGZmZVUUXvllToxPfPkE9PX7Lz861nH0UAnYHrWQRSIj1fz+Vg1X16P1epZB5AHpU5MEdE56xgakvR4RPTNOo6i8PFqPh+r5lsajpUobN8Hd34wM7N8KXXFZGa2VCtoyeTEVHsjsw6gYHy8ms/HqvmWimNV1F55bsqrsYhYKv4gKsXHq/l8rJrPxyrfXDGZmZWUu4ubmVmuFDQvuSnPzMzyxRVThiQpIiLrOMyshDweky2m5eq/kIr6Nqo+SZ0lrZR1HHnl987iW9ixk1SCz0dV8FE7rpgyImkEsJ2k/wGPRsSNERGuor5O0vHAjsD7kt6IiJ9kHVPe1L1nJO0JrAs8B0yMiNeyjCvv6v+9SToYWJ7kc/HPETE3y9iWZiX4RlA8kvYGfkhyLcWnwDaSjoYvP2AsIWlfYBAwFJgDbJBtRPkl6Qjg58BsYBhwqKRNso0q3+olpROBg4A3gCOBw7OMqxJE0pRXqUctOTHVmKRvAKsBf4iI+4BzgAeA3pLaZxpcPn0M/JjkQ6MHMARA0kYZxpQ7kloCWwAHR8S5wCnprP7ZRVUMkpYDekXEdkAv4FXgL5LaZBvZ0suJqYbSb7SHk3zzP1TSGhHxATAGWBP4Rpbx5VQ74C7guxGxY0R8LulQYISk5TOOLRckbQq0Iblb9lBJy0XE8yTHbVdJK2QaYM4s4JxSC6CLpBuATYHvR8Q8YF9JA2odXyUV8wyTzzHVjKTdSJqhzoqIVyV1Bn4i6UKSW90vD8zMMsa8kHQIyTEZFxGjJa1D8gH7bWAgMALYLyI+zjLOPJDUATgQuBX4FzCYpLocCbQnadazVINzSlsDU4AZJMfrr8A+EfGppINImtu/l1mwFVDULjFOTFWW9uxZFvhtOul36f/Xk5w7ORf4CDgmIt6tfYT5ImkQcBTwCHBI2mR3Osm5uB8CK5B8eDyXXZT5IOnbEfGcpCeBQRFxVPqFZy9J+5BUm4dFxJxsI82PBueUhgD3A32B04BfAldJuj2dtndE5G08t6WCfK69uiStFBEzJbUDbgImRcRJ9ea3Bz73t//5SeknwJCIeFvSHsDWwEvAyIj4TFIr95YCSVsBl5J8wfkFcAswPiJ+lX4ZWhd4NyLyOBhezTWolHoB50bEdyVdAHQh+bITktYGvgA+joi3Mgx5iW24UZ8Ye+/DFdte15WWnVirMax8jqmKJB0JXCrpdGA7km9o20g6o26ZiPjQSQnSE83vkHygHgwQETcA9wK9gcPTD9x5GYWYG5JaAU8Dz5M0250I3AgMlLRpRHwREc86KSUaJKWNSBLRZEknk5zbHZYmpe+RJPOXip6U5ivoSSY35VWJpKEkXZwPBc4DOkTETZJ2BB6W9FlE/F+mQeZEmsA3B14HTgJOkfRORPwtIv4laS7wWER8kWmgGZG0LNAzIp6VNBBYH7gPOAw4Avic5KNjC+D76eisS+WxWpB6SWkPkvOTBwI/BXaKiHXTeYcB+wAPZhWnfcmJqQok9SZpDjgM6EfyoXFc2qX3Y2ATwHcxACR9HziWJIkPBz4A/g4cKalDRJwVEbdkGWMOrAacK2kqybf9MSRd6N8HHgL+FxF3S3oZeM1J6evSzgw7kHQ+ek/SuSRJ/ArgSZL334ERUaoOSAXt++CmvEpL/wD+AWxMcmL1gIgYmJ4XOQw4HpgZEa9mGGaerANcFhGTgJNJOoKsDJxA0hNvpaX9djsR8RLwFElnmesi4gKS91F34P8BV0vqGhFjI+KFDEPNjQXcTmg5YBdg7fT13cD/kVyzNAfYPyKeqV2E1VfJi2tr/RfoiqmCJJ1AcsX9JyTnRloCrdN7vA0Bjgb29d0dvuJZkt53t0XEs8DFku4BLgR2johPsg0vNy4m+WZ/UtoMfAVJ4j4K2IzkcgNL1VWNaUeHlyJipKQPgd9ImhwRD5B8Cfp5lnHagjkxVYikbYE9Sc6VHAasSPJhcjBwFclJ+wPSD1/70r0kXXMPkHQvyQdsW5Kq0kkplVZNL0maSfLh+gFJU1434NiI8PVKzO/csFVEnJ/e5usYYIqkiyPiH5KWAUZKOjoi7sk22uor6tDqTkyV8zRJN+fPJK0GdIuIq9Jv/08B10fE59mGmD9pV/qLgD2AH5FcEHqoe5QtWESMkfQ5cCYwl+TLjpMS8+/o0AnYRVJXkh53/Um+MA6U1C4iLpPUGvijpC3dIzafnJgqpMEH6YPAapK+C/yG5MPDSWkhImIKcIGkUSTX1vmC0EZExO2SJqbPp2UdTx5I6gKsHBF3StoO2B54M+3McImkT4Ct09s1XSzp2qUiKRWzYHJiqpJ3SDpAvAAMjYj/ZhxPIUTER1nHUBROSF+zIsmXmzdJbmd1HvADScdHxPnpra2WA74jacX0HpWlV9C85MRUJf8FxgI/clIyq76I+G96a6bDgR+nzejvAUekF9ieFxGXSGofER9mHK41wYmpCtLrJPaJiE+zjsVsKVK/5+J7EXGNpHeBiyTNiIjRS1tSKuqFFk5MVeKkZFZbDXounp7+vxzwGTAh0+AyIffKMzPLg3o9F/9IcvHsCF/QXixOTGZWOmnPxSeSp0tnR5G6odWLyInJzErJ45sVl++VZ2ZmueKKycyspIralOeKyczMcsUVk5lZSRW1u7grJss9SfMkTZL0jKTr0mHYF3dbl0naM31+iaT1Gll2gKQtFmMfr0nq1NzpDZZZpBuySvpFOkS42VcVeDwmJyYrgo8jondErE9yseSR9WdKWqzKPyIObWIYkgEkw5WbWQ05MVnRPAD0TKuZByTdDDwrqaWkP0h6TNJTko6AZCgESRdIekHSXSRDk5POu1dS3/T5TpKekPSkpHGSepAkwB+k1dpWkjpL+me6j8ck9U/X7SjpDkmTJV1CM+6dKelfkiam6xzeYN456fRxkjqn09aSdHu6zgOS1q3EwbTyUoUfteRzTFYYaWW0M3B7OmljYP2IeDX9cP8gIjZJx9uZIOkOYCOS4dvXA1YhGTF3VIPtdgb+CmydbqtDer/Di4HZEfHHdLm/A+dExPh0zK2xwLdJRkEdHxG/krQLMKIZP87wdB/LA49J+mdEzABWAB6PiB9I+lm67WOBkcCR6c1KNwMuArZbjMNoS5NinmJyYrJCWF7SpPT5A8ClJE1sj9a71cxAkiEN9kxfrwh8C9gauDoi5pGMZHr3Ara/OXB/3bYi4r2FxLEDsJ6+bHBvL6ltuo890nVvlfR+M36m4yUNSZ9/M411BvAFcE06fTRwQ7qPLYDr6u27dTP2YVZITkxWBB9HRO/6E9IP6PoDCgo4LiLGNljuexWMowWwecMh37WIZ4YlDSBJcv0i4iMlQ8ovt5DFI93vzIbHwKwp7pVnlq2xwFGSlgGQtLakFYD7gX3Sc1BdgW0XsO7DJKObrpGu2yGdPgtoV2+5O4Dj6l5IqksU9wP7p9N2BlZuItYVgffTpLQuScVWpwXJUOCk2xyfDtXwqqS90n1I0oZN7MPMvfLMMnYJyfmjJyQ9A/yFpEXgRpKBG58FrgAearhiepPPw0mazZ7ky6a0McCQus4PwPFA37RzxbN82TvwlySJbTJJk97/moj1dqCVpOeA35MkxjpzgE3Tn2E74Ffp9AOAEWl8k4FBzTgmZoWkiMg6BjMzq7CN+/SN8Q89VrHtrdC6xcSI6FuxDTbCFZOZmeWKOz+YmZVVMfs+ODGZmZWVe+WZmZlVgCsmM7MSKvLQ6u6VZ2ZWQpJuBxq9m/0imh4RO1VwewvlxGRmZrnic0xmZpYrTkxmZpYrTkxmZpYrTkxmZpYrTkxmZpYr/x9POT/sDO64LQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAGlCAYAAABA7gkgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3wUdf7H8dcHAiq9CZKEDkoVkKLCKYiiCIgFC3YUj7OenvqzHyq2U8966tnPLio2EBQr6nEqTSyAIlIkCdKLFIGEz++PGUISIFlks7sZ3k8f+zAz893vfHay7Cef73x3xtwdERGRVFQu2QGIiIjsiJKUiIikLCUpERFJWUpSIiKSspSkREQkZSlJiYhIykpLdgAiIhJ/5as1cs9dH7f+fP2Sce7eJ24dxkhJSkQkgjx3PXvsd3Lc+vt92sN14tbZTtBwn4hIJBlYufg9YtmjWR8z+9HMZpvZNdvZ3sjMPjKzb81svJllltSnkpSIiOwyMysPPAwcDbQGTjWz1kWa/RN4zt33B4YDd5TUr5KUiEgUGWAWv0fJugKz3X2Ou28ERgDHFmnTGvg4/PmT7WzfhpKUiEhUJXa4LwNYUGA5K1xX0DfACeHPxwNVzax2cZ0qSYmISCzqmNnkAo+hf6CPK4EeZvY10APIBvKKe4Jm94mIRFVsw3SxWurunYvZng00KLCcGa7L5+45hJWUmVUBBrr7yuJ2qiQlIhJJFvOsvDiZBLQwsyYEyWkQcFqhiMzqAMvdfTNwLfB0SZ1quE9ERHaZu+cCFwPjgJnAq+4+3cyGm9mAsFlP4EczmwXUA24rqV9VUiIiURXf4b4SuftYYGyRdcMK/DwSGLkzfaqSEhGRlKVKSkQkioxEn5MqFUpSIiKRFPOXcFNa2U+zIiISWaqkRESiSsN9IiKSsjTcJyIiUnpUSYmIRFLCrzhRKsr+KxARkchSJSUiEkVb7idVxilJiYhElYb7RERESo8qKRGRSIrGxAklKRGRqCpX9s9Jlf00KyIikaVKSkQkinQVdBERSWkRmIJe9tOsiIhEliopEZFIisbsvrL/CkREJLJUSYmIRFUEzkkpSYmIRJWG+0REREqPKikRkSgy03CfiIikMA33iYiIlB4lKdkhM9vLzEab2Soze20X+jndzN6PZ2zJYGbvmtnZyY5DJGZbhvzi8UgSJakIMLPTzGyyma0xs4Xhh+mf4tD1iUA9oLa7n/RHO3H3F939yDjEU4iZ9TQzN7M3i6xvH64fH2M/N5nZCyW1c/ej3f3ZPxjujvZ9SPh7W2Nma8O41xR4NPwDfbqZNS9m+2Azyyuwj7lm9h8z23cn9vGMmd26s7HtrETtR1KXklQZZ2aXA/cDtxMklIbAI8Cxcei+ETDL3XPj0FdpWQIcbGa1C6w7G5gVrx1YoFT+rbj75+5exd2rAG3C1TW2rHP3X0pjv8AX4T6rA0cA64EpZta2lPYnCRdecSJejyRRkirDzKw6MBy4yN3fcPe17r7J3Ue7+/+FbfYws/vNLCd83G9me4TbeppZlpldYWaLwyrsnHDbzcAw4JTwr+0hRSsOM2sc/tWeFi4PNrM5ZvZb+Nf56QXW/7fA87qZ2aRwGHGSmXUrsG28md1iZhPCft43szrFHIaNwFvAoPD55YFTgBeLHKsHzGyBma02sylmdki4vg9wXYHX+U2BOG4zswnAOqBpuO68cPu/zez1Av3faWYfmcVvXMTMqpvZU+HvJdvMbg1fH2bW3Mw+DY/hUjN7JVz/Wfj0b8LXc0px+3D3PHf/2d0vBD4Fbiqw/9fM7NdwH5+ZWZtw/VDgdOCqcB+jw/XXmNnP4e9thpkdX6Cv7cYbbmtpZh+Y2XIz+9HMTi5uP7ITNNwnSXYwsCfwZjFtrgcOAjoA7YGuwA0Ftu9D8Nd0BjAEeNjMarr7jQTV2SvhX/RPFReImVUGHgSOdveqQDdg2nba1QLGhG1rA/cCY4pUQqcB5wB1gYrAlcXtG3gOOCv8+SjgeyCnSJtJBMegFvAS8JqZ7enu7xV5ne0LPOdMYChQFZhfpL8rgHZhAj6E4Nid7e5eQqw74xkgF2gOdASOBM4Lt90CvA/UBDKBfwG4+6Hh9vbh63mF2L0BHFJg+V2gBcHvYSph4nf3x8Of7wr3cUzY/ufw+dWBm4EXzKx+cfGG75sPCH4ndQn+2HjEzFoXsx/ZjShJlW21gaUlDMedDgx398XuvoTgw+PMAts3hds3uftYYA2w3x+MZzPQ1sz2cveF7j59O236AT+5+/PunuvuLwM/AAU/gP7j7rPcfT3wKkFy2SF3/x9Qy8z2I0hWz22nzQvuvizc5z3AHpT8Op9x9+nhczYV6W8dwXG8F3gBuMTds0roL2ZmVg/oC1wWVsiLgfsIK0aC31sjIN3df3f3/+6gq52RQ5DEAXD3p939N3ffQFBhtQ+r9+1y99fcPcfdN4fJ8SeCP4qKi7c/MM/d/xMe56+B14E/fA5UQlvuJ6XhPkmiZUCdLcNtO5BO4Spgfrguv48iSW4dUGVnA3H3tQTDbOcDC81sjJm1jCGeLTFlFFj+9Q/E8zxwMXAY26kszexKM5sZDjetJPhrv7hhRIAFxW1096+AOQQfB6/uqJ2ZTbetkxQO2VG7IhoBFQiO5cow5scIqg2Aq8L9Tgz7PzfGfouTASwPYy5vZv8Ih+9WA/PCNjs8ZmZ2lplNKxBv2wLtdxRvI+DALc8Jn3c6QYUvuyQa56T0Zd6y7QtgA3AcMHIHbXIIPgi2VDUN2XYoLFZrgUoFlgt9kLj7OGCcme0F3Ao8QeHho4LxFNQQeO8PxrTF88Bs4Dl3X1fw1FCYGK4CDgemu/tmM1tB8KEJsKMhumKH7szsIoKKLCfs/47tduLeZnvrS7CA4HdbZ3uVsrv/Cvw5jONPwIdm9pm7z/4D+9rieODz8OfTCCbfHEGQoKoDOzxmZtaI4Pd9OMGkjDwzm7al/Y7iDV/np+7eewcxxXP4VMogVVJlmLuvIpjc8LCZHWdmlcysgpkdbWZ3hc1eBm4ws73DCQjDCIan/ohpwKFm1jAc9rl2ywYzq2dmx4bnGDYQDBtu3k4fY4F9LZg2nxae2G8NvPMHYwLA3ecCPQjOwRVVleDczhIgzcyGAdUKbF8ENLadmMFnwXTtW4EzCIb9rjKzYocld4a7LyQ4h3OPmVUzs3Jm1szMeoT7P8nMMsPmKwg+zLcc70VA0xhfR3kza2Jm/wJ6EgwHQ3DMNhBU65UIztsVVHQflcMYloT9nkNQSW3Zz47ifYfg/XBm+N6tYGZdzKzVzr4W2Q5NnJBkC8+vXE4wGWIJwV+mFxPMeIPgg3Qy8C3wHcEJ8D/0vRN3/wB4JexrCoUTS7kwjhyCIaMewAXb6WMZwXmIKwg+AK8C+rv70j8SU5G+/+vu26sSxxFUarMIhhZ/p/BQ3pYvKi8zs6kl7SccXn0BuNPdv3H3nwhmCD5v4czJODmLYOLIDIIP9pHAlokIXYCvzGwNMAq41N3nhNtuAp4Nh89O3kHfB4fPXQ2MJ0jaXdz9u3D7cwTHKjvc/5dFnv8U0Drcx1vuPgO4h6C6XwS0AyYUaL/deN39N4IJIYMI3ju/AncSVKjb7KekAybRY/GdjCQiIqmgXI1GvkeP6+LW3++jzp/i7p3j1mGMVEmJiERVgof7zKxP+F232WZ2zXa2NzSzT8zsazP71sz6ltSnkpSIiOwyC75o/jBwNMF55lPNrHWRZjcAr7p7R8LvxJXUr2b3iYhEkVmip453BWZvOTdqZiMIZojOKNDG2TppqToxzDRWkhIRiarEzsrLoPCEpCzgwCJtbgLeN7NLCGaEHlFSpxruExGRWNSx4G4LWx5D/0AfpxJcySWT4Ioqz5f01Y9IV1Jplap7hRr1kh1GmbDfPlWTHUKZsilXs2JjlVa+7N/CPFGyfpnPsmVL43mR4nh1BcEl2Iqb3ZcNNCiwnBmuK2gI0AfA3b8wsz0JrkqyeEedRjpJVahRj+ZDSjwvJ8DH1/RMdghlStby9ckOocyoV33PZIdQZhzZ46C49WXEPUmVZBLQwsyaECSnQQRXLinoF4KrkjwTfmF7T8IvgO+IhvtERGSXhZfvupjgy/MzCWbxTTez4WY2IGx2BfBnC26J8zIwuKQ7B0S6khIR2W0ZW6+0mCDhnRTGFlk3rMDPM4DuO9OnKikREUlZqqRERCLJEn1OqlQoSYmIRFQUkpSG+0REJGWpkhIRiagoVFJKUiIiERWFJKXhPhERSVmqpEREoigJ35MqDaqkREQkZamSEhGJINP3pEREJJVFIUlpuE9ERFKWKikRkYiKQiWlJCUiElFRSFIa7hMRkZSlSkpEJIr0PSkREZHSpUpKRCSionBOSklKRCSCovJlXg33iYhIylIlJSISUVGopJSkRESiquznKA33iYhI6lIlJSISRRaN4T5VUiIikrJUSYmIRJQqKSmke/PajLr0YMZc1o0hhzTabpuj2tblrUsO4s1LDuLOk9rmr5928+G8duGBvHbhgTx4evtEhZw0H30wjq4d29B5/5bcf89d22zfsGEDQ846jc77t6R3z278Mn9e/rbp33/LUb3+RLfO7flT1w78/vvvCYw8OSaM/4ABPQ+g/yHteerhe7fZPuWrCZzS9xAOaFKTD8a8VWjbfbcP44QjDuSEIw7kvVGvJyrkpPn4w3F079SGgzq04l/3bv+9NXTwaRzUoRVH9+qe/97auHEjl154Hj0P7kiv7p2Y8PmnCY48/swsbo9kUSUVJ+UMrj9mP4Y+8zW/rv6dEed35ZMfljJnydr8Ng1r7cWQQ5tw1hOTWf17LrUqV8jftmFTHic98lUyQk+4vLw8rrr8r7w+6l3SMzI54tCD6NO3Py1btc5v88KzT1OjRg0mf/sDb7z2Cjf//Tqeeu4lcnNzOX/I2fz7yWdo2649y5cto0KFCsXsrezLy8vj9huu4LEX36Ze/QxOO6YnPXv3pdm+LfPb7JOeyS33/JtnH3uw0HM/++g9fvj+G159bwIbN27gvJP78qfDelOlarVEv4yEyMvL49orLuXVt8ZSPyOTPocdzJF9+7Nfy63vrZee+w81atTky2kzeWvkK9x643U8/sxLvPDsUwCM/+JrlixZzOkDj+G98V9Qrpz+lk8mHf04aZdZnV+WrSdrxXpy85x3v1vEYa32LtRmYOcMRny1gNW/5wKwfO2mZISadFMnT6RJ02Y0btKUihUrcvyJp/DumNGF2rw7ZjSDTj8TgAHHD+Sz8R/j7nzy0Qe0btuOtu2CarNW7dqUL18+4a8hkb6fNpkGjZuS2agJFSpWpM8xAxn//phCbTIaNGLfVm23+UCd89OPHHBgN9LS0qhUqTItWrVlwvgPExl+Qn09ZRJNmjajUfjeOu6EkxlX5L01buxoTj4teG/1P24g//30E9ydWT/M5E+H9gRg773rUq16DaZ9PSXRLyFutlxxoqxXUkpScVK32h78umrrsNOiVb9Tr+oehdo0rlOJRrUr8dx5nXlhaBe6N6+dv61iWjlGnN+VF4Z2oVeR5BY1C3NyyMjMzF9Oz8hgYU72Nm3SMxsAkJaWRrXq1Vm+bBk/z56FmXHisX05rHsXHrzvnwmNPRkW/7qQfdK3Hq+69dNZtCgnpufu27ot/xv/IevXr2PF8mVM+t/n/Lowu+QnllELc7JJz9h6rOpnZLBwYeFjtXDh1jZpaWlUrVad5cuX0abt/owb+w65ubnMnzeXb7+ZSk7WgoTGH3cWx0eSaLgvgcqXMxrVrsS5T0+hXrU9eOa8zpzw0Jf89nsuR90zgcW/bSCz5l48ec4BzPp1DVkr1ic75JSTm5vHV1/8jw8//YK9KlXi+P5H0r7DAfQ4rFeyQ0tJ3Q49nOnfTOXs43tTs1Yd2nfqQnkNX23XqWcO5qdZP3BUz4PIbNCQzl0PjnyVXhaUiXermaX8O2Xx6g3sU33P/OV61fdk0W8bCrVZtGoD439YQu5mJ3vl78xbuo6GtSsFzw/bZq1Yz+S5K2iVXjVxwSdY/fR0srOy8pdzsrOpn56xTZstf8Xm5uayetUqatWuTXp6Bgd3/xO169ShUqVK9D7yaL795uuExp9odfepz685W4/X4oU51KuXHvPz/3zJ//HqexN47KW3cXcaNW1eGmGmhPrpGeRkbz1WC7OzqV+/8LGqX39rm9zcXH5bvYpatWqTlpbG8Dv+yUf/ncyzL7/B6lUradq8RULjjyuLxsSJhCYpMzvfzKaFj7lm9omZHWlmX5jZVDN7zcyqhG3nmdmdZjYVOMnMTjWz78zsezO7M5Fxx+L77NU0qr0XGTX2JK28cXS7eoz/YUmhNh/PXELnJjUBqFGpAo3rVCJr+Xqq7ZlGhfKWv75Doxr8vHjtNvuIio6dujDn59nMnzeXjRs38ubIVzi6b/9Cbfr07c+IF58HYNSbr3NIj8MwM3odcSQzp3/PunXryM3NZcJ/P2O/lq2S8TISpk37Tvwydw5Zv8xj08aNvDf6dXr07hvTc/Py8li5YhkAs2Z+z6yZ0zn40MNLM9yk6nBA50LvrbfeeJUji7y3juzbn1dfCt5b77z1Ot0P7YmZsW7dOtauDf7dffrxh6SlpRWacFEWRSFJJXS4z90fBR41swrAx8DTwA3AEe6+1syuBi4HhodPWebuB5hZOvAl0AlYAbxvZse5+1tF92FmQ4GhABWq1S3117RF3mbn9nd+5NGzO1K+nPHm1Bx+XryWi3o1ZXrOasb/sJQJs5fRrXkt3rrkIDY73DPuJ1at30T7BtW58dhWbHannBlPfTav0KzAqElLS+POex7gpOP6kZeXx2lnDqZl6zbccctNdDigE0f3O4Yzzj6XC84bTOf9W1KjZk2efOZFAGrUrMkFl1zGEYcejJnR+6g+HNkntg/ssiotLY1rb7mbC848ns15eRx3ypk0368VD99zK23aHUDPI/vy/TdT+NufT2f1qpV8+uG7PHLv7bz50URyN23inIF9AKhctSq3P/AEaWnRHeVPS0vj9n/ez6kn9CMvbzOnnnE2LVu14c7bbqJDx04c1fcYTjvzHC4eOpiDOrSiRs2aPPb0CwAsXbKYU0/oR7ly5dinfgb/euw/SX41AmDunvidmj0CLAEmAc8AW+rzisAX7j7EzOYBPdx9vpkdCwx097PC5w8B2rj75cXtZ6/0fb35kEdK6VVEy6fX9Ex2CGVK1nKdL4xVvQLD4FK8I3scxDdfT4lL2VKxbnOvd9I98egKgKxHjpvi7p3j1mGMEv4nlZkNBhoBFwP9gA/c/dQdNI9uOSEiIiVK9DmpTsCVwBnuvplgCK+7mTUPt1c2s32389SJQA8zqxNOojgVKPtfBxcRKU0JnoJuZn3M7Eczm21m12xn+30F5iXMMrOVJfWZ6ErqYqAW8El4Im4yMBh42cy2fKnoBmBWwSe5+8LwBX9CcLjGuPvbiQpaRKQsSuSEh7CAeBjoTXAKZ5KZjXL3GVvauPvfCrS/BOhYUr+Jnjhxzg42ddlO28ZFll8GXi6FsEREZNd1BWa7+xwAMxsBHAvM2EH7U4EbS+o0utN8RER2Y0mYOp4BFLxERxZw4PYamlkjoAnBLO9iKUmJiERUnJNUHTObXGD5cXd//A/2NQgY6e55JTVUkhIRkVgsLWEKejbQoMByZrhuewYBF8WyUyUpEZGISvBw3ySghZk1IUhOg4DTthNTS6Am8EUsnZaJa/eJiEhqc/dcghnc44CZwKvuPt3MhpvZgAJNBwEjPMYrSaiSEhGJqgRfcs/dxwJji6wbVmT5pp3pU0lKRCSiknlh2HjRcJ+IiKQsVVIiIlFk0aiklKRERCLIgAjkKA33iYhI6lIlJSISScm9o268qJISEZGUpUpKRCSiIlBIKUmJiESVhvtERERKkSopEZEoMg33iYhIijKgXLmyn6U03CciIilLlZSISERpuE9ERFKWZveJiIiUIlVSIiJRFJHZfaqkREQkZamSEhGJoOBWHWW/lFKSEhGJJF0FXUREpFSpkhIRiagIFFJKUiIiUaXhPhERkVKkSkpEJIoi8j2pSCep1vWrMeHGI5IdRplQs8vFyQ6hTFkx6aFkh1BmbMrdnOwQyozyEbhqebxFOkmJiOyu9D0pERFJaRHIUZo4ISIiqUuVlIhIRGm4T0REUlYEcpSG+0REJHWpkhIRiSKLxnCfKikREUlZqqRERCIo+J5UsqPYdUpSIiKRpPtJiYiIlColKRGRiDKL3yO2/VkfM/vRzGab2TU7aHOymc0ws+lm9lJJfWq4T0QkohI53Gdm5YGHgd5AFjDJzEa5+4wCbVoA1wLd3X2FmdUtqV9VUiIiEg9dgdnuPsfdNwIjgGOLtPkz8LC7rwBw98UldaokJSISRXEc6ouxIMsAFhRYzgrXFbQvsK+ZTTCzL82sT0mdarhPRCSCSuFWHXXMbHKB5cfd/fGd7CMNaAH0BDKBz8ysnbuvLO4JIiIiJVnq7p2L2Z4NNCiwnBmuKygL+MrdNwFzzWwWQdKatKNONdwnIhJRZha3RwwmAS3MrImZVQQGAaOKtHmLoIrCzOoQDP/NKa5TJSkREdll7p4LXAyMA2YCr7r7dDMbbmYDwmbjgGVmNgP4BPg/d19WXL8a7hMRiahEX3DC3ccCY4usG1bgZwcuDx8xUZISEYkoXRZJRESkFKmSEhGJop24nFEqU5ISEYkg01XQRURESpcqKRGRiIpAIaVKSkREUpcqKRGRiCoXgVJKSUpEJKIikKM03CciIqlLSSqO3h/3Hvu32Y82LZtz913/2Gb7hg0bOOO0U2jTsjmHdDuQ+fPmAfDRhx/QrWsnOndoR7eunRj/yccJjjzxendrxTdv/p3v376RK8/pvc32hvVrMvbRS5j4yrWMe+JSMurWyN/29kMXsvCzu3j9gfMTGXJS6b0Vuw/ef48D9m9F+zb7cu/dd26zfcOGDQw+YxDt2+zLYYcczPz58wptX/DLL9SvU40H77snQRGXjuA+UAm9wGypUJKKk7y8PC7760W8Pfpdvv52Bq+NeJmZM2YUavPM009Rs0ZNpv8wm0su/RvXX3c1ALVr12HkW6OZPO07nnj6Wc4dfGYyXkLClCtn3H/NyRx78SN0HHgrJ/XpRMum+xRqc8ffjufFMRPpesod3P74uwy/ZED+tvue+5AhNzyX6LCTRu+t2OXl5XHFZZfw+ttjmPT194x8bQQ/zCx8rJ575mlq1KzJN9NncdEll3Lj9dcU2n7d1VfQ+8gS78VXJpSz+D2S9hqSt+tomTRxIs2aNadJ06ZUrFiRk04ZxDuj3y7U5p3Rb3P6mWcDcMLAExn/8Ue4Ox06diQ9PR2A1m3a8Pv69WzYsCHhryFRurRtzM8LljIvexmbcvN4bdxU+vfcv1Cblk3r8+nEHwH4dNIs+vdsl79t/MRZ/LY2usenKL23Yjd50kSaNmtGkybBsRp40imMeafw3SLGvPM2p55+FgDHnXAi48d/THDdU3hn1Fs0atyElq3bJDx22T4lqTjJyckmM3Pr/b4yMjLJzs7etk2DoE1aWhrVqldn2bLCV6l/843X6dDxAPbYY4/SDzpJ0utWJ2vRivzl7EUryNi7eqE2383K5theHQA4tld7qlXZi1rVKyc0zlSh91bsFhY5VukZGeQUOVYLc3Ly26SlpVGtWnWWL1vGmjVruO+eu7nm+mFERRSG+zS7L4XMmD6dG667mnfGvp/sUJLu2vve5L6rT+KMAQcyYepsshetIC9vc7LDKrP03irZHbfezEWXXEqVKlWSHYoUoCQVJ+npGWRlLchfzs7OIiMjY9s2CxaQmZlJbm4uq1etonbt2gBkZWVxyknH8+TTz9G0WbOExp5oOYtXkVmvZv5yRr2aZC9ZVajNwiWrGHTlkwBU3qsixx3egVVr1ic0zlSh91bs6hc5VjnZ2aQXOVb109PJylpAxpZjtXoVtWrXZvKkibz95usMu/4aVq1aiZUrxx577slfLrgo0S8jbjQFfReZ2eVm9n34uMzMGpvZTDN7wsymm9n7ZrZX2LaZmb1nZlPM7HMza5nM2Ivq3KULs2f/xLy5c9m4cSOvvTKCfv0HFGrTr/8AXnz+WQDeeH0kPQ7rhZmxcuVKThjQj1tu+wfdundPRvgJNXn6fJo33JtG6bWpkFaek446gDHjvy3UpnaNyvlDDP937lE8+/aXyQg1Jei9FbtOnbswZ/Zs5s0LjtXrr71C337HFGrTt98AXn4xmHjz1hsj6dHjMMyMcR99yvc/zuH7H+dwwcWXcuX/XVu2ExThRWbj9F+yJC1JmVkn4BzgQOAg4M9ATaAF8LC7twFWAgPDpzwOXOLunYArgUcSHnQx0tLSuO+Bhzim31F0aNeKgSedTOs2bRh+0zDeGR2cuB187hCWLV9Gm5bNefD+e7n1tmAq8aOPPMTPP8/mjluHc2CnDhzYqQOLFy9O5sspVXl5m/nbna8y+pGLmPbGDbz+/tfMnPMrf7+gH/16BBMkDu3cgm/f+jvfvjWMurWrcueT4/Kf/+FTl/Hi3edyWNd9mf3eLRxxcKtkvZSE0Hsrdmlpadx934Mcf8zRdO7QhuMHnkSr1m24dfiNjA0nUJw1+FyWL1tG+zb78tCD93PTrXckOWopjm2Z1ZLwHZtdCtTecmthM7sFWEKQiFqE664GKgD3h9t+LNDFHu6+zaeTmQ0FhgI0aNiw06yf55fq64iKml0uTnYIZcqKSQ8lO4QyY1OuziXGqkf3rkydMjkuZUuNRq380Ovj91WN0X/pOsXdO8etwxil4jmpgvNj84C9CCq+le7eoaQnu/vjBFUXnTp1Tk4GFhFJtiTPyouXZJ6T+hw4zswqmVll4Phw3TbcfTUw18xOArBA+8SFKiIiyZC0JOXuU4FngInAV8CTwIpinnI6MMTMvgGmA8eWdowiImWZWfweyZLU4T53vxe4t8jqtgW2/7PAz3OBaFyrREREYpKK56RERGQXGbqflIiIpLAI5Chdu09ERFKXKikRkYiKwhR0JSkRkQhK9qy8eNFwn4iIpCxVUiIiEaXZfSIikrLKforScJ+IiAxF6toAACAASURBVKQwVVIiIhEV6dl9ZvYvYIdXEXf3v5ZKRCIiIqHiKqnJCYtCRETiKrgsUrKj2HU7TFLu/mzBZTOr5O7rSj8kERHZZbvL/aTM7GAzmwH8EC63N7OUunW7iIhEUyyz++4HjgKWAbj7N8ChpRmUiIjsukTfT8rM+pjZj2Y228yu2c72wWa2xMymhY/zSuozptl97r6gSNmYF1vIIiKSLIkc7jOz8sDDQG8gC5hkZqPcfUaRpq+4+8Wx9htLJbXAzLoBbmYVzOxKYGasOxARkd1CV2C2u89x943ACOJwB/VYktT5wEVABpADdAiXRUQkRW2Z3RevB1DHzCYXeAwtsssMYEGB5axwXVEDzexbMxtpZg1Keh0lDve5+1Lg9JLaiYhIpC1198672Mdo4GV332BmfwGeBXoV94RYZvc1NbPR4cmuxWb2tpk13cVARUSklFk4DT0ejxhkAwUro8xwXT53X+buG8LFJ4FOJXUay3DfS8CrQH0gHXgNeDmG54mISBJZHB8xmAS0MLMmZlYRGASMKhSPWf0CiwOIYX5DLEmqkrs/7+654eMFYM/YYhYRkd2Bu+cCFwPjCJLPq+4+3cyGm9mAsNlfzWy6mX0D/BUYXFK/xV27r1b447vhfPcRBNfyOwUY+4dfiYiIlDqzxN9Pyt3HUiQ/uPuwAj9fC1y7M30WN3FiCkFS2vIq/1Jwvzu7IxERSawIXBWp2Gv3NUlkICIiIkXFdMUJM2sLtKbAuSh3f660ghIRkV0XhQvMlpikzOxGoCdBkhoLHA38F1CSEhGRUhXL7L4TgcOBX939HKA9UL1UoxIRkV2W6AvMloZYhvvWu/tmM8s1s2rAYgp/YUtERFKMYQmf3VcaYklSk82sBvAEwYy/NcAXpRqViIgIsV2778Lwx0fN7D2gmrt/W7phiYjILknyMF28FPdl3gOK2+buU0snJBERiYeoz+67p5htTglXrk0Fy9Zt5IUp85MdRpmwYtJDyQ6hTBkyYlqyQygz7urXKtkhlBl5mz3ZIaSc4r7Me1giAxERkfiKZfp2qovpy7wiIlK2GNEY7otCohURkYhSJSUiElHlyn4hFdOdec3MzjCzYeFyQzPrWvqhiYjI7i6W4b5HgIOBU8Pl34CHSy0iERGJi3IWv0eyxDLcd6C7H2BmXwO4+4rw1sAiIpKigmvulf3xvlgqqU1mVp7gu1GY2d7A5lKNSkREhNgqqQeBN4G6ZnYbwVXRbyjVqEREZJdFYeJELNfue9HMphDcrsOA49x9ZqlHJiIiuyQCo30x3fSwIbAOGF1wnbv/UpqBiYiIxDLcN4bgfJQR3D6+CfAj0KYU4xIRkV1gsHvcT8rd2xVcDq+OfuEOmouIiMTNTl9xwt2nmtmBpRGMiIjETxSuexfLOanLCyyWAw4AckotIhERiYsIjPbFVElVLfBzLsE5qtdLJxwREZGtik1S4Zd4q7r7lQmKR0RE4sDMoj1xwszS3D3XzLonMiAREYmPCOSoYiupiQTnn6aZ2SjgNWDtlo3u/kYpxyYiIru5WM5J7QksA3qx9ftSDihJiYiksKhfFqluOLPve7Ympy28VKMSERGh+CRVHqhC4eS0hZKUiEgK2x2uOLHQ3YcnLBIREYmrCOSoYr+QHIGXJyIiZVlxldThCYtCRETiK8m3fY+XHSYpd1+eyEBERCS+LAIDYlG4/qCIiKQAM+tjZj+a2Wwzu6aYdgPNzM2sc0l97vRV0EVEJPUFs/sSuL/gMnoPA72BLGCSmY1y9xlF2lUFLgW+iqVfVVIiIhFVzuL3iEFXYLa7z3H3jcAI4NjttLsFuBP4PabXEONrlRh8/8V4bji5F9ed2IN3n3tkm+3j33iBm04/ipvPPJo7h55Iztyf8rdl/TSTO847nmGn9uam049i04aYfn9l1vvj3mP/NvvRpmVz7r7rH9ts37BhA2ecdgptWjbnkG4HMn/ePAA++vADunXtROcO7ejWtRPjP/k4wZEnx/71q3L3gJbcc2wrjmlTd5vthzatxb9PbMvtfffj9r770bN5rfxtV/VqyuMnt+PKnk0SGXLSfPLh+xzatR3dO7Xmofvv3mb7l//7nD49D6LR3pV55+3CF845/cRjaN24HmcPOj5R4UZJBrCgwHJWuC5feNPcBu4+JtZONdwXJ5vz8njpn8P424MvULPuPtx2zgDaH9Kb9CYt8tsceNSx9DzhDACmffYBrz5wC5fd/xx5ubk8edPfGHLTvTRo0Zo1q1ZQPq1Csl5KqcvLy+Oyv17EmHc/ICMzkz8d1IX+/QfQqnXr/DbPPP0UNWvUZPoPs3n1lRFcf93VvPDSK9SuXYeRb40mPT2d6d9/zzH9jmLO/OwkvprSZwaDu2Zyx0c/s3zdJm45el+mZq0ie9WGQu2+nL+CZydteyzGzFhMxfLlOLxF7USFnDR5eXnccNWlvPTGGOqnZ9Lv8O4c2ac/+7Zsld8mI7MB9z78BI89dN82z7/gkr+xfv16XnjmyUSGXWosvl+UqmNmkwssP+7uj+9ELOWAe4HBO7NTVVJxMnfGNPbObMTeGQ1Jq1CRLr2PYdpn7xdqs1flrbfm2vD7uvyZNzMmfk5m85Y0aBF8SFepXpNy5csnLvgEmzRxIs2aNadJ06ZUrFiRk04ZxDuj3y7U5p3Rb3P6mWcDcMLAExn/8Ue4Ox06diQ9PR2A1m3a8Pv69WzYsGGbfURJs9qVWPTbBpas2UjeZufLeSvolFk95udP/3UNv+duLsUIU8e0KZNo3KQZjRoH761jTziJ998dXahNg4aNad2mHeXKbfvx96cevahcpUqiwi1rlrp75wKPogkqG2hQYDkzXLdFVaAtMN7M5gEHAaNKmjyhSipOVi5ZRK266fnLNevWZ+70adu0+2Tkc3zw8pPkbtrEFQ+9BMCiX+ZgZtx36ZmsWbGcLr2Poc+Z5ycs9kTLyckmM3PrezkjI5OJE7/atk2DoE1aWhrVqldn2bJl1KlTJ7/Nm2+8ToeOB7DHHnskJvAkqVWpAsvWbcpfXr5uE83qVNqmXZeGNWhZtwq/rt7A81OyWV7gObuLhQtzqJ+Rmb+8T3oGX0+ZlMSIkifREyeASUALM2tCkJwGAadt2ejuq4D8f8BmNh640t0nU4ykVVJmVsPMLkzW/pPlsBPP4vbXP2PgRdcw5pl/AcEQxU/fTOK8mx/gqsdH8vWn45g5aUKSI01tM6ZP54brruahRx5LdigpYWrWKi57cwbXjvmR7379jfO7NUx2SJJsFgwVx+tREnfPBS4GxgEzgVfdfbqZDTezAX/0ZSRzuK8GEJkkVWPveixfnJO/vGLxQmrsXW+H7bv0PoZpn34AQM26+7Bvx65UrVGLPfbci3bdDuOXH78v9ZiTJT09g6ysredXs7OzyMjI2LbNgqBNbm4uq1etonbt4JxKVlYWp5x0PE8+/RxNmzVLXOBJsnzdJmpX2nqOslalCqwoUiWt2ZhH7ubgus+fzF5Gk1rbVlq7g/r101mYnZW//GtONvXrpxfzDIkndx/r7vu6ezN3vy1cN8zdR22nbc+SqihIbpL6B9DMzKaZ2X+2ZFoze9PMng5/PtfMbgt/vtzMvg8flyUx7u1q3Ko9ixfMY0nOAnI3bWTSB6Npf0jvQm0W/TI3/+fvJnxM3QaNAWhzYA+yZ//Iht/Xk5eby6ypX1G/wISLqOncpQuzZ//EvLlz2bhxI6+9MoJ+/Qv/odWv/wBefP5ZAN54fSQ9DuuFmbFy5UpOGNCPW277B9267x43jZ6zbB37VN2DvStXpHw546DGNZmStbpQmxp7bR2575RZnZxV0Z4duiPtD+jM3Dmz+WV+8N56+43X6N2nf7LDSppy4S3k4/FIlmSek7oGaOvuHcxsEHAIMIpgymL9sM0hwAgz6wScAxxIMNT6lZl96u5fJyHu7SqflsZpVw7n/kvPwjfn0b3/yWQ03Ze3H7+XRi3b0eHQ3nwy8llmTJpA+bQ0KletzjnD7gGgcrXq9D71PG47ZwBmRruDD2P/7r2S/IpKT1paGvc98BDH9DuKvLw8zh58Lq3btGH4TcM4oFNn+h8zgMHnDuHcwWfSpmVzatasxfMvjgDg0Uce4uefZ3PHrcO549bgIv2j332funW3nZYdFZsdnpmUxdWHN6WcGZ/+vJzsVb8zcP99mLt8HVOzVnPUfntzQGY18hzWbsjl0S9+yX/+349sTnq1PdkzrRz/Or41j3+5gO8W/pbEV1R60tLSuOWu+zn9xGPYnJfHKaefzX6tWnP37TfTvmMnjjy6P9OmTua8M09h1aoVfPDeWO79xy18/EXwUXJC317M/mkWa9euoXObZvzzwUfpeXjvEvaampJwTqpUmHtybg1lZo2Bd9y9rZllAK8D5wJXATWB84FPgC7h+truPix87i3AEnd/cDv9DgWGAtTaJ6PTnW/p3E4szujUKNkhlClDRmw7KUa2765+rUpuJAD07dWNb76eEpfU0rBlO7/yyW1G2f6wSw9pOsXdS7yMUbylxOw+d882sxpAH+AzoBZwMrDG3X/bmbn+4bTIxwEat9pfN2cUkd1W1O8nVdp+I5g3v8WXwGUESepz4Mrw/4T/P87MKplZZeD4AttERCSiklZJufsyM5tgZt8D7xIknSPdfbaZzSeopj4P2041s2eAieHTn0yl81EiIqnHKBeBW3UkdbjP3U8rsuqpcP0moHKRtvcSXFJDRERKYGi4T0REpFSlxMQJERGJs6jfPl5ERMq2ZH4JN1403CciIilLlZSISARp4oSIiEgpUyUlIhJRUTgnpSQlIhJREchRGu4TEZHUpUpKRCSCjGhUIUpSIiJRZLAzd5BIVVFItCIiElGqpEREIqrs11GqpEREJIWpkhIRiSBD35MSEZEUVvZTlIb7REQkhamSEhGJqAiM9ilJiYhEk+l7UiIiIqVJlZSISATpskgiIpLSNNwnIiJSilRJiYhEVNmvo1RJiYhIClMlJSISRRG5VYeSlIhIBEVldl8UXoOIiESUKikRkYjScJ+IiKSssp+iNNwnIiJxYmZ9zOxHM5ttZtdsZ/v5ZvadmU0zs/+aWeuS+lSSEhGJKLP4PUrel5UHHgaOBloDp24nCb3k7u3cvQNwF3BvSf1Gerhvj/LlaFGjarLDKBNy8zYnO4QyZWiXhskOocx4YMK8ZIdQZixasyHZIeyKrsBsd58DYGYjgGOBGVsauPvqAu0rA15Sp5FOUiIiu6tgCnpCz0plAAsKLGcBBxZtZGYXAZcDFYFeJXWq4T4RkYiK83BfHTObXOAx9I/E5O4Pu3sz4GrghpLaq5ISEZFYLHX3zsVszwYaFFjODNftyAjg3yXtVJWUiEgkWVz/i8EkoIWZNTGzisAgYFShiMxaFFjsB/xUUqeqpEREIiqR3+V191wzuxgYB5QHnnb36WY2HJjs7qOAi83sCGATsAI4u6R+laRERCQu3H0sMLbIumEFfr50Z/tUkhIRiaAkzO4rFTonJSIiKUuVlIhIFMV4pYhUpyQlIhJRUUhSGu4TEZGUpUpKRCSiYvx+U0pTkhIRiSADypX9HKXhPhERSV2qpEREIkrDfSIikrI0u09ERKQUqZISEYmoKAz3qZISEZGUpUpKRCSCojIFXUlKRCSSYr5ZYUrTcJ+IiKQsVVIiIlGkq6CLiEgqi0CO0nCfiIikLlVSIiIRFMzuK/u1lCopERFJWaqkREQiquzXUUpSIiLRFYEspeE+ERFJWUpScfTV5x9xZp+unHZkZ158/P5ttr/6n0c4u9/BnDvgEC4ffBy/Zi/I37YoJ4srzx3IWX0P4ux+B7Mw65dEhp5wH7z/Hh3btaJ963255+47t9m+YcMGzj5jEO1b78thhxzM/HnzCm1f8Msv7FO7Gg/cd0+CIk4uvbdiN3fK5zx5fh+eGHokX732+A7b/ThhHHcf05Jff/qu0PrVi3O4/6QDmPjGU6UdaqmzOP6XLEpScZKXl8cDw6/izide5dl3/sfHY95g3uwfCrVp0aodj438iKdHfU6Powbw2D9vyt92+9UXMmjIxTw39kv+/eoH1KxdJ8GvIHHy8vK44tJLeOPtMUya9j0jXx3BDzNnFGrz3DNPU6NGTb6ZMYuLLrmUYTdcU2j7tVdfQe+j+iQy7KTReyt2m/Py+ODR4Zx40xOc+/A7zPxsDEt/mb1Nu43r1jB19PPU36/9Nts+eeofNOl0SCLCLXVm8Xski5JUnPzw7VQyGjYhvUFjKlSsSK++xzPho3cLtel40CHsuVclAFq378ySX3MAmDf7B/Lycunc/TAAKlWukt8uiiZPmkjTZs1o0rQpFStWZOBJp/DO6FGF2owZ/TannXEWAMedcCLjP/kYdwdg9Ki3aNS4Ca1atUl47Mmg91bsFv70LTXrN6TGPg0oX6EiLQ/ty+yvPtqm3X9ffJCuA88jrULFQut/+uJDqtfLpE7D5okKWUqgJBUnSxYtZO/6GfnLe++TzpJFC3fYfszIF+h66OEALJj3M1WqVufvl5zFecf35N933UheXl6px5wsC3OyychskL+ckZHBwpzsQm1ycnLIDNukpaVRvVp1li1bxpo1a7jvnru59vphCY05mfTeit2aZYuoWqd+/nLV2vuwZtmiQm0WzZ7O6iULadalZ6H1G9ev5avXn6DbqRclItSEsDg+kqVUkpSZ1TCzC8Ofe5rZO6Wxn7Lq/VGv8uP0aQwacgkAebl5fDflCy64ajiPvvYhCxfM4703X05ylKnp9ltv5uJLLqVKlSrJDiUl6b1VPN+8mU+e+geHDbl6m20TXnqIzscOpuJelZMQmexIaU1BrwFcCDwS6xPMrLy7l9k/8fauV58lC7dWA0t+zWHvevW3aTf5f+N54dF7eeD50VSsuEfw3H3q07xlO9IbNAbgT0f0ZcY3k4EzEhB54tVPzyA7a+uJ/ezsbOqnZxRqk56eTlbWAjIyM8nNzWXV6lXUrl2byRMn8vYbr/P3665h1aqVlCtXjj333JO/XBCdv36L0nsrdlVq1+O3pVurzN+W/UqV2vXylzeuX8vS+T8x4rpgKHntiqW8ceuFnHDDIyyc9S2z/jeOT5+5mw1rf8OsHGkV9+CA/mX4WEVgCnppJal/AM3MbBqwCVhrZiOBtsAU4Ax3dzObB7wC9AbuMrPlwM3AHsDPwDnuvsbMOgH3AlWApcBgd9/xeEcS7NeuI1nz57Awaz516tbn47FvcsM/C88s+mnGt9x74xXc9cSr1Ky9d/76lu0OYM1vq1i5fCk1atVh6pefs1/bDol+CQnTqXMXfp49m3lz55KekcHrr73C08++UKhN3/4DeOmF5zjwoIN5642R9Oh5GGbG+x9/mt/m9ltupnKVKpFOUKD31s6o36IdK3Lms/LXLKrWrssPn42l/5X/zN++R+WqXPzSl/nLI649k57nXsU+Ldpx2p0v5q+f8NK/qLBnpTKdoIJhurKfpUorSV0DtHX3DmbWE3gbaAPkABOA7sB/w7bL3P0AM6sDvAEc4e5rzexq4HIzuwP4F3Csuy8xs1OA24Bzt7djMxsKDAWol55ZSi9vW2lpaVz69zv5vyEnsXlzHkcPPI0mLVry9IN3sF/bDnTvdTT/vvtG1q9by42XBaHXq5/J7f9+kfLly3PBVTdz+eDjcXf2bdOe/iedlbDYEy0tLY1/3v8gxx1zNJvz8jjz7HNo1boNt958Ix07daJf/wGcNfhc/nzuWbRvvS81a9XiP8+9lOywk0bvrdiVK5/GEef/nZE3DmHz5s20O2IgdRq14L8vPMg+LdrS/MBeyQ5RdpJtmTEV107NGgPvuHvbMEld7+69w23/Bia4+wthJdXD3eebWX/gGSAr7KYi8AVwH/A/YE64vjyw0N2PLCmO/dp28Mdf/zheLyvSOjWukewQypQp81YmO4QyY8xPS5IdQpnx3N8G8utP38el/Gm9f0d/ftSnJTeMUecm1ae4e+e4dRijRF0WaUOBn/OK7Hdt+H8DPnD3Uws+0czaAdPd/eDSDVFEJFrK/mBf6U1B/w2oupPP+RLobmbNAcysspntC/wI7G1mB4frK5jZ7vEFGRGR3VypVFLuvszMJpjZ98B6YFEMz1liZoOBl81sj3D1De4+y8xOBB40s+phzPcD00sjdhGRyIhAKVVqw33uftoO1l9c4OfGRbZ9DHTZznOmAYfGOUQRkQhL7jX34kVXnBARkbgwsz5m9qOZzTaza7az/XIzm2Fm35rZR2bWqKQ+laRERCIqkReYNbPywMPA0UBr4FQza12k2ddAZ3ffHxgJ3FVSv0pSIiISD12B2e4+x903AiOAYws2cPdP3H1duPglUOKXWZWkREQiKJ4Xlw0LqTpmNrnAY2iRXWYACwosZ4XrdmQI8G4x2wHdPl5EJLriO29iaby+zGtmZwCdgR4ltVWSEhGReMgGGhRYzgzXFWJmRwDXE1xtaEPR7UUpSYmIRFSCp6BPAlqYWROC5DQIKPRVJDPrCDwG9HH3xbF0qiQlIhJRibztu7vnmtnFwDiCa6w+7e7TzWw4MNndRwF3E9zN4jULgvvF3QcU16+SlIiIxIW7jwXGFlk3rMDPR+xsn0pSIiIRVfavN6Ep6CIiksJUSYmIRFGBLziVZUpSIiIRpQvMioiIlCJVUiIiEWQkdgp6aVGSEhGJqAjkKA33iYhI6lIlJSISVREopVRJiYhIylIlJSISUVGYgq4kJSISUVGY3afhPhERSVmqpEREIioChZSSlIhIZEUgS2m4T0REUpYqKRGRCAougl72SyklKRGRKDLN7hMRESlVqqRERCIqAoWUKikREUldqqRERKIqAqWUkpSISCSZZvelulnTv1nas2Xt+cmOo4g6wNJkB1GG6HjFTscqdql6rBolO4BUE+kk5e57JzuGosxssrt3TnYcZYWOV+x0rGK3uxyrKExBj3SSEhHZXRmROCWl2X0iIpK6VEkl3uPJDqCM0fGKnY5V7HaPYxWBUkqVVIK5++7xjyNOdLxip2MVOx2rskOVlIhIRGkKuoiIpKwozO7TcJ+IiKQsVVJJZGbm7p7sOEQkmiJQSClJJdmewPotC0paO2ZmewOb3H1lsmNJRXrv/HE7OnZmVs7dNycjpriIyP2klKSSxMyGAL3M7Bdgoru/6e6uD5ttmdlfgaOAFWa2wN2vTXZMqWbLe8bMTgRaAjOBKe4+L5lxpbqC/97MbDCwF8Hn4r/dPTeZsUlA56SSwMxOBq4g+K7GBqCHmV0IWz9sJGBmg4BjgTOAtUC75EaUuszsL8CNwBrgbOA8M+uS3KhSW4EEdRlwFrAAOB8Ymsy44sfi+EgOJakEM7N9gIbA3e7+KXAf8DnQwcyqJTW41LQeuJrgA6QxcDyAmXVMYkwpx8zKA92Awe5+P3BVuKl78qIqG8xsT6CNu/cC2gBzgcfMrFJyIyt7zKyPmf1oZrPN7JrtbD/UzKaaWW5Y9ZdISSqBwr90hxJUBOeZWRN3XwWMBpoC+yQzvhRVFfgQ6O3uR7n7JjM7DxhiZnslObaUYGZdgUoEV/U+w8z2dPcfCI5bfzOrnNQAU4zZNmdqygF1zewNoCsw0N3zgEFm1jPR8cWLEZyTitejxP0Ffyg9DBwNtAZONbPWRZr9AgwGXor1deicVIKY2TEEQ1X3uPvccCLAtWb2MMHl+fcCNCkAMLNzCI7JR+7+gpntR/Bh2wo4EhgCnOru64vrZ3dgZrWAM4ExwFvAcQRV5+NANYKhPwkVOQd1KJADLCM4Xk8Ap7j7BjM7i2BIvm/Sgo2DBA/SdQVmu/scADMbQTBUP2NLgy3nSM0s5gkpSlKlzMzKARWB28NVd4T/H0nwC7wfWAdc5O6LEx9hajGzY4ELgK+Ac8JhvdsIzt1dAVQm+CCZmbwoU4OZtXL3mWb2DXCsu18Q/vFzkpmdQlCF/tnd1yY30tRR5BzU8cBnQGfgBuBm4EUzey9cd7K7p9r96FJZBsE5vS2ygAN3tVMlqdJXzd1Xmlk34G2CD9rL3X0GMCOspDapKshPUNcSfOAuNLMTgEOB84C73H2jmaVp1hWY2SHAU2Y2ErgJONnMhrn7cDN7i2CG32J3T8Ub+yVckQqqDdDP3XuY2UPAb8BUd59iZp8Am4H17p6dxJDjIs5T0OuY2eQCy48n4hqISlKlyMzOB3qb2Q/ARIK/3D42szvd/WoAd1+dzBhTRXiS+leCD9fBwB3u/kY4LNAfGGpmjwB5yYsyNZhZGvAd8APB0N5K4E3gdDN7z90nUmCIZXdXJEF1BGoA083sSoJzwQPDr3/0Bf4Xpe/ixfnafUtLuFFkNtCgwHJmuG6XaOJEKTGzMwimTV9PMHTQJ5wkcRQw0MxuSWZ8qSRM5o8QjP9fDpwdnpfC3d8iONfymrtv3h2n6JtZxS0noM3sSOCvQDPgzwTnUjYRnH7oRvDe0r/rAgokqBOAW4FvgLbAee7e193Xm9mfCd578sdNAlqYWRMzqwgMAkbtaqeqpEqBmXUgGDL4M3AwwQfIJeHsl/VAF4K/5nZ7ZjYQuJggoZ8LrCKY+XO+mdVy93vc/Z1kxpgCGgL3m9kioC7BbNCrgRXAF8Av7v6xmf0MzCvTV0koJeFEiCMIJi4tN7P7CRL6cwRJ6wzgzChVUUBCZ064e66ZXQyMA8oDT7v7dDMbDkx291Hh9/behP9v7/5jvarrOI4/X5QDB4qo1LByNYvMXBlRkSZDKiax1nA0m/RHC8MfKY6Wa2vL0uXWis3piBGRY05TI7XpchdKu+NitgiGBDQGG6051yoQK5U0e/XH53Pl7nrhXtj38v1+z309GON8z/d7zvnc7y7nfT7nvD+fN1OAz0q61fb7j7XfXHG1WP3P8AAwg/JQdrHtefU5ylcoV8GHbO9vYzM7yXuBdba3A1+nJJFMG4COyAAABjFJREFUAW6iZPSdMUTK8Jhiex+wg5Jos972Ssrv0duAbwL3S5pme4PtPW1sascYojc5AVgATK+vnwS+RRkT9SJwle2dJ6+FzWT7cdvTbZ9n+/a67hbbj9blLbbfbnui7bOGC1CQnlRLSbqJMtL/MNBLuZoYL+kMyvOo64EvjMVbVsewm5LF93hNJlldH17/EJhv+3B7m9cxVlOu+L8m6RXb91CC+HWUDKqMGRugvzdZkyT22V4j6Z/AdyXtst1HuSD6djvbOdqacHWXINUiki4DFgGzKD2myZQTy5eA+ygP/BfXE3Ec0Ut5ZrdYUi/lZDuJ0ttMgKpqb2qfpEOUE+0LlNt95wA32M54KF5PjLjU9l0qU419FXhO0mrbD0g6BVgj6Xrbv2lva0fXSAfhdroEqdb5I7CwpkmfC5xj+77aK9gB/Nz2q+1tYuep6fmrgCuAmymDT69O6vTQbD8m6VXg+8B/KRc+CVC8PpPE2cACSdMomXuXUC4e50k6zfY6SeOBFZI+kaEfnS9BqkUGnVR/C5wr6dOUbKLFCVBHZ/s5YKWkuwFl8Omx2e6RtLUu/73d7ekEkt4CTLH9K0lzgU8Cz9ZEiLWSDgOz65RRqyX9bCwEqJSPj6P5KyV5Yg/wRdt729yermD7pXa3oVskOL3BZMqFzrOUKbXuBJZLWmb7rjq91gTgA5Im1+Egzdf9MSpBapTspaRh3pwAFTH6bO+t00MtBb5Rb7UfBK6pg3nvtL1W0ukZQN9dEqRGQR2HcaXt/7S7LRFjyMAMyIO2H5T0N2CVpAO27x1rAaoBHakEqdGSABVxcg3KgLy9/jsBeAV4qq2Na5Nk90VEdJgBGZArKAN1l2TwfPdKkIqIxqkZkNvK4lhNMlGy+yIiOpVTn60REqQiIhqov3x8t8sEsxER0bESpCIiomPldl9EREPldl/ESSDpNUnbJe2UtL6Wmj/Rfa2TtKgur+2veHuUz86RdPEJHOPPks4e6fpBnzmuyWIlfaeWQY94A7XwT7skSEU3eNn2RbYvpAzMvHbgm5JO6I6A7auHKZ0yh1KSPSLaJEEquk0f8O7ay+mT9CiwW9KbJP1A0hZJOyRdA6V8g6SVkvZI+jWl/Dr1vV5JM+vy5ZK2SXpG0hOS3kkJhstrL+5SSVMlPVSPsUXSJXXbsyRtlLRL0lpGMBuNpF9I2lq3WTrovTvq+ickTa3rzpPUU7fpk3R+K77MaDAdqSnVir/tkmdS0TVqj2k+0FNXzQAutL2/nuhfsP2RWi/oKUkbgQ9RStRfALyVUgn47kH7nQr8GJhd93VmnX9xNfBv2yvq534K3GF7c60ZtgF4H6W662bbt0laACwZwY/z5XqMU4Etkh6yfQCYCPzB9nJJt9R93wCsAa6tE6l+DFgFzD2BrzGiqyRIRTc4VdL2utwH/IRyG+73A6a7mUcpw7Covp4MvAeYDdxv+zVKhdYnh9j/LGBT/75sHzxKOz4FXKAjl5WnS5pUj3FF3faXkp4fwc+0TNLCuvyO2tYDwP+AB+v6e4GH6zEuBtYPOPb4ERwjxjCRCWYjTpaXbV80cEU9WQ8sjijgRtsbBn3uMy1sxzhg1uCy9jrOeyGS5lAC3sdtvySplzIR6lBcj3to8HcQMawGRKk8k4qm2ABcJ+kUAEnTJU0ENgFX1mdW04DLhtj2d5Sqre+q255Z1/8LOG3A5zYCN/a/kNQfNDYBV9V184Epw7R1MvB8DVDnU3py/cZRyp1T97m5lpfYL+nz9RiS9MFhjhHRCAlS0RRrKc+btknaCfyIcqfgEUoRyt3APcDTgzesE5Aupdxae4Yjt9seAxb2J04Ay4CZNTFjN0eyDG+lBLldlNt+fxmmrT3AmyX9CfgeJUj2exH4aP0Z5gK31fWLgSW1fbuAz43gO4kxrgkp6LLdtoNHRMTomPHhme57ekvL9jdp/Littme2bIcjlJ5URER0rCROREQ0VAPyJtKTioiIzpWeVEREUzWgK5UgFRHRUE0oH5/bfRER0bHSk4qIaKCmlI/POKmIiAaS1AMcs37ZcfqH7ctbuL8RSZCKiIiOlWdSERHRsRKkIiKiYyVIRUREx0qQioiIjpUgFRERHev/C3Z123r7Bb4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "sUl65VGRs0FJ" }, "source": [ "# END" ] } ] }