Created
August 22, 2020 00:04
-
-
Save morganmcg1/b2a26e213482d3355a3d3a64c91e94ac to your computer and use it in GitHub Desktop.
Karpathy's minGPT in Fastai
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# A Quick Demo of Andrej Karpathy's minGPT Play Char Demo \n", | |
| "- You can find the Play Char demo in the minGPT repo here: https://github.com/karpathy/minGPT\n", | |
| "- Goal: Generate Shakespere\n", | |
| "\n", | |
| "This notebook is partially based on the fastai Transformers tutorial: http://docs.fast.ai/tutorial.transformers\n", | |
| "\n", | |
| "**Note**:\n", | |
| "- This needs the minGPT repo downloaded in the same folder as this notebook" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 1, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "from fastai.text.all import *\n", | |
| "from minGPT.mingpt.model import GPT, GPTConfig, GPT1Config" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Data\n", | |
| "\n", | |
| "You can download the raw text file at: https://github.com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou'" | |
| ] | |
| }, | |
| "execution_count": 2, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "raw_text = open('input.txt', 'r').read()\n", | |
| "raw_text[:100]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Data Loaders" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 3, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "class CharTransform(Transform):\n", | |
| " def __init__(self, data, block_size):\n", | |
| " chars = list(set(data))\n", | |
| " data_size, vocab_size = len(data), len(chars)\n", | |
| " print('data has %d characters, %d unique.' % (data_size, vocab_size))\n", | |
| " \n", | |
| " self.stoi = { ch:i for i,ch in enumerate(chars) }\n", | |
| " self.itos = { i:ch for i,ch in enumerate(chars) }\n", | |
| " self.block_size = block_size\n", | |
| " self.vocab_size = vocab_size\n", | |
| " self.data = data\n", | |
| " self.n_sequences = math.ceil(len(self.data) / (self.block_size + 1))\n", | |
| " \n", | |
| " def encodes(self, o):\n", | |
| " i = np.random.randint(0, len(self.data) - (self.block_size + 1))\n", | |
| " chunk = self.data[i:i+self.block_size+1]\n", | |
| " dix = [self.stoi[s] for s in chunk]\n", | |
| " return torch.tensor(dix)\n", | |
| " \n", | |
| " def decodes(self, o):\n", | |
| " t = ''.join([self.itos[s.item()] for s in o])\n", | |
| " return TitledStr(t)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "**Note**:\n", | |
| "- Note `block_size` in Karpathy's code is equivalent to `Sequence Length` in fastai\n", | |
| "- We do not specify a validation set here as Karpathy does not in their notebook. Therefore we set `split_idx=0` in `TfmdLists`" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 4, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "data has 1115394 characters, 65 unique.\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "sl = 128\n", | |
| "block_size = sl\n", | |
| "n_samples = math.ceil(len(raw_text) / (block_size + 1))\n", | |
| "\n", | |
| "tls = TfmdLists(list(range(n_samples)), tfms=[CharTransform(raw_text, 128)], split_idx=0, dl_type=LMDataLoader)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "We specify `dl_type=LMDataLoader` for when we will convert this `TfmdLists` to `DataLoaders`: we will use an `LMDataLoader` since we have a language modeling problem, not the usual fastai `TfmdDL`." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 5, | |
| "metadata": { | |
| "scrolled": true | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "e them?\n", | |
| "\n", | |
| "LADY GREY:\n", | |
| "What you command, that rests in me to do.\n", | |
| "\n", | |
| "KING EDWARD IV:\n", | |
| "But you will take exceptions to my boon.\n", | |
| "\n", | |
| "LADY GRE\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "show_at(tls.train, 0)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "The fastai library expects the data to be assembled in a `DataLoaders` object (something that has a training and validation dataloader). We can get one by using the `dataloaders` method. We just have to specify a batch size and a sequence length. " | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 6, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "bs = 256\n", | |
| "dls = tls.dataloaders(bs=bs, seq_len=sl)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 16, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "#o = dls.one_batch(); len(o), o[0].size(), o[1].size(), o" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 8, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>text</th>\n", | |
| " <th>text_</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>que time would lie unswept,\\nAnd mountainous error be too highly heapt\\nFor truth to o'er-peer. Rather than fool it so,\\nLet the hi</td>\n", | |
| " <td>ue time would lie unswept,\\nAnd mountainous error be too highly heapt\\nFor truth to o'er-peer. Rather than fool it so,\\nLet the hig</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>an action and capacity,\\nOf notill stand there,\\nRemembering how I love thy company.\\n\\nROMEO:\\nAnd I'll still stay, to have thee sti</td>\n", | |
| " <td>n action and capacity,\\nOf notill stand there,\\nRemembering how I love thy company.\\n\\nROMEO:\\nAnd I'll still stay, to have thee stil</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "dls.show_batch(max_n=2)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Callback to Grab First Output" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Here we need to write the event `after_pred` and replace `self.learn.pred` (which contains the predictions that will be passed to the loss function) by just its first element. This is because Karpathy's model actually also calculates the loss in the forward pass, and returns (logits, loss). We only need the logits.\n", | |
| "\n", | |
| "In callbacks, there is a shortcut that lets you access any of the underlying `Learner` attribute so we can write `self.pred[0]` instead of `self.learn.pred[0]`. That shorcut only works for read access, not write, so we have to write `self.learn.pred` on the right side (otherwise we would set a `pred` attribute in the `Callback`)." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 9, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "class DropOutput(Callback):\n", | |
| " def after_pred(self): self.learn.pred = self.pred[0]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## GPT Model" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "I had to use 6 layers instead of 8 as my 2080 GPU has 13GB of ram" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 10, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "mconf = GPTConfig(dls.char_transform.vocab_size, sl, n_layer=6, n_head=8, n_embd=512)\n", | |
| "model = GPT(mconf)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Learner" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 11, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), opt_func=partial(Adam, sqr_mom=0.95, wd=0.1), \n", | |
| " cbs=[DropOutput]) #.to_fp16()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 12, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "/home/morgan/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/fastprogress/fastprogress.py:74: UserWarning: Your generator is empty.\n", | |
| " warn(\"Your generator is empty.\")\n" | |
| ] | |
| }, | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "SuggestedLRs(lr_min=0.0019054606556892395, lr_steep=2.511886486900039e-05)" | |
| ] | |
| }, | |
| "execution_count": 12, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| }, | |
| { | |
| "data": { | |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3jV9fn/8ed9TnbIJGGEsMKSPRJAFC1aB3VTFyoIBXG0tl/LV6v91i5b21p/rdbaVhFU1GqlWi1arVorWlFGIgTZS0aY2SQhO/fvj3NiYwyQQM75nHE/rutcnPE55/MiV5I7n/cUVcUYY0z4cjkdwBhjjLOsEBhjTJizQmCMMWHOCoExxoQ5KwTGGBPmrBAYY0yYi3A6QEelpaVpv379nI5hjDFBJS8vr0hV09t6LegKQb9+/cjNzXU6hjHGBBUR2X2s16xpyBhjwpwVAmOMCXNWCIwxJsxZITDGmDBnhcAYY8KcFQJjjAlzQTd89GR9VlTFZ0WVdI2PJjU+irQu0URHuKhrbKK2vonahkZcLiEm0k1MhIsIt9VIY0x4CJtC8M/1B3ngn5vbfXyES0iIiSAxNpKEmAiSYiNJiYuia3wUqfHR9EiKpm/XePp1jad7YjQi4sP0xhjjO2FTCK4d35vTs1IpqaqjuLKOwspa6hqaiI50ER3hJirCRVOTUtvQSE19E9X1jVTWNHCkpp6KmgbKjtaxr7Sc4qo6KmoavvDZsZFueqXE0is59vN/+6fFk5XuKRQxkW6H/tfGGHNiYVMIUuOjSI2P6pTPqmto4tCRGnYVV7Gr+Ci7iqooKD3KvrJq1hWUUXq0/vNjRSAjKZbMlFgyU+LolRJL/7Q4hmckkZUWb01QxhjHhU0h6ExRES56p8bROzWOswZ9+fXK2gZ2FVWxo7CSnYVV7CquYl9pNcu3F3Gooobm3UFjIl0M7ZnIyF5JjM5MZnTvZLLS4nG5rJnJGOM/Emx7Fufk5GgwrzVU19DEZ0VVbNhfzvp9R1i/v5wN+8qpqmsEICEmgjG9kxnbJ4XsvimM6Z1MUmykw6mNMcFORPJUNafN16wQOK+xSdlRWMnavWWs3VvGJ7tL2Xqogib1NC0Nz0hkUlZXJg3oyoT+XekSbRdyxpiOsUIQhCprG8jfW8bqXSV8vKOYNXvKqGtsIsrtYvKgNKYO78F5w7p3Wr+HMSa0WSEIATX1jXyyu5R/bz7Mm+sPsq+sGrdLOGNAVy4bncHUET1IiLEmJGNCUWOT8smeUrLS4unaJfqkPsMKQYhRVTbsP8Kb6w/wWv4B9pQcJSrCxXlDu3FNTm/OHpRuHc7GhJDDFTVMuP9dfnbFCGae3vekPuN4hcAam4OQiDCiVxIjeiVx5wVDWLO3jKVr9/Na/n7e+PQgvVNjuX5CX67JyTzpvx6MMYGjuLIOgDQfNQVbIQhyIsK4PimM65PC/100lLc2HOS5Fbt54J+beeidrVyZncktZ2fRLy3e6ajGmJPUXAh81SdohSCEREW4uHR0BpeOzmDboQqe+mgXL+UV8OLqPXxtZE++OWUAwzOSnI5pjOmg4qpaAJ9d4du01hA1qHsCv5g2kg+/dw43nz2A97cUcvEjH3LHX9awt+So0/GMMR1Q1Nw01MU3VwQ+LwQi4haRNSLyehuvzReRjSKyTkTeFZGT6wUxx9QtMYZ7vnYay+85l29OGcCb6w/y1d+8zy/e2ER5df2JP8AY47jiyloiXEKij0YG+uOK4H+ATcd4bQ2Qo6qjgJeAX/shT1hKio3ke1NP4707p3Dp6Aye+M9Ovvqb93ktfz/BNnLMmHBTXFlHanyUz0YD+rQQiEgmcDGwsK3XVfU9VW1up1gBZPoyj4GM5Fh+c81oXrt9MhnJMXz7hTXMXZxLQak1FxkTqIqran06AtDXVwQPA98Dmtpx7FzgTd/GMc1G9ErilW+eyY8uGcaKncVc8NAHPLdit10dGBOAiirrfNY/AD4sBCJyCXBYVfPacewMIAd48Biv3ywiuSKSW1hY2MlJw5fbJcyZ3J935n+F7L4p3Pvqem5+No+SqjqnoxljWiiuqqWrD5eT8eUVwZnAZSKyC/gLcK6IPNf6IBE5D/gBcJmq1rb1Qaq6QFVzVDUnPT3dh5HDU6/kWBZ/YwI/vGQY728pZOrDH/DhtiKnYxljvEoq64KzaUhVv6+qmaraD5gO/FtVZ7Q8RkTGAo/jKQKHfZXFnJjLJcyd3J9XvnUGibGRzFi0kj+8t92aioxxWHVdI1V1jXQNxqahYxGR+0TkMu/DB4EuwF9FZK2ILPV3HvNFwzOSeO32yVw+JoMH39rC//xlLTX1jU7HMiZsfT6ZzIdNQ36ZWayqy4Bl3vs/avH8ef44v+mY2Cg3D187hsHdE3jwrS3sKq5iwcwceiTFOB3NmLDTvLxE1/ggbBoywU1E+NY5A1kwM5vthyu5/A8fsuVghdOxjAk7/11eIoSahkxwuWB4D16+7QwArn7sI3J3lTicyJjw8t/lJeyKwDhoaM9EXrr1DLp2iWbGopX8e/MhpyMZEzY+bxqyKwLjtN6pcfz11kkM6pbAvGfyeGVNgdORjAkLxZW1xEa6iYvyXZeuFQLTbmldonnh5tOZ2D+V+UvyeTnPioExvlZcVefTqwGwQmA6qEt0BItmjeeMAV2586V8uzIwxsc8hcC3Ow1aITAdFhvlZuGN45mU1ZX/XZLPq2v2OR3JmJBVXFnrsy0qm1khMCclNsrNolnjmdi/K/OXrOW1/P1ORzImJBVXWtOQCWCxUW4Wzc4hp28q85es5YOttiCgMZ1JVSmuqiXVh5PJwAqBOUVxUREsnJ3DwG4J3PJsHp/sKXU6kjEh40hNA/WN6tMlqMEKgekEiTGRLJ4znm6J0cx5ejXbDtkMZGM6Q3Gl72cVgxUC00m6JcTw7JyJRLpdzFy0iv1l1U5HMiboFVf5fp0hsEJgOlGfrnE8M2cCVbUNzHl6NRU19U5HMiao2RWBCUpDeybyhxvGse1wJbc/v4aGxvbsUmqMaYs/1hkCKwTGB84enM7PLh/B+1sL+fHSDba5jTEnqXmdoZQ4314R+GU/AhN+rp/Yh93FVTz+wU76p8Vz01lZTkcyJuiUVNWSFBtJVIRv/2a3QmB85u6pp7Gn5Cj3v7GJvl3jOX9Yd6cjGRNUivywzhBY05DxIZdL+O01YxjZK4n/+csaNuwvdzqSMUHFs7yEb/sHwAqB8THPukQ5JMVGMvfpXA4dqXE6kjFBo7iyjlQfrzMEVgiMH3RLjGHRrPEcqaln3jO5VNc1Oh3JmKDgjyWowQqB8ZNhGYk8Mn0sn+4rZ/6StTQ12UgiY46nobGJ0qO+X4IarBAYPzpvWHd+cNFQ3lx/kN++s9XpOMYEtNKj9aji83WGwEYNGT+bO7k/OworefS97QzoFs+0sZlORzImIBVXeWcVW2exCTUiwn2Xj2BSVlfufulTcneVOB3JmIDkj03rm1khMH4X6Xbxpxnj6JUSyy3P5rG35KjTkYwJOM0LzvmjacgKgXFEclwUi2bl0NCk3LQ4l8raBqcjGRNQPl9wzpqGTCjLSu/CH64fx/bCSr77oo0kMqal4so63C4hKTbS5+eyQmAcNXlQGj+8eCjvbDxkI4mMacGzRWUULpf4/Fw2asg4btYZ/dh8sIJH39vOkB4JXDo6w+lIxjiuqLKOrn6YVQx2RWACQPNIovH9Urjzr/l8WmBrEhlTXFnrlxFDYIXABIioCBd/mpFNWpdo5j2Ty2Fbk8iEucLKWp9vSNPMCoEJGGldonnixpzP1ySqqbc1iUx4amxSDpTV0Cs51i/ns0JgAsqwjEQeunYM+QXl3P3yOtvdzISlQ0dqaGhSMlPi/HI+nxcCEXGLyBoReb2N16JF5EUR2S4iK0Wkn6/zmMB34fAe3HXhEP6+dj9/XLbD6TjG+F3zJMvMlNC5IvgfYNMxXpsLlKrqQOAh4AE/5DFB4JtTBnDZ6AwefGsLb2846HQcY/yqoLQagN6pIXBFICKZwMXAwmMccjmw2Hv/JeCrIuL7QbMm4IkIv75qFKMzk7jjxbVsOnDE6UjG+E1zIchIjvHL+Xx9RfAw8D2g6Riv9wL2AqhqA1AOdG19kIjcLCK5IpJbWFjoq6wmwMREullwYw4JMRHctDj38yn3xoS6gtKjdE+MJjrC7Zfz+awQiMglwGFVzTveYW0896XeQVVdoKo5qpqTnp7eaRlN4OueGMMTN+ZQVFnLrc/lUddwrL8pjAkde0uP+q2jGHx7RXAmcJmI7AL+ApwrIs+1OqYA6A0gIhFAEmDrEpsvGJWZzP+7ejSrd5Xyw1fX20giE/IKSqv91lEMPiwEqvp9Vc1U1X7AdODfqjqj1WFLgVne+1d5j7GfcvMll47O4PZzBvJi7l7+vHKP03GM8ZmGxiYOlNfQO0SuCNokIveJyGXeh4uAriKyHZgP3OPvPCZ4fPf8wUwZks5PX9tA3u5Sp+MY4xMHj9TQ2KShcUXQkqouU9VLvPd/pKpLvfdrVPVqVR2oqhNUdac/8pjg5HYJv7t2LBnJsdz2XJ4tQ2FC0t4Sz4ihUOkjMKbTJcVF8vjMbCpqGvjmnz+xzmMTcgpK/TuZDKwQmCB0Wo9EHrx6FLm7S/nZ6xudjmNMpyoorUYEMvy0zhDYfgQmSF0yKoNPC8p5/IOdjOyVxDXjezsdyZhOUVBaTY/EGKIi/Pd3ul0RmKB114VDmDwwjXtfXc/avWVOxzGmUxSUHvVrsxBYITBBLMLt4vfXjaVbYjS3PptHYYXNPDbBzzOHwH8dxWCFwAS5lPgoHp+ZTVl1Hd+yzmMT5OobmzhQXk1vuyIwpmOGZyTxwJWjWLWrhJ+8tsFmHpugdbC8hib179BRsM5iEyIuH9OLzQcr+NOyHQzu1oXZZ/Z3OpIxHbbXgaGjYFcEJoTcdcEQzh/Wnfte38gHW22VWhN8ChyYTAZWCEwIcbmEh68dw+DuCXzr+U/YfrjS6UjGdEhB6VFcAj39tA9BMysEJqTER0ewcFYO0REublq82kYSmaBSUFpNz6RYIt3+/dVshcCEnMyUOB6fmcOhI7XMXLSS0qo6pyMZ0y4FpdX08nP/AFghMCEqu28KC2flsLOoillPreJITb3TkYw5ob0OTCYDKwQmhJ05MI3HZoxj04EjzHlqNUfrGpyOZMwx1TU0cfBIjd87isEKgQlx557Wnd9NH8sne0q55dk86httwpkJTAfKq1HF75PJwAqBCQMXjezJA1eO4j/birj3Fdvq0gSmglJnho6CTSgzYeLqnN7sKTnK7/+9nX5p8dw2ZYDTkYz5Aif2IWhmhcCEjfnnD2Z38VEe+Odm+qTGcfGonk5HMuZze0uqcbuEnkn+nUMAVghMGBERfn3VKPaXVfPdJWvpkRRDdt8Up2MZA8Cu4ioykmOI8PMcArA+AhNmYiLdLLgxh4ykGOYuXs3WQxVORzIGgK2HKhjSPcGRc1shMGEnNT6KZ+dOJMrt4sZFqz5vmzXGKXUNTewsrGKwFQJj/Kd3ahzPzJ3A0boGbly0iuJKW4rCOGdnUSUNTcqQHlYIjPGr03ok8uTs8ewvr2b2U6upsNnHxiFbDnqaKK0QGOOAnH6p/OmGbDYdOMItz+ZR29DodCQThrYcrCDCJWSldXHk/FYITNg757Ru/PqqUXy0o5jvvriWxiabcGb8a8vBCrLS44mKcOZXsg0fNQb4+rhMSqrq+Pk/NpEct577rxiBiDgdy4SJLYcqGNvHuaHMVgiM8brprCyKq+r407IdpHWJZv75g52OZMJAZW0DBaXVTB/f27EM7SoEIjIAKFDVWhGZAowCnlHVMl+GM8bfvnfhEEoq63jk3W0kx0YyZ7LtfWx8q3kuy5AeiY5laG+D1MtAo4gMBBYB/YHnfZbKGIeICPdPG8HU4T247/WNvJRX4HQkE+I+HzHk0BwCaH8haFLVBmAa8LCqfhewhVpMSIpwu/jddWOYPDCNu19ex1sbDjodyYSwLQcriItyO7LYXLP2FoJ6EbkOmAW87n0u0jeRjHFedISbx2dmMyoziW8/v4bl24ucjmRC1JaDFQzqnoDL5dzghPYWgm8Ak4D7VfUzEekPPOe7WMY4Lz46gqdmjycrPZ6bFueycmex05FMiFFVthyq4DQHm4WgnYVAVTeq6ndU9QURSQESVPVXx3uPiMSIyCoRyReRDSLy0zaO6SMi74nIGhFZJyIXneT/wxifSI7zrEuUkRzDN55eTe6uEqcjmRBSVFlHSVUdgx2aUdysXYVARJaJSKKIpAL5wFMi8tsTvK0WOFdVRwNjgKkicnqrY+4FlqjqWGA68MeOxTfG99ITonlh3un0SIxh1pOryNtd6nQkEyKaO4pPC4ZCACSp6hHg68BTqpoNnHe8N6hHpfdhpPfWesqmAs1jppKA/e3MY4xfdUuM4fl5p5OeEM3sJ1exdq+NnDanbot36KhTq442a28hiBCRnsA1/Lez+IRExC0ia4HDwDuqurLVIT8BZohIAfAG8O32frYx/tYjKYYXbj6dlPgoZi5ayacF5U5HMkFuy8EjdI2PIj0h2tEc7S0E9wFvATtUdbWIZAHbTvQmVW1U1TFAJjBBREa0OuQ64GlVzQQuAp4VkS9lEpGbRSRXRHILCwvbGdmYztczKZbn500kMSaSGYtWsmG/FQNz8rYcqnT8agDa31n8V1Udpaq3eR/vVNUr23sS7wzkZcDUVi/NBZZ4j/kYiAHS2nj/AlXNUdWc9PT09p7WGJ/ITInjLzefTnyUmxkLV37ezmtMRzQ1KdsOVTi29HRL7e0szhSRV0TksIgcEpGXRSTzBO9JF5Fk7/1YPH0Km1sdtgf4qveYoXgKgf3JbwJe79Q4np93OlERLq5/YoVteWk6rKC0mqN1jY53FEP7m4aeApYCGUAv4DXvc8fTE3hPRNYBq/H0EbwuIveJyGXeY/4XmCci+cALwGxVtTWATVDolxbPC/NOx+0Spi9Ywcb9R5yOZILI5oOe7xenh45C+wtBuqo+paoN3tvTwHHbaFR1naqO9TYpjVDV+7zP/0hVl3rvb1TVM1V1tKqOUdW3T+l/Y4yfZaV34cVbJhEd4eK6J1ZYB7Jpt7V7y4hwSVBdERSJyAzvKCC3iMwAbJqlMUD/tHiW3DKJLtERXL9wBZ/ssXkG5sRyd5cyPCORuCjndwNobyGYg2fo6EHgAHAVnmUnjDF4+gyW3DqJ1PgoZi5caWsTmeOqa2gif28Z4/o6txlNS+0dNbRHVS9T1XRV7aaqV+CZXGaM8eqVHMuSWyaRmRLH7KdWsTTf5keatm3YX05tQxM5fVOdjgKc2p7F8zsthTEhontiDEtuncTYPil854U1PPnhZ05HMgGoeZmSnH5BdEVwDLahqzFtSIqN5Jk5Ez7f3OZXb27GBsOZlnJ3lZKZEkv3xBinowCnVgjsO9uYY4iJdPOHG8Zxw8Q+PPb+Dv7vlfU0NtmPjPEsPZ27u5ScAOkfgBPsWSwiFbT9C18A57bTMSYIuF3Cz68YQUpcFI++t50jNfU8dM0YoiJO5e8vE+z2llRTVFlLdr/A6B+AExQCVXV+gKsxQUxEuPPCISTFRnL/G5uorGngsRnZxEa5nY5mHJK727OnRSBdEdifJsb4wbyzs3jgypH8Z1shMxetpLy63ulIxiG5u0tJiI4IiMXmmlkhMMZPrh3fh0evH0d+QRnTF6ygsKLW6UjGAXm7ShnbNwW3g3sUt2aFwBg/umhkTxbOGs+uoiquefxjCkqPOh3J+FF5dT1bD1cEVLMQWCEwxu++Mjid526aQHFlLVc/9jHbD9vKpeHikz2lqAZW/wBYITDGEdl9U3nxlknUNypXPfYxa2x9orCQt6sUt0sY0yfZ6ShfYIXAGIcM7ZnIy7dNIjEmkuufWMn7W20rjlCXu7uEYT0DY6G5lqwQGOOgvl3jeem2SfRLi+emxav5+9p9TkcyPlLf2MTavWVkB1izEFghMMZx3RJiePGW0xnXJ4U7XlzLC6v2OB3J+MCn+8qpqW9ifABNJGtmhcCYAJAYE8niOROYMjid7//tUxZ/tMvpSKaTLd/mWZp80oCuDif5MisExgSImEg3j8/M4cLh3fnx0g0s+GCH05FMJ1q+o4hhPRNJjY9yOsqXWCEwJoBERbh49PpxXDKqJ794YzO/f3eb05FMJ6iua+ST3WWcOTDwrgbgBGsNGWP8L9Lt4nfTxxIV4eI372yloUm547xBiATOTFTTMat3lVDX2MSZA9OcjtImKwTGBCC3S3jwqtG4Rfjdu9toUmX++YOtGASp5TuKiHQLE/oHXkcxWCEwJmC5XcIDV47C7RJ+/+/tNDYpd104xIpBEPpoezFje6cE3PyBZoGZyhgDgMsl/GLaSFwu4Y/LdtCoyj1TT7NiEETKjtaxfn85d3x1sNNRjskKgTEBzuUSfn75CNwiPP7+TlTh+1+zYhAsPt5RjCoB21EMVgiMCQoul3Df5cNxCSz4YCeNTcq9Fw+1YhAElu8oIj7KzejegbW+UEtWCIwJEiLCTy4bjsslLPrwM5pU+dElw6wYBLjl24uZmNWVSHfgjta3QmBMEBERfnTJMFziKQYVNQ38YtpI2wc5QO0vq+azoipumNjH6SjHZYXAmCAjItx78VASYiJ4+F/b2FdazWMzskmKi3Q6mmll+XbPshKTBwXm/IFm9meEMUFIRLjjvME8dO1ocneXMO1Py9ldXOV0LNPK8u1FpHWJYkgA7U/cFisExgSxaWMzeW7uREqq6pj2x4/YdOCI05GMl6ry8c5iJg1IC/h+HCsExgS5iVld+dttZxDldjFz0Uq2H650OpIB9pZUc+hIbcDOJm7JCoExISArvQt/njcRgBkLV7K35KjDiczqXSUAjO8XeBvRtGaFwJgQMSC9C8/OnUhNQyPXPbGCA+XVTkcKa7m7S0iIiWBwt8DuHwAfFgIRiRGRVSKSLyIbROSnxzjuGhHZ6D3meV/lMSYcDO2ZyDNzJlB+tJ4bnlhJUWWt05HC1updpeT0TcHlCuz+AfDtFUEtcK6qjgbGAFNF5PSWB4jIIOD7wJmqOhy4w4d5jAkLozKTefIb49lfXs2sJ1dxpKbe6Uhhp6Sqju2HKxkfBP0D4MNCoB7NvVaR3pu2Omwe8AdVLfW+57Cv8hgTTsb3S+WxGdlsPVTB3KdXU13X6HSksJK3uxQgIPcnbotP+whExC0ia4HDwDuqurLVIYOBwSKyXERWiMjUY3zOzSKSKyK5hYWFvoxsTMiYMqQbD107htzdpdz6XB51DU1ORwobubtKiHK7GNkryeko7eLTQqCqjao6BsgEJojIiFaHRACDgCnAdcBCEfnSykyqukBVc1Q1Jz093ZeRjQkpl4zK4JfTRvL+1kLu/Gs+qq0vyo0vrN5VwqjMJGIi3U5HaRe/jBpS1TJgGdD6L/4C4O+qWq+qnwFb8BQGY0wnmT6hD3ddOISl+ft56F+2B7Kv1dQ38um+cnKCpFkIfDtqKL35r3sRiQXOAza3OuxV4BzvMWl4mop2+iqTMeHqm1MGcE1OJo+8u42/fVLgdJyQlr+3jPpGDYr5A818uehcT2CxiLjxFJwlqvq6iNwH5KrqUuAt4AIR2Qg0AneparEPMxkTlkSEn18xkr0l1dz98jp6JccyMStwN0oJZrnejuLsvsFTCCTY2gxzcnI0NzfX6RjGBKXyo/V8/U/LKa6q42+3nUFWehenI4Wc2U+tYn9ZNW9/9ytOR/kCEclT1Zy2XrOZxcaEkaS4SJ6aPQG3CDc+uYpDR2qcjhRSGpuUvN2lQdU/AFYIjAk7fbrG8dQ3xlNaVcesJ1dRXm0TzjrL1kMVVNQ0BFX/AFghMCYsjcpM5vGZOeworGTe4lxq6m3CWWfI9S40l9PXrgiMMUFg8qA0fnvNGFbvLuH259fQ0GgTzk7Vis9K6JkUQ2ZKrNNROsQKgTFh7NLRGfzk0uH8a9Mhfv6PTU7HCWoNjU18uK2IyQMDfyOa1mzPYmPC3Kwz+rG35CgLP/yMrPR4bpzUz+lIQSm/oIzy6nqmDOnmdJQOs0JgjOH7Fw1lV3EVP1m6gT6pcUH5y8xp728pxCUweWBgb1TfFmsaMsbgdgm/mz6WIT0Suf35NWw5WOF0pKCzbGsh4/qkkBQX6XSUDrNCYIwBID46gkWzcoiLcjPn6dUcrrA5Bu1VVFnLuoJyvjI4OBfFtEJgjPlcRnIsC2flUFJVx5ynV1NV2+B0pKDwn22e5fGDtUnNCoEx5gtGZSbz6PVj2bj/CLc//4kNK22HZVsKSesSxfCMRKejnBQrBMaYL/nq0O787IoRvLelkB/+fb3tY3AcjU3KB1sLOXtQelDsT9wWGzVkjGnTDRP7sq+0mj8u20FmShzfOmeg05EC0qf7yik9Ws9XhgRn/wBYITDGHMddFw5hX1k1D761hay0eL42sqfTkQLOsi2HEYGzBgVvIbCmIWPMMYkID1w5irF9kpm/JJ/1+8qdjhRw3t9ayOjMZFLjo5yOctKsEBhjjism0s3jM7NJiYtk3jO5HLalqz9XWlXH2r1lTAniZiGwQmCMaYduCTE8MSuHsqP13Pxsnq1W6vXBtkJUCdr5A82sEBhj2mV4RhIPXTuatXvL+N5L62wkEfDOxkOkdYliVGay01FOiRUCY0y7TR3Rk7suHMLS/P088u52p+M4qrahkWVbCjlvaHfcQTpstJmNGjLGdMg3pwxgx+FKHvrXVgZ0i+eSURlOR3LEip0lVNY2cMHw7k5HOWV2RWCM6RAR4ZdXjiSnbwr/uySftXvLnI7kiLc3HCQuys0ZA4JvtdHWrBAYYzosOsIzkig9IZp5z+Syr6za6Uh+1dSk/GvTIb4yOJ2YSLfTcU6ZFQJjzEnp2iWaJ2ePp6a+kdlPrqL8aL3Tkfxm3b5yDh2pDYlmIbBCYIw5BYO7J/D4zGx2Fx9l3rO51DaEx7DStzccxO0SzgnS1UZbs0JgjDklZwxI48GrR7HqsxLmL8mnqSn0h5W+s/EQE/unkhwXvLOJW7JRQ8aYU3b5mF4cLK/hl29upmdiDPdeMtmXP2MAAAzySURBVMzpSD6zs7CSbYcruX5iH6ejdBorBMaYTnHz2VkcKK9h4YefkZEcy5zJ/Z2O5BPvbDwEwPnDQqN/AKwQGGM6iYjww0uGcbC8hp/9YyM9kmK4KARXK31n4yGGZySSmRLndJROY30ExphO43YJD08fQ3afFO54cS2rPitxOlKn2ldWTd6e0pC6GgArBMaYThYT6eaJG3PITIll3jO5bD9c4XSkTvOrNzcT6XZx5bhMp6N0KisExphOlxIfxeJvTCDS7WLWk6s5WB78S1d/vKOY1/L3c9tXBtA7NXSahcAKgTHGR3qnxvH0N8ZTdrSO2U+t4khN8E44a2hs4idLN5CZEsttUwY4HafT+awQiEiMiKwSkXwR2SAiPz3OsVeJiIpIjq/yGGP8b0SvJB6bmc32w5Xc8kxe0E44e3bFbrYcquCHlwwLiSUlWvPlFUEtcK6qjgbGAFNF5PTWB4lIAvAdYKUPsxhjHHLWoHQevHoUH+8s5s6/rgu6CWdFlbX89p2tnDUojQtCrJO4mc8KgXpUeh9Gem9tfQf8DPg1EPyNiMaYNk0bm8k9XzuN1/L3c9/rG4NqU5tf/3MzNfWN/OSy4YgE974Dx+LTPgIRcYvIWuAw8I6qrmz1+ligt6q+7sscxhjn3XJ2FnMn9+fpj3bx0L+2OR2nXfL3lrEkt4A5Z/ZnQHoXp+P4jE8nlKlqIzBGRJKBV0RkhKquBxARF/AQMPtEnyMiNwM3A/TpEzrTuo0JJyLCvRcPpaKmnkfe3UZiTAQ3nZXldKxjUlV++toG0rpEc/u5A52O41N+GTWkqmXAMmBqi6cTgBHAMhHZBZwOLG2rw1hVF6hqjqrmpKcH9ybRxoQzEeGXXx/FRSN78PN/bOLF1XucjnRMf1+7n0/2lPG9qUNIiIl0Oo5P+XLUULr3SgARiQXOAzY3v66q5aqapqr9VLUfsAK4TFVzfZXJGOM8t0t4+NqxfGVwOvf87VOWrN7rdKQvqapt4JdvbmJUZhJXhdjksbb48oqgJ/CeiKwDVuPpI3hdRO4Tkct8eF5jTICLinDx2IxsJg9M43svr2PBBzucjvQFj72/g0NHavnxpcNxBfnG9O3hsz4CVV0HjG3j+R8d4/gpvspijAk8sVFuFs7KYf6L+fzijc2UVNVz99Qhjo/M2VtylMc/2MkVYzLI7pviaBZ/sdVHjTGOiY5w88h1Y0mKi+Sx93dQdrSO+6eNxO3QX+ENjU383yuf4hbh7q+d5kgGJ1ghMMY4yu0S7r9iBKlxUTz63nbKq+t5ePoYoiP8P4P35//YxH+2FfGrr4+kZ1Ks38/vFFtryBjjOBHhzguHcO/FQ3lz/UHmPL2aytoGv2Z4bsVunv5oF3Mn92f6hPAapm6FwBgTMG46K4vfXD2aFTtLuOGJFZRU1fnlvMu3F/HjpRs4Z0g6/3fRUL+cM5BYITDGBJQrszN5bEY2mw5WMO2Py9l04IhPz7dx/xFuey6PAenxPHLdWMf6J5xkhcAYE3DOH9adF+ZNpLqukWl/XM7fPino9HOoKn9euZtpf1xOTKSbRbPGh/zEsWOxQmCMCUjZfVN5/TuTGZ2ZzPwl+dz76qfU1HfOMtZHauq5/YU1/OCV9Uzon8o/vnNWyG020xE2asgYE7C6JcTw55sm8uu3trDgg538Y90Bpo3N5JrxmZzWI7HDn1daVcera/ex6MPPOFBew91TT+OWs7PCYtLY8UgwLQcLkJOTo7m5tgqFMeHm4x3FPLdiN29vPEh9ozI6M4kLhvfgnCHdGNozoc2JaKrK/vIa1u8rZ2n+ft7ZcIi6xiZGZSbx40uHkd031YH/iTNEJE9V29z8ywqBMSaolFTV8eqafbyyZh+f7isHoEdiDNn9UnCL0KRKkyqFFbVsPlhBRY1nGGpyXCTTxvbimpzeDO3Z8auJYGeFwBgTkg5X1PD+lkKWbSlk/f5yBHC5BLcIKXFRDOmRwGk9EzitRwIjeiU5MkktUByvEFgfgTEmaHVLiOHqnN5cndPb6ShBzUYNGWNMmLNCYIwxYc4KgTHGhDkrBMYYE+asEBhjTJizQmCMMWHOCoExxoQ5KwTGGBPmgm5msYgUAmVAeYunk1o8but+879pQNFJnLblZ3b0mNbPH++xZW9frvYcE6zZ2/OcZe9YrvYcEw7Zk1U1vc1PVdWguwELjvW4rfst/s3tjPN15JjjZbXslv1knrPslv1Usrd1C9amodeO87it+62PP9XzdeSY42Vt/diyH/98HTkmWLO35znLfmyWve3njvvZQdc0dCpEJFePsehSoLPszrDszrDs/hWsVwQna4HTAU6BZXeGZXeGZfejsLoiMMYY82XhdkVgjDGmFSsExhgT5qwQGGNMmLNC4CUiZ4nIYyKyUEQ+cjpPR4iIS0TuF5Hfi8gsp/N0hIhMEZH/eL/2U5zO01EiEi8ieSJyidNZOkJEhnq/5i+JyG1O5+kIEblCRJ4Qkb+LyAVO5+kIEckSkUUi8pLTWVoKiUIgIk+KyGERWd/q+akiskVEtovIPcf7DFX9j6reCrwOLPZl3pY6IztwOdALqAcKfJW1tU7KrkAlEEPwZQe4G1jim5Rt66Tv903e7/drAL8Ndeyk7K+q6jxgNnCtD+N+QSdl36mqc32b9CSczAy4QLsBZwPjgPUtnnMDO4AsIArIB4YBI/H8sm9569bifUuAxGDKDtwD3OJ970tBlt3lfV934M9Blv08YDqeX0iXBFN273suAz4Crg+27N73/QYYF6TZ/fZz2p5bSGxer6ofiEi/Vk9PALar6k4AEfkLcLmq/hJo8zJeRPoA5ap6xIdxv6AzsotIAVDnfdjou7Rf1Flfd69SINoXOdvSSV/3c4B4PD/41SLyhqo2+TQ4nfd1V9WlwFIR+QfwvO8Sf+GcnfF1F+BXwJuq+olvE/9XJ3+/B5SQKATH0AvY2+JxATDxBO+ZCzzls0Tt19HsfwN+LyJnAR/4Mlg7dCi7iHwduBBIBh71bbQT6lB2Vf0BgIjMBor8UQSOo6Nf9ynA1/EU3zd8muzEOvr9/m08V2NJIjJQVR/zZbgT6OjXvStwPzBWRL7vLRiOC+VCIG08d9zZc6r6Yx9l6agOZVfVo3iKWCDoaPa/4SlkgaDD3zMAqvp050fpsI5+3ZcBy3wVpoM6mv0R4BHfxemQjmYvBm71XZyTExKdxcdQAPRu8TgT2O9Qlo6y7M6w7M6w7A4L5UKwGhgkIv1FJApPp95ShzO1l2V3hmV3hmV3mtO91Z3Um/8CcID/Dp+c633+ImArnl79Hzid07IHzs2yW/Zwyn6imy06Z4wxYS6Um4aMMca0gxUCY4wJc1YIjDEmzFkhMMaYMGeFwBhjwpwVAmOMCXNWCEzQE5FKP59voYgM66TPahSRtSKyXkReE5HkExyfLCLf7IxzG9PM5hGYoCcilarapRM/L0JVGzrr805wrs+zi8hiYKuq3n+c4/sBr6vqCH/kM+HBrghMSBKRdBF5WURWe29nep+fICIficga779DvM/PFpG/ishrwNvi2TltmXh28NosIn/2Ln+M9/kc7/1K8ewOly8iK0Sku/f5Ad7Hq0XkvnZetXyMZzVLRKSLiLwrIp+IyKcicrn3mF8BA7xXEQ96j73Le551IvLTTvwymjBhhcCEqt8BD6nqeOBKYKH3+c3A2ao6FvgR8IsW75kEzFLVc72PxwJ34NlvIAs4s43zxAMrVHU0niXA57U4/++85z/hImQi4ga+yn/XqakBpqnqOOAc4DfeQnQPsENVx6jqXeLZqnEQnnXxxwDZInL2ic5nTEuhvAy1CW/nAcO8f8QDJIpIApAELBaRQXiWC45s8Z53VLWkxeNVqloAICJrgX7Ah63OU4dn9ymAPOB87/1JwBXe+88D/+8YOWNbfHYe8I73eQF+4f2l3oTnSqF7G++/wHtb433cBU9hcHpfChNErBCYUOUCJqlqdcsnReT3wHuqOs3b3r6sxctVrT6jtsX9Rtr+eanX/3a0HeuY46lW1TEikoSnoHwLz1r7NwDpQLaq1ovILjz7OrcmwC9V9fEOnteYz1nTkAlVbwO3Nz8QkTHeu0nAPu/92T48/wo8TVLgWZr4uFS1HPgOcKeIROLJedhbBM4B+noPrQASWrz1LWCOiDR3OPcSkW6d9H8wYcIKgQkFcSJS0OI2H88v1RxvB+pG/rsr1K+BX4rIcjwbj/vKHcB8EVkF9ATKT/QGVV2DZ/Pz6cCf8eTPxXN1sNl7TDGw3Dvc9EFVfRtP09PHIvIp8BJfLBTGnJANHzXGB0QkDk+zj4rIdOA6Vb38RO8zxgnWR2CMb2QDj3pH+pQBcxzOY8wx2RWBMcaEOesjMMaYMGeFwBhjwpwVAmOMCXNWCIwxJsxZITDGmDBnhcAYY8Lc/wfyVX/H3jDnSwAAAABJRU5ErkJggg==\n", | |
| "text/plain": [ | |
| "<Figure size 432x288 with 1 Axes>" | |
| ] | |
| }, | |
| "metadata": { | |
| "needs_background": "light" | |
| }, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "learn.lr_find()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 13, | |
| "metadata": { | |
| "scrolled": true | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: left;\">\n", | |
| " <th>epoch</th>\n", | |
| " <th>train_loss</th>\n", | |
| " <th>valid_loss</th>\n", | |
| " <th>time</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>0</td>\n", | |
| " <td>3.338925</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:18</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>1</td>\n", | |
| " <td>3.062790</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:18</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>2</td>\n", | |
| " <td>2.832788</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>3</td>\n", | |
| " <td>2.687037</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>4</td>\n", | |
| " <td>2.595154</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>5</td>\n", | |
| " <td>2.532446</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>6</td>\n", | |
| " <td>2.486231</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>7</td>\n", | |
| " <td>2.451136</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>8</td>\n", | |
| " <td>2.417365</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>9</td>\n", | |
| " <td>2.380837</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>10</td>\n", | |
| " <td>2.334122</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11</td>\n", | |
| " <td>2.279138</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>12</td>\n", | |
| " <td>2.206434</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>13</td>\n", | |
| " <td>2.130157</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>14</td>\n", | |
| " <td>2.056702</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>15</td>\n", | |
| " <td>1.991350</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>16</td>\n", | |
| " <td>1.929794</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>17</td>\n", | |
| " <td>1.873611</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>18</td>\n", | |
| " <td>1.829401</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>19</td>\n", | |
| " <td>1.782693</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>20</td>\n", | |
| " <td>1.759206</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21</td>\n", | |
| " <td>1.719195</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>22</td>\n", | |
| " <td>1.684518</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>23</td>\n", | |
| " <td>1.652220</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>24</td>\n", | |
| " <td>1.630679</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>25</td>\n", | |
| " <td>1.604731</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>26</td>\n", | |
| " <td>1.581347</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>27</td>\n", | |
| " <td>1.560486</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>28</td>\n", | |
| " <td>1.543253</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>29</td>\n", | |
| " <td>1.525474</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>30</td>\n", | |
| " <td>1.508890</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31</td>\n", | |
| " <td>1.493199</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>32</td>\n", | |
| " <td>1.478913</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>33</td>\n", | |
| " <td>1.466304</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>34</td>\n", | |
| " <td>1.452765</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>35</td>\n", | |
| " <td>1.441874</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>36</td>\n", | |
| " <td>1.430259</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>37</td>\n", | |
| " <td>1.420458</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>38</td>\n", | |
| " <td>1.408505</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>39</td>\n", | |
| " <td>1.398455</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>40</td>\n", | |
| " <td>1.389998</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41</td>\n", | |
| " <td>1.382348</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>42</td>\n", | |
| " <td>1.371678</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>43</td>\n", | |
| " <td>1.360793</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>44</td>\n", | |
| " <td>1.351078</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>45</td>\n", | |
| " <td>1.342533</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>46</td>\n", | |
| " <td>1.337409</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>47</td>\n", | |
| " <td>1.329052</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>48</td>\n", | |
| " <td>1.319077</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>49</td>\n", | |
| " <td>1.310749</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>50</td>\n", | |
| " <td>1.302483</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51</td>\n", | |
| " <td>1.294455</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>52</td>\n", | |
| " <td>1.287185</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>53</td>\n", | |
| " <td>1.277416</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>54</td>\n", | |
| " <td>1.269452</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>55</td>\n", | |
| " <td>1.259615</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>56</td>\n", | |
| " <td>1.252440</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>57</td>\n", | |
| " <td>1.243651</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>58</td>\n", | |
| " <td>1.234919</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>59</td>\n", | |
| " <td>1.227910</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>60</td>\n", | |
| " <td>1.217211</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61</td>\n", | |
| " <td>1.209070</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>62</td>\n", | |
| " <td>1.199655</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>63</td>\n", | |
| " <td>1.191217</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>64</td>\n", | |
| " <td>1.185001</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>65</td>\n", | |
| " <td>1.175376</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>66</td>\n", | |
| " <td>1.167852</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>67</td>\n", | |
| " <td>1.159078</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>68</td>\n", | |
| " <td>1.151620</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>69</td>\n", | |
| " <td>1.142666</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>70</td>\n", | |
| " <td>1.133764</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71</td>\n", | |
| " <td>1.127385</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>72</td>\n", | |
| " <td>1.119594</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>73</td>\n", | |
| " <td>1.112152</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>74</td>\n", | |
| " <td>1.104161</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>75</td>\n", | |
| " <td>1.097133</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>76</td>\n", | |
| " <td>1.089942</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>77</td>\n", | |
| " <td>1.083349</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>78</td>\n", | |
| " <td>1.074918</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>79</td>\n", | |
| " <td>1.068063</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>80</td>\n", | |
| " <td>1.061971</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81</td>\n", | |
| " <td>1.056730</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>82</td>\n", | |
| " <td>1.051151</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>83</td>\n", | |
| " <td>1.044623</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>84</td>\n", | |
| " <td>1.039983</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>85</td>\n", | |
| " <td>1.036456</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>86</td>\n", | |
| " <td>1.030712</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>87</td>\n", | |
| " <td>1.023932</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>88</td>\n", | |
| " <td>1.019215</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>89</td>\n", | |
| " <td>1.013263</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>90</td>\n", | |
| " <td>1.009832</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91</td>\n", | |
| " <td>1.005648</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>92</td>\n", | |
| " <td>1.004165</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>93</td>\n", | |
| " <td>1.001175</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>94</td>\n", | |
| " <td>0.997694</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>95</td>\n", | |
| " <td>0.995172</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>96</td>\n", | |
| " <td>0.990947</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>97</td>\n", | |
| " <td>0.989438</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>98</td>\n", | |
| " <td>0.986149</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>99</td>\n", | |
| " <td>0.982865</td>\n", | |
| " <td>None</td>\n", | |
| " <td>00:19</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "/home/morgan/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/fastprogress/fastprogress.py:74: UserWarning: Your generator is empty.\n", | |
| " warn(\"Your generator is empty.\")\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "learn.fit_one_cycle(100, 6e-4, div_final=10) \n", | |
| "\n", | |
| "# div_final=10 will ensure we finish at the same lr as Karpathy" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 14, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXyU5b338c8vyWTfVwIBEvY1soTFHZW6gEurtKWtrbU9tXZ5qp76nKpd7XKs5zztOdVj9ahd9OhxQ611q7UKahXEgBDCJoQ1LEkIkAWy53r+mIHGkECATO7M5Pt+vfJycs+dmW/u1/D1yjX3fY055xARkdAX4XUAERHpHSp0EZEwoUIXEQkTKnQRkTChQhcRCRNRXj1xZmamy8/P9+rpRURC0ooVK/Y557K6us+zQs/Pz6e4uNirpxcRCUlmtr27+zTlIiISJlToIiJhQoUuIhImPJtDFxE5WS0tLZSXl9PY2Oh1lKCLjY0lLy8Pn8/X459RoYtIyCgvLycpKYn8/HzMzOs4QeOco7q6mvLycgoKCnr8c5pyEZGQ0djYSEZGRliXOYCZkZGRcdJ/iajQRSSkhHuZH3Eqv2fIFfrGvXX86q8bqa5v8jqKiEi/EnKFvqWqnnvf3EyVCl1E+tjBgwf57W9/e9I/N2/ePA4ePBiERB8XcoUe4/NHbmxp9ziJiAw03RV6W1vbcX/ulVdeITU1NVixjgq5s1xioyIBaGw5/gEUEeltt912G2VlZUyZMgWfz0diYiK5ubmsWrWKdevW8clPfpKdO3fS2NjITTfdxA033AD8Y6mT+vp6LrvsMs455xzee+89hgwZwgsvvEBcXFyv5Au5Qo/xqdBFBO58cS3rdtf26mNOGJzMj6+Y2O39v/zlLyktLWXVqlUsWbKE+fPnU1paevTUwt///vekp6fT0NDAjBkzuOaaa8jIyPjYY2zatIknnniChx56iM985jM8++yzXHvttb2SP+QKPVZTLiLST8ycOfNj54nfc889PP/88wDs3LmTTZs2HVPoBQUFTJkyBYDp06ezbdu2XssTgoXuH6E3tWqELjKQHW8k3VcSEhKO3l6yZAl/+9vfWLp0KfHx8cyZM6fL88hjYmKO3o6MjKShoaHX8oTem6JR/shNGqGLSB9LSkqirq6uy/tqampIS0sjPj6eDRs2sGzZsj5OF8Ij9EaN0EWkj2VkZHD22WczadIk4uLiyMnJOXrfpZdeygMPPEBhYSFjx45l9uzZfZ4vdAtdb4qKiAf+93//t8vtMTExvPrqq13ed2SePDMzk9LS0qPbb7311l7NFnJTLrFRelNURKQrIVfoUZER+CKNw80aoYuIdBRyhQ6QHOujpqHF6xgiIv1KSBZ6SpyPWhW6iMjHhGShJ8dphC4i0tkJC93MYs1suZmtNrO1ZnZnF/t82cyqzGxV4OufghPXL0WFLiJyjJ6M0JuAC51zZwBTgEvNrKsTLJ9yzk0JfD3cqyk7UaGLSChITEwEYPfu3SxYsKDLfebMmUNxcXGvPN8Jz0N3zjmgPvCtL/DleuXZT1FqvApdRELH4MGDWbRoUdCfp0dz6GYWaWargErgdefc+13sdo2ZlZjZIjMb2s3j3GBmxWZWXFVVdcqhMxNjqGlo0cVFItKnvve9731sPfSf/OQn3HnnnVx00UVMmzaNyZMn88ILLxzzc9u2bWPSpEkANDQ0sHDhQgoLC/nsZz/bq2u59OhKUedcGzDFzFKB581sknOutMMuLwJPOOeazOxG4BHgwi4e50HgQYCioqJTHuUPSokFoLK2iWEZ8af6MCISyl69Dfau6d3HHDQZLvtlt3cvXLiQm2++mW9+85sAPP300/zlL3/hlltuITk5mX379jF79myuvPLKbj8T9P777yc+Pp6SkhJKSkqYNm1ar8U/qbNcnHMHgSXApZ22Vzvnjnwm3EPA9F5J143BKf7F4PfU9N7/2URETmTq1KlUVlaye/duVq9eTVpaGrm5udxxxx0UFhYyd+5cdu3aRUVFRbeP8fbbbx9d/7ywsJDCwsJey3fCEbqZZQEtzrmDZhYHzAXu7rRPrnNuT+DbK4H1vZawC0dG6Htrj12aUkQGiOOMpINpwYIFLFq0iL1797Jw4UIef/xxqqqqWLFiBT6fj/z8/C6Xze2ou9H76erJCD0XWGxmJcAH+OfQXzKzn5rZlYF9vhM4pXE18B3gy0FJG3Ck0PfUqNBFpG8tXLiQJ598kkWLFrFgwQJqamrIzs7G5/OxePFitm/fftyfP++883j88ccBKC0tpaSkpNey9eQslxJgahfbf9Th9u3A7b2W6gQSY6JIjo2i/MDhvnpKEREAJk6cSF1dHUOGDCE3N5cvfOELXHHFFRQVFTFlyhTGjRt33J//xje+wfXXX09hYSFTpkxh5syZvZYt5JbPPWJUdiKbKupPvKOISC9bs+Yfb8ZmZmaydOnSLverr/d3VH5+/tFlc+Pi4njyySeDkiskL/0HGJOTxKZKFbqIyBEhW+ijc5LYf6iZffVNJ95ZRGQACNlCH5Pjv6T2o71df76fiIQn/8Xr4e9Ufs+QLfSxOUkAfFShQhcZKGJjY6murg77UnfOUV1dTWxs7En9XMi+KZqVFENqvI+NemNUZMDIy8ujvLyc01k6JFTExsaSl5d3Uj8TsoVuZv43RjVCFxkwfD4fBQUFXsfot0J2ygX80y4bK+rC/s8vEZGeCOlCHzMoibrGVi0BICJCiBf6kTdGN+hMFxGR0C70Udn+UxfLdIGRiEhoF3p6QjTpCdFsVqGLiIR2oQOMykqkrEqFLiIS8oU+MjtRI3QREcKh0LMSOHC4hWqt6SIiA1zIF/qRN0Y1SheRgS58Cl3z6CIywIV8oQ9OiSPOF0lZ5SGvo4iIeCrkCz0iwhiZnaARuogMeCFf6AAjsxJ1cZGIDHhhUeijshLZdbCBQ02tXkcREfFMeBR64I3RLVWaRxeRgSusCl1XjIrIQBYWhT48I4HICNO56CIyoIVFoUdHRTA8I16FLiIDWlgUOvjPdNGpiyIykIVNoY/KTmTbvkO0tLV7HUVExBMnLHQzizWz5Wa22szWmtmdXewTY2ZPmdlmM3vfzPKDEfZ4RmUl0tru2LH/cF8/tYhIv9CTEXoTcKFz7gxgCnCpmc3utM9XgQPOuVHAfwB3927ME9MiXSIy0J2w0J3fkZb0Bb5cp92uAh4J3F4EXGRm1mspe2CkCl1EBrgezaGbWaSZrQIqgdedc+932mUIsBPAOdcK1AAZXTzODWZWbGbFVVVVp5e8k8SYKAYlx2oJABEZsHpU6M65NufcFCAPmGlmkzrt0tVovPMoHufcg865IudcUVZW1smnPYFR2TrTRUQGrpM6y8U5dxBYAlza6a5yYCiAmUUBKcD+Xsh3UkZl+xfpcu6Y/5eIiIS9npzlkmVmqYHbccBcYEOn3f4MXBe4vQB403nQqiOzEznU3Mbe2sa+fmoREc/1ZISeCyw2sxLgA/xz6C+Z2U/N7MrAPr8DMsxsM/DPwG3BiXt8o7L0xqiIDFxRJ9rBOVcCTO1i+4863G4EPt270U7eyOwEwF/o547u/Tl6EZH+LGyuFAXISowhOTZKI3QRGZDCqtDNjAmDkyndVeN1FBGRPhdWhQ4wfXgaa3fX0tDc5nUUEZE+FZaF3truWF1+0OsoIiJ9KuwKferQNABWbD/gcRIRkb4VdoWelhDNyKwEFbqIDDhhV+gARcPTKd62n7Z2XTEqIgNHWBb67JHp1Da2smFvrddRRET6THgW+gj/Qo9Ly6o9TiIi0nfCstBzU+LIz4hn2RYVuogMHGFZ6ABnj8pkaVk1jS06H11EBoawLfRPTMjhUHMb75Xt8zqKiEifCNtCP2tkJkkxUfx1bYXXUURE+kTYFnp0VARzxmXz+roKnb4oIgNC2BY6wCUTc6g+1MzKHbrISETCX1gX+vljsoiOjOC10r1eRxERCbqwLvSkWB/nj83i+Q930dSqs11EJLyFdaEDXDt7ONWHmnlNb46KSJgL+0I/d1QmQ9PjeGzZdq+jiIgEVdgXekSEcd2Z+Szfup9VO7VGuoiEr7AvdICFM4eRHBvFbxdv9jqKiEjQDIhCT4yJ4ivnFPDXdRUs37rf6zgiIkExIAod4OvnjSQ3JZY7X1yrC41EJCwNmEKPi47k9nnjWbu7lqeLd3odR0Sk1w2YQge4ojCXGflp3P2XDVTVNXkdR0SkVw2oQjcz7rq6kMPNbfzgT2twTlMvIhI+BlShA4zKTuTWi8fw2toKHnx7i9dxRER6zQkL3cyGmtliM1tvZmvN7KYu9pljZjVmtirw9aPgxO0dXzt3BPMmD+KXf9nASyW7vY4jItIronqwTyvwXefcSjNLAlaY2evOuXWd9nvHOXd570fsfWbGrz49haq697nlqVUkxkQxZ2y217FERE7LCUfozrk9zrmVgdt1wHpgSLCDBVtcdCQPXzeD0dlJ3PjYCoq36fx0EQltJzWHbmb5wFTg/S7uPtPMVpvZq2Y2sZufv8HMis2suKqq6qTD9raUOB+PfGUmuSlxXP+HD3jrI+8ziYicqh4XupklAs8CNzvnajvdvRIY7pw7A7gX+FNXj+Gce9A5V+ScK8rKyjrVzL0qKymGx/9pFkPS4rj+D8t5+J0tOvtFREJSjwrdzHz4y/xx59xzne93ztU65+oDt18BfGaW2atJg2hwahzPfuMsLp4wiJ+/vJ7vPrOaxhatny4ioaUnZ7kY8DtgvXPu193sMyiwH2Y2M/C41b0ZNNgSYqL47RemccvcMTy3cheffXAZFbWNXscSEemxnozQzwa+CFzY4bTEeWZ2o5ndGNhnAVBqZquBe4CFLgTnLSIijJvmjuaBa6ezqaKOK+79Ox/q80hFJESYV71bVFTkiouLPXnuntiwt5avPVpMRW0Td31qMtdMz/M6kogIZrbCOVfU1X0D7krRnho3KJkXvnUO04el8d1nVvPzl9bR2tbudSwRkW6p0I8jPSGaR786k+vOHM7Df9/K9X/8gJrDLV7HEhHpkgr9BHyREdx51STuvmYyy7ZU86n732X3wQavY4mIHEOF3kOfnTGMx746i6raJr78h+Ucbm71OpKIyMeo0E/CrBEZ3H/tdDZV1vODP5XqAiQR6VdU6CfpnNGZ3HTRaJ5buYtbnynRG6Ui0m/0ZLVF6eQ7F47GOfjNG5vISorhtsvGeR1JREQj9FMREWHc8okxLJwxlAffLmPVzoNeRxIRUaGfjjvmjycnOZZ/WbSaplat/SIi3lKhn4bkWB+/+NQkPqqo51d//cjrOCIywKnQT9OF43L4/KxhPPj2FlZq3RcR8ZAKvRfcMW88Ockx/OiFUtrbdSqjiHhDhd4LEmOiuP2y8ZTuquWlNXu8jiMiA5QKvZdcecZgxuQk8p9/+0jnpouIJ1TovSQiwrhl7hi2VB3iuZW7vI4jIgOQCr0XXTJxENOHp3HXq+vZf6jZ6zgiMsCo0HtRRIRx19WTqW1s5Z43NnkdR0QGGBV6LxuTk8TVU4fwxPIdVNbpM0lFpO+o0IPgWxeMoqWtnd+9s9XrKCIygKjQgyA/M4G543P477e3ULqrxus4IjJAqNCD5LsXjwXgq498QGOL1nkRkeBToQfJ2EFJ/PH6GVTUNvF08U6v44jIAKBCD6Lzx2QxIz+N+5eUaTVGEQk6FXoQmRnfuWg0e2oaeaa43Os4IhLmVOhBds6oTKYOS+UHfypl18EGr+OISBhToQeZmfGtOaMA+NR972rqRUSC5oSFbmZDzWyxma03s7VmdlMX+5iZ3WNmm82sxMymBSduaJo7IYcfzB9PZV0Tv9YHYYhIkPRkhN4KfNc5Nx6YDXzLzCZ02ucyYHTg6wbg/l5NGQb+6dwRLJiexx/e28ZuTb2ISBCcsNCdc3uccysDt+uA9cCQTrtdBTzq/JYBqWaW2+tpQ9zNc0fjnOPeNzd7HUVEwtBJzaGbWT4wFXi/011DgI4nW5dzbOljZjeYWbGZFVdVVZ1c0jCQlxbPNdPyeGL5Dh5+Z4vXcUQkzPS40M0sEXgWuNk5V9v57i5+5JjPYnPOPeicK3LOFWVlZZ1c0jBx89wxAPz85fWs2L7f4zQiEk56VOhm5sNf5o87557rYpdyYGiH7/OA3acfL/wMSoml9M5LGJwSy/efL9WnG4lIr+nJWS4G/A5Y75z7dTe7/Rn4UuBsl9lAjXNOH67ZjcSYKO6YP54Ne+tYtEIXHIlI7+jJCP1s4IvAhWa2KvA1z8xuNLMbA/u8AmwBNgMPAd8MTtzwMX9yLtOGpXLbc2vYuu+Q13FEJAyYc8dMdfeJoqIiV1xc7Mlz9xcrtu/nmvuXAvDhDz9BWkK0x4lEpL8zsxXOuaKu7tOVoh6aPjydf1tQSFSE8YMXSr2OIyIhToXusc8UDeWmi0bzcskeXl9X4XUcEQlhKvR+4Ovnj2TcoCS+9mgxizdUeh1HREKUCr0fiI6K4KdXTQLg+j9+wGPLtnucSERCkQq9n5hZkM7TXz+T2SPSufPFtZSUH/Q6koiEGBV6PzKzIJ0Hrp1ORkIMV//2PfbUaBEvEek5FXo/kxofzb9ePYnWdseZd71JRW2j15FEJESo0PuhC8flcOvF/jVfbnlqFW3t3lwrICKhRYXeT337wtHcfc1k3iur5r7Fm1mx/QD1Ta1exxKRfkyF3o99pmgolxfm8uvXP+Ka+9/jk/e9y776Jq9jiUg/pULvx8yMf716MjML0pk7Poed+w/zyfveZUtVvdfRRKQfUqH3c8mxPp7++pk8fF0RP7tqEuUHGrjwV2/xl1ItZikiH6dCDyGfLsrj5rmjiTD4zpOrKN1V43UkEelHVOghxMy4ee4Yin/wCTISorn83r/zxPIdtOssGBFBhR6S0hOiue8L0wC4/bk1fO/ZEpW6iKjQQ9W0YWk8dcNsPjllMM+sKOeHL5Ti1dr2ItI/RHkdQE7drBEZzBqRwaCUOB54q4zH39/BohvPpCg/3etoIuIBjdDDwP+9ZCxXTx0CwBd/t5wV2w94nEhEvKCPoAsjVXVNfPqB99hWfZixOUl8ZsZQvnJ2Pv7P+RaRcKCPoBsgspJi+OP1MwHYWFHHz15ax89fXk9NQ4vHyUSkL2iEHoZ27j9MZIRx3+LNPP7+DgCunT2MO6+cRGSERusioex4I3S9KRqGhqbHA/CzwKcgPf7+Dh5btoPyAw18tmgoF4zLJtYX6WVEEQkCjdAHiAfeKuOXr24AYNygJP7r89MYlZ3ocSoROVkaoQs3nj+SEZkJvLa2gsUbK5n767eYkJvMj66YwOwRGV7HE5FeoBH6ALRz/2HO/bfFAJjBbZeO4+KJg8hNidVUjEg/d7wRugp9gKo53EJlXSP/+bdNvLzGv3JjZmIM935uKmeO1IhdpL/SlIscIyXeR0q8j3s/N5XhGfG8smYPUZERfO6hZQDMmzyIn141iczEGI+TikhPnXCEbma/By4HKp1zk7q4fw7wArA1sOk559xPT/TEGqH3PzUNLXzxd+9TUl5DZITR1u6YODiZb8wZyfzJubpASaQfOK0pFzM7D6gHHj1Ood/qnLv8ZEKp0Psn5xyNLe1s2VfP/Hv+fnT72aMymDYsjXNHZzGzQGvFiHjltKZcnHNvm1l+b4eS/snMiIuOZOLgFD784SdYv7eWDXvq+OlL63h3czX3vrmZBdPzOGdUJueMztSUjEg/0ltz6Gea2WpgN/7R+tqudjKzG4AbAIYNG9ZLTy3BkpYQzVkjMzlrZCZF+Wm8XLIHM+OBt8pYtKIcgFsvHsPIrESyk2OYPlwjdxEv9egsl8AI/aVuplySgXbnXL2ZzQN+45wbfaLH1JRL6Fq8sZL7l5SRGufjr+sqjm6fN3kQRcPTmTA4mVkF6ZpzFwmCoJ7l4pyr7XD7FTP7rZllOuf2ne5jS/90wdhsLhibDcCLq3fzP8u2MyM/jfsWl/HKmr0AzB2fw4TByUwZmsI5o7KIjtI6cCLBdtqFbmaDgArnnDOzmfhXcKw+7WQSEq44YzBXnDEY8Jf4H97dRkFmAr95YxN/W+8fvQ9NjyMqIoL5k3P52rkjSIn3eRlZJGz15CyXJ4A5QCZQAfwY8AE45x4ws28D3wBagQbgn51z753oiTXlEt427K3l4Xe2MqsgnZ+9tI7axlYAUuN9HDzcwpkjMvjcrGHMHpFOcqxPV6iK9JCuFBVPNTS38caGCoamxXPL06vYUnXomH3umDeO2SMySI71kZ+Z4EFKkdCgQpd+o7WtnXfLqpk9Ip3nVu7ijufXEOeL5HBz29F9zhiaSkFGPLmpcSyYnsfILK0KKXKECl36vcUbKvnZy+soyEhgU2U9O/YfBjh6xWp0ZARXnDGYC8ZlkRgTxXmjs4jQh3XIAKRCl5BS39TKo0u3MSM/nZdW7+aRpdsBiImKoKm1HYCRWQkMSYtnSGosC6bn6Rx4GTBU6BLSahtbiPNFUlHbyH2LN9PU0s66PbVs2Fv3sf0yE6O5/uwCrp42hIbmNkZoqkbCkApdwk5bu+PV0j0MSY3jzQ2V3Pvm5mP2mTYslclDUmhtd5w/JovZIzOorG1kVHaSB4lFeocKXcJeZV0jMVGRbK6s5xcvr6OkvIZh6fFs2XfsGTVThqYya0Q6dY2tXFE4mNkjdFWrhA4VugxI7e2O5z7cRUJ0JK3tjt+8sYnNlfXkZ8SzrfrwMfvPn5zL9WfnU32omdkFGboASvolFbpIB23tjj+v3kXN4RYiIyP44Z9KAUiOjTp6AVR0ZASXF+ay88BhPjEhh8/PGs6BQ83kpcVpNC+eUqGLHEdtYwt1ja0kxkRxzxubeG3tXsbnJvN6h4XHjshNieUrZxewbk8tl00axNzxOTp9UvqUCl3kFKzYvp+yqkOMzk7ku8+sZkvVIYakxrHrYMPRfeKj/RdFnTs6k+9ePJbdBxuYODiZ4Rm62lWCQ4UucpqcczS1thMTFcEb6yv5++Z9FOalcNtza2gOnBvf0TXT8jhwuJk5Y7NYMD2PA4dbyEiI1po1ctpU6CJB0tTaxqaKevIzE3jwrTKeKt5JUX46L5fsObpPVITR2u7/d/ajyyfQ0NLGkNQ4Li/MJSpSywrLyVGhi/SxzZX1PFO8k1kj0nno7a0s3XLsitJjchL5qKKe8bnJ3DJ3NNFREfgiIzhrZIbeeJVuqdBFPFbb2EJyrI+/b9rHE8t3MGFwMk8s30H5gYZj9p2Rn0ZeWjwFmQl86czhxERFEhEBMVGarhEVuki/1NzazlsfVTEjP41XS/dy/5IyRmQlUFJew/5DzQAkxkRR3+Q/lfLHV0xg4uAUDjW3MmdMlkbxA5QKXSSE1Bxu4fHl2xmaFs9f1u792Hz8ERMHJxPriyQxJopvXTCKzMRo9tQ0arpmAFChi4SwitpGUuJ8vL91P//x+kcA7D/UfHSJ4Y5mFaRz1shM9tY28LVzR5CbEsfBhmZyU+L6OrYEiQpdJMy0tLXzypo95KXFsXL7QX7xynoA0hOij07XHFlLHuDT0/OYV5jL2l01XHdWPokx/o8T1mg+9KjQRcJce7vDDGoaWnj4na0kxESxp6aBRwNryUdHRtDc5j9fPiXOR2ZiNGVVh/jB/PFcNWUIy7fu55KJOTqNMgSo0EUGqIraRpzzj+j//bWNbK8+RGZiDG9sqDxm3xFZCSycMZTlWw/wuZlDuWBsNit3HGDqsDQitbxBv6FCF5GPWVNeQ0NLG7UNLdzx/Boq65q6XYUyPyOeWy8ZywurdnP11CFcOmkQG/bWMTYnSevYeECFLiLdcs7R7qDdOZ4u3smO6sMMy4jn+8/7V6GM80XS0PKPD/E+Mn2TlRTD3ddM5p1N+7h4wiDOHJnB/kPNpMb5VPRBpEIXkZPW0NxGZV0jOcmx/M/S7RRv38/sERnc+eK6LvefOiyVD3ccJDcllns/N5WXSvZw/pgsLhiXzb76JlLifPg0R3/aVOgi0msaW9rYuu8QBZkJPLp0Gy+u3sP04Wn88b1tXe5/1sgM3iurJikmige/VMSaXQcZnZPEBWOzOdzcSmxUpEb0J0GFLiJBV13fxLtl1VwwNotH3tvGcyt3MW14Gs9/uOvo6ZMdzRmbxZKNVSTFRvGfn53Cjv2HaWt3fGHWcPbVNxEVaTp/vgsqdBHxzOHmVpaWVVOUn86Ty3fwyHvbGJebzPKt+48ua9BRdlIMlXVNAHz7glFU1DZS09DCDy+fQH1TKwcPt3DmyAxa29qJMBtwo/vTKnQz+z1wOVDpnJvUxf0G/AaYBxwGvuycW3miUCp0kYGt5nALr5TuYc7YLN7+qIqnPvAvPfzhjgN8sO3AMftHR0UcXXv+wsC8fEl5DXdfM5nm1naWbKzimxeMIjsphs1V9WG73s3pFvp5QD3waDeFPg/4P/gLfRbwG+fcrBOFUqGLSFecc5SU1zA8I57KuiYeW7adiYOT+WDbARatKAcgKTaKusZjR/cdzchPIyMhhtLdNfzLpeMYk5PIKyV7uP7sApJio6iqbwrJKZ3TnnIxs3zgpW4K/b+BJc65JwLfbwTmOOeOXVGoAxW6iJys3QcbSIqNot3Bi6t3Mz43ibrGVh5btp0xOUnER0fy//7qX+9mcEosu2saj3mM1HgfSbFR7NzfwKUT/adb3r+kjC+fnc/8ybk89M4Wrp6WR0FGAo8s3canpg4hLy2OpVuqmTI0lfjoKJpa2zxbzjjYhf4S8Evn3N8D378BfM85d0xbm9kNwA0Aw4YNm759+/aT+DVERE7sUFMr7c4R64vk1dK9xPkiaWt33P9WGfvqmhiVnchbH1UBx55j350RmQls2XcIgM8U5fF0cTlTh6Xyfy8Zy12vbGBWQTrfvnAUf3h3G9OHp3HemCz+9OEuRuckMnFwCtv2HSIjMZqkWB/V9U0kn8YpnMEu9JeBuzoV+r8451Yc7zE1QhcRr2yurKOhuZ3BqbG8sGo3ja1tTBmayr+/tpGd+w/z+VnD+a83N9Hu4IozBvPi6t0AjMxKoKzq0Akff0RWAlsC+80vzD26BPLtl43jrlc3cMN5I7hj3vhTyn68QkQeRJYAAAZUSURBVI86pUf8uHJgaIfv84DdvfC4IiJBMSo76ejtr5xTcPT289/MPHr7potGU9vQQlpCND+9ciJlVfUU5adTuquGZVuquWZaHs9/uIvi7fu5dvZwHn5nK0s2VvLp6UN5bd1ewP8xgx3Xs7/r1Q2Af4G0YOiNEfp84Nv8403Re5xzM0/0mBqhi0i4am937Nh/mOEZ8dQ2tvLOpiouGJvNuj21vFyyh+vOyqcgM+GUHvu0Ruhm9gQwB8g0s3Lgx4APwDn3APAK/jLfjP+0xetPKaWISJiIiDDyA4WdEufj8sLBAMzIT2dGfnrQnveEhe6c+9wJ7nfAt3otkYiInBKtlCMiEiZU6CIiYUKFLiISJlToIiJhQoUuIhImVOgiImFChS4iEiY8+4ALM6sCTnV1rkxgXy/GCbZQyhtKWSG08oZSVgitvKGUFU4v73DnXFZXd3hW6KfDzIq7u/S1PwqlvKGUFUIrbyhlhdDKG0pZIXh5NeUiIhImVOgiImEiVAv9Qa8DnKRQyhtKWSG08oZSVgitvKGUFYKUNyTn0EVE5FihOkIXEZFOVOgiImEipArdzC41s41mttnMbvM6zxFmts3M1pjZKjMrDmxLN7PXzWxT4L9pge1mZvcEfocSM5vWB/l+b2aVZlbaYdtJ5zOz6wL7bzKz6/ow60/MbFfg+K4ys3kd7rs9kHWjmV3SYXvQXytmNtTMFpvZejNba2Y3Bbb312PbXd7+enxjzWy5ma0O5L0zsL3AzN4PHKunzCw6sD0m8P3mwP35J/o9+iDrH81sa4djOyWwPTivBedcSHwBkUAZMAKIBlYDE7zOFci2DcjstO3fgNsCt28D7g7cnge8ChgwG3i/D/KdB0wDSk81H5AObAn8Ny1wO62Psv4EuLWLfScEXgcxQEHg9RHZV68VIBeYFridBHwUyNRfj213efvr8TUgMXDbB7wfOG5PAwsD2x8AvhG4/U3ggcDthcBTx/s9+ijrH4EFXewflNdCKI3QZwKbnXNbnHPNwJPAVR5nOp6rgEcCtx8BPtlh+6PObxmQama5wQzinHsb2H+a+S4BXnfO7XfOHQBeBy7to6zduQp40jnX5Jzbiv9jEGfSR68V59we59zKwO06YD0whP57bLvL2x2vj69zztUHvvUFvhxwIbAosL3z8T1y3BcBF5mZHef36Ius3QnKayGUCn0IsLPD9+Uc/8XYlxzwVzNbYWY3BLblOOf2gP8fEpAd2N5ffo+Tzed17m8H/jT9/ZEpjONk6vOsgT/vp+IfmfX7Y9spL/TT42tmkWa2CqjEX25lwEHnXGsXz300V+D+GiCjr/J2zuqcO3JsfxE4tv9hZjGds3bKdFpZQ6nQrYtt/eWcy7Odc9OAy4Bvmdl5x9m3P/8e0H0+L3PfD4wEpgB7gF8FtveLrGaWCDwL3Oycqz3erl1s6w95++3xdc61OeemAHn4R9Xjj/PcnubtnNXMJgG3A+OAGfinUb4XzKyhVOjlwNAO3+cBuz3K8jHOud2B/1YCz+N/4VUcmUoJ/LcysHt/+T1ONp9nuZ1zFYF/LO3AQ/zjz2XPs5qZD385Pu6cey6wud8e267y9ufje4Rz7iCwBP98c6qZHfmA+47PfTRX4P4U/NN3fZq3Q9ZLA9NczjnXBPyBIB/bUCr0D4DRgXe4o/G/6fFnjzNhZglmlnTkNnAxUIo/25F3qK8DXgjc/jPwpcC73LOBmiN/nvexk833GnCxmaUF/iS/OLAt6Dq9x/Ap/Mf3SNaFgbMbCoDRwHL66LUSmJ/9HbDeOffrDnf1y2PbXd5+fHyzzCw1cDsOmIt/3n8xsCCwW+fje+S4LwDedP53Grv7PYKddUOH/7Eb/rn+jse2918Lp/qurhdf+N8Z/gj/PNr3vc4TyDQC/zvoq4G1R3Lhn7t7A9gU+G+6+8e74fcFfoc1QFEfZHwC/5/SLfhHAF89lXzAV/C/obQZuL4Ps/5PIEtJ4B9Cbof9vx/IuhG4rC9fK8A5+P8cLgFWBb7m9eNj213e/np8C4EPA7lKgR91+De3PHCsngFiAttjA99vDtw/4kS/Rx9kfTNwbEuBx/jHmTBBeS3o0n8RkTARSlMuIiJyHCp0EZEwoUIXEQkTKnQRkTChQhcRCRMqdBGRMKFCFxEJE/8fJPP1H2bGkj0AAAAASUVORK5CYII=\n", | |
| "text/plain": [ | |
| "<Figure size 432x288 with 1 Axes>" | |
| ] | |
| }, | |
| "metadata": { | |
| "needs_background": "light" | |
| }, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "learn.recorder.plot_loss()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Note there is no validation loss or validation perplexity as we we didn't specify a validation set (as per Karpathy's notebook)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 15, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "O God, O God! to our bloody supply the horsemen are stay and forth. But wherefore stand you to the crown?\n", | |
| "\n", | |
| "DUKE VINCENTIO:\n", | |
| "These are the people's voices,\n", | |
| "To make the varlet concerning me to town.\n", | |
| "I was no brother out the man that wilt thou have made forth thyself.\n", | |
| "\n", | |
| "GLOUCESTER:\n", | |
| "This is a man, and thy supposed king,\n", | |
| "Like men cracking in the world,\n", | |
| "Which were instruction mortal tiden let us hence;\n", | |
| "And, for the dangerous seats of the earth\n", | |
| "Would be the sheep that was thus fined\n", | |
| "An an unusual good deeds, that does not.\n", | |
| "\n", | |
| "CORIOLANUS:\n", | |
| "Why that is this?\n", | |
| "\n", | |
| "FRANCISCA:\n", | |
| "What was? much of you, if I show much, we have said\n", | |
| "Shall be your first with your shame, take you into\n", | |
| "the court: back up my packet again of his silk,\n", | |
| "And that you might repossess him well\n", | |
| "Then when he did show more wantons,\n", | |
| "Than the proudest hollow can find it off.\n", | |
| "\n", | |
| "MARCIUS:\n", | |
| "Though they shall feel, they change put us on you.\n", | |
| "\n", | |
| "CLARENCE:\n", | |
| "Or the duke's death, the love been done:\n", | |
| "I am so broad to live in thee and thy looks.\n", | |
| "\n", | |
| "RICHARD:\n", | |
| "I will do well, I see thee here.\n", | |
| "\n", | |
| "GRUMIO:\n", | |
| "Ay, sir, the man take note upon your gates: arry this good\n", | |
| "lady, by you.\n", | |
| "\n", | |
| "VINCENTIO:\n", | |
| "Thou art perfect these are at the last I see thee\n", | |
| "In such a disguised, though they be not taught but they are not; but they\n", | |
| "are full of vanity. They say, the duke was too much,\n", | |
| "To make a schoolar, a silken posterity,\n", | |
| "That it may call you now?\n", | |
| "\n", | |
| "BIANCA:\n", | |
| "What, my gracious lady?\n", | |
| "\n", | |
| "PETRUCHIO:\n", | |
| "Why, then, 'tis no less than this, but a sharp-pointed match.\n", | |
| "What is the bloody which this story rich,\n", | |
| "To make thee stranger about the stone,\n", | |
| "And both thy speech. What is it thou,\n", | |
| "To be the man to be thus apprehended;\n", | |
| "And so, I trust I. What if it be so,\n", | |
| "I will content thee with thy wisdoms: better for thy death,\n", | |
| "By this time I had rather had been so\n", | |
| "fidiused for a maid: there is no less less extend.\n", | |
| "\n", | |
| "KING RICHARD II:\n", | |
| "Is it even that so strict me?\n", | |
| "\n", | |
| "GLOUCESTER:\n", | |
| "And teach you, my lord, is dead?\n", | |
| "\n", | |
| "GLOUCESTER:\n", | |
| "And why the king's daughter is too slander'd?\n", | |
| "\n", | |
| "LADY GREY:\n", | |
| "To tell you plain, I pray you,\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "from minGPT.mingpt.utils import sample\n", | |
| "\n", | |
| "context = \"O God, O God!\"\n", | |
| "x = torch.tensor([dls.char_transform.stoi[s] for s in context], dtype=torch.long)[None,...].to(dls.device)\n", | |
| "y = sample(model, x, 2000, temperature=0.9, sample=True, top_k=5)[0]\n", | |
| "completion = ''.join([dls.char_transform.itos[int(i)] for i in y])\n", | |
| "print(completion)" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.7.6" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 2 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment