notebook/dataset-config/dataset-config-other-binary-examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using pre toknized binidx/numpy datasets with infctx trainer\n",
    "\n",
    "The following is the backward compatibility support for the following binary dataset formats\n",
    "\n",
    "- binidx\n",
    "- (NOT SUPPORTED) numpy\n",
    "\n",
    "> Important note: These example focuses only on how to configure your dataset, and does not properly perform checkmarking - for trainer configurations refer to the training notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intial setup\n",
    "\n",
    "Before we go into the dataset setup, lets perform an initial setup for all the folders we need, and a small toy model which we would use throughout the various examples within this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the folders we will need\n",
    "!mkdir -p ../../model/\n",
    "!mkdir -p ../../datapath/\n",
    "!mkdir -p ../../checkpoint/\n",
    "\n",
    "# Initialized a simple L6-D512 model, for both the v4 neox (50277) tokenizer\n",
    "!cd ../../RWKV-v5/ && python3 ./init_model.py --n_layer 6 --n_embd 512 --vocab_size neox --skip-if-exists ../model/L6-D512-neox-init.pth\n",
    "\n",
    "# and rwkv world (65529) tokenizers\n",
    "!cd ../../RWKV-v5/ && python3 ./init_model.py --n_layer 6 --n_embd 512 --vocab_size world --skip-if-exists ../model/L6-D512-world-init.pth\n",
    "\n",
    "# If you have a custom vocab size, you can indicate accordingly as well with an int\n",
    "!cd ../../RWKV-v5/ && python3 ./init_model.py --n_layer 6 --n_embd 512 --vocab_size 20259 --skip-if-exists ../model/L6-D512-V20259-init.pth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training using a binidx dataset\n",
    "\n",
    "The following is the `binidx-enwiki.yaml` settings, for using a textual dataset via huggingface, with most of the comments removed.\n",
    "\n",
    "---\n",
    "```.yaml\n",
    "trainer:\n",
    "  # Low max step limit, so that this dataset run can complete quickly\n",
    "  max_steps: 10\n",
    "  # Resonable batch size, for a more realistic it/s rate\n",
    "  target_batch_size: 32\n",
    "\n",
    "model:\n",
    "  load_model: ../model/L6-D512-neox-init.pth\n",
    "  ctx_len: 1024\n",
    "  lr_init: 3e-4\n",
    "\n",
    "data:\n",
    "  # Directory where the formatted HF dataset will be saved into\n",
    "  data_path: ../datapath/example-binidx/\n",
    "  # Source here points to the binidx file to use (without the .bin / .idx suffix !!!)\n",
    "  source: ../dataset/dataset-config/sample_data_text_document\n",
    "  tokenizer: binidx\n",
    "  test_split: 0.001\n",
    "  test_split_shuffle: false\n",
    "```\n",
    "---\n",
    "\n",
    "### Lets download the example binidx files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the dataset dir\n",
    "!mkdir -p ../../dataset/dataset-config/\n",
    "\n",
    "# Download the binidx file\n",
    "!cd ../../dataset/dataset-config/ && wget -nc https://huggingface.co/datasets/picocreator/RWKV-notebook-assets/resolve/main/wiki40b_world_text_document.bin\n",
    "!cd ../../dataset/dataset-config/ && wget -nc https://huggingface.co/datasets/picocreator/RWKV-notebook-assets/resolve/main/wiki40b_world_text_document.idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### And convert it to HF datapath format + save it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets preload the requried dataset\n",
    "!cd ../../RWKV-v5 && python3 preload_datapath.py ../notebook/dataset-config/example-binidx.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finally run the training process (with the HF datapath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train using the converted binidx format\n",
    "!cd ../../RWKV-v5 && python3 lightning_trainer.py fit -c ../notebook/dataset-config/example-binidx.yaml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rwkv-infctx",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}