widget_understanding_limited_palette.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [widget] Understanding Limited Palette's Color Control Settings\n",
    "\n",
    "The widget below illustrates how images generated in \"Limited Palette\" mode are affected by changes to color control settings. \n",
    "\n",
    "Press the **\"▷\"** icon to begin the animation. \n",
    "\n",
    "The first run with any particular set of settings will probably show an empty image because the widget is janky and downloads only what it needs on the fly. What can I say: I'm an ML engineer, not a webdeveloper.\n",
    "\n",
    "## What is \"Limited Palette\" mode?\n",
    "\n",
    "In \"Unlimited Palette\" mode, pytti directly optimizes pixel values to try to maximize the similarity between the generated image and the input prompts. Limited Palette mode uses this same process, but adds additional constraints on how the colors in the image (i.e. the pixel values) are selected. \n",
    "\n",
    "We start by specifying a number of \"palettes\". In this context, you can think of a palette as a container with a fixed number of slots, where each slot holds a single color. During optimization steps, colors which are all members of ths same \"palette\" container are optimized together. This has the effect that the \"palette\" objects become sort of \"attached\" to semantic objects in the image. Let's say for example you have an init image of an ocean horizon, so half of the picture is water and half of it is the sky. If we set the number of palettes to 2, chances are one palette will primarily carry colors for painting the ocean and the other will carry colors for painting the sky. This is not a hard-and-fast rule, but you should anticipate that palette size settings will interact with the diversity of semantic content in the generated images.\n",
    "\n",
    "For advice and additional insights about palette and color behaviors in pytti, we recommend the community document [Way of the TTI Artist](https://docs.google.com/document/d/1EvkiHa12ButetruSBr82MJeomHfVRkvczB9-FgqtJ48/edit#) by oxysoft#6139 and collaborators.\n",
    "\n",
    "## Description of Settings in Widget\n",
    "\n",
    "All settings except `smoothing_weight` are specific to Limited Palette mode.\n",
    "\n",
    "* **`palette_size`**: Number of colors in each palette. \n",
    "* **`palettes`**: Total number of palettes. The image will have palette_size*palettes colors total.\n",
    "* **`gamma`**: Relative gamma value. Higher values make the image darker and higher contrast, lower values make the image lighter and lower contrast.\n",
    "* **`hdr_weight`**: How strongly the optimizer will maintain the gamma. Set to 0 to disable.\n",
    "* **`palette_normalization_weight`**: How strongly the optimizer will maintain the palettes’ presence in the image. Prevents the image from losing palettes.\n",
    "* **`smoothing_weight`**: Makes the image smoother using \"total variation loss\" (old-school image denoising). Can also be negative for that deep fried look.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import panel as pn\n",
    "\n",
    "pn.extension()\n",
    "\n",
    "outputs_root = Path('images_out')\n",
    "folder_prefix = 'permutations_limited_palette_2D'\n",
    "folders = list(outputs_root.glob(f'{folder_prefix}_*'))\n",
    "\n",
    "def format_val(v):\n",
    "    try:\n",
    "        v = float(v)\n",
    "        if int(v) == v:\n",
    "            v = int(v)\n",
    "    except:\n",
    "        pass\n",
    "    return v\n",
    "\n",
    "def parse_folder_name(folder):\n",
    "    metadata_string = folder.name[1+len(folder_prefix):]\n",
    "    pattern = r\"_?([a-zA-Z_]+)-([0-9.]+)\"\n",
    "    matches = re.findall(pattern, metadata_string)\n",
    "    d_ = {k:format_val(v) for k,v in matches}\n",
    "    d_['fpath'] = folder\n",
    "    d_['n_images'] = len(list(folder.glob('*.png')))\n",
    "    return d_\n",
    "\n",
    "df_meta = pd.DataFrame([parse_folder_name(f) for f in folders])\n",
    "\n",
    "variant_names = [v for v in df_meta.columns.tolist() if v not in ['fpath']]\n",
    "variant_ranges = {v:df_meta[v].unique() for v in variant_names}\n",
    "[v.sort() for v in variant_ranges.values()]\n",
    "\n",
    "###########################\n",
    "\n",
    "url_prefix = \"https://raw.githubusercontent.com/dmarx/pytti-settings-test/main/images_out/\"\n",
    "\n",
    "image_paths = [str(p) for p in Path('images_out').glob('**/*.png')]\n",
    "d_image_urls = {im_path:im_path.replace('images_out/', url_prefix) for im_path in image_paths}\n",
    "\n",
    "###########################\n",
    "\n",
    "n_imgs_per_group = 40\n",
    "\n",
    "kargs = {k:pn.widgets.DiscreteSlider(name=k, options=list(v), value=v[0]) for k,v in variant_ranges.items()}\n",
    "kargs['i'] = pn.widgets.Player(interval=100, name='step', start=1, end=n_imgs_per_group, step=1, value=1, loop_policy='reflect')\n",
    "\n",
    "@pn.interact(\n",
    "    **kargs\n",
    ")\n",
    "def display_images(\n",
    "    palettes,\n",
    "    palette_size,\n",
    "    gamma,\n",
    "    hdr_weight,\n",
    "    smoothing_weight,\n",
    "    palette_normalization_weight,\n",
    "    i,\n",
    "):\n",
    "    folder = df_meta[\n",
    "        (palettes == df_meta['palettes']) &\n",
    "        (palette_size == df_meta['palette_size']) &\n",
    "        (gamma == df_meta['gamma']) &\n",
    "        (hdr_weight == df_meta['hdr_weight']) &\n",
    "        (smoothing_weight == df_meta['smoothing_weight']) &\n",
    "        (palette_normalization_weight == df_meta['palette_normalization_weight'])\n",
    "    ]['fpath'].values[0]\n",
    "    im_path = str(folder / f\"{folder.name}_{i}.png\")\n",
    "    im_url = d_image_urls[im_path]\n",
    "    return pn.pane.HTML(f'<img src=\"{im_url}\" width=\"700\">', width=700, height=350, sizing_mode='fixed')\n",
    "\n",
    "pn.panel(display_images).embed(max_opts=n_imgs_per_group, max_states=999999999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings shared across animations\n",
    "\n",
    "```\n",
    "scenes: \"fractal crystals | colorful recursions || swirling curves | ethereal neon glow \"\n",
    "\n",
    "scene_suffix: \" | text:-1:-.9 | watermark:-1:-.9\"\n",
    "\n",
    "steps_per_frame: 50\n",
    "save_every: 50\n",
    "steps_per_scene: 1000\n",
    "interpolation_steps: 500\n",
    "\n",
    "image_model: \"Limited Palette\"\n",
    "lock_palette: false\n",
    "\n",
    "animation_mode: \"2D\"\n",
    "translate_y: -1\n",
    "zoom_x_2d: 3\n",
    "zoom_y_2d: 3\n",
    "\n",
    "ViT-B/32: true\n",
    "cutouts: 60\n",
    "cut_pow: 1\n",
    "\n",
    "seed: 12345\n",
    "\n",
    "pixel_size: 1\n",
    "height: 128\n",
    "width: 256\n",
    "```\n",
    "\n",
    "### Detailed explanation of shared settings\n",
    "\n",
    "```\n",
    "scenes: \"fractal crystals | colorful recursions || swirling curves | ethereal neon glow \"\n",
    "```\n",
    "\n",
    "We have two scenes (separated by `||`) with two prompts each (separated by (`|`). \n",
    "\n",
    "```\n",
    "scene_suffix: \" | text:-1:-.9 | watermark:-1:-.9\"\n",
    "```\n",
    "\n",
    "We add prompts with negative weights (and 'stop' weights: `prompt:weight:stop`) to try to discourage generation of specific artifacts. Putting these prompts in the `scene_suffix` field is a shorthand for concatenating this prompts into all of the scenes. I find it also helps keep the settings a little more neatly organized by reducing clutter in the `scenes` field.\n",
    "\n",
    "```\n",
    "steps_per_frame: 50\n",
    "save_every: 50\n",
    "steps_per_scene: 1000\n",
    "```\n",
    "\n",
    "Pytti will take 50 optimization steps for each frame (i.e. image) of the animation. \n",
    "\n",
    "We have two scenes: 1000 steps_per_scene / 50 steps_per_frame = 20 frames per scene = **40 frames total** will be generated.\n",
    "\n",
    "```\n",
    "interpolation_steps: 500\n",
    "```\n",
    "\n",
    "a range of 500 steps will be treated as a kind of \"overlap\" between the two scenes to ease the transition from one scene to the next. This means for each scene, we'll have 1000 - 500/2 = 750 steps = 15 frames that are just the prompt we specified for that scene, and 5 frames were the guiding prompts are constructed by interpolating (mixing) between the prompts of the two scenes. Concretely:\n",
    "\n",
    "* first 15 frames: only the prompt for the first scene is used\n",
    "* next 5 frames: we use the prompts from both scenes, weighting the *first* scene more heavily\n",
    "* next 5 frames: we use the prompts from both scenes, weighting the *second* scene more heavily\n",
    "* last 15 frames: only the prompt for the second scene is used.\n",
    "\n",
    "```\n",
    "image_model=\"Limited Palette\"\n",
    "lock_palette: false\n",
    "```\n",
    "\n",
    "We're using the Limited Palette mode described above, letting the palette change throughout the learning process rather than fitting and freezing it upon initialization.\n",
    "\n",
    "```\n",
    "animation_mode: \"2D\"\n",
    "translate_y: -1\n",
    "zoom_x_2d: 3\n",
    "zoom_y_2d: 3\n",
    "```\n",
    "\n",
    "After each frame is generated, we will initialize the next frame by scaling up (zooming into) the image a small amount, then shift it (translate) down (negative direction along y axis) a tiny bit. The zoom creates a forward motion illusion: adding the y translation creates the effect of the scene rotating away as the viewer passes over it. NB: more dramatic depth illusions are generally achieved using `animation_mode: 3D`, but that mode generates images more slowly and this project already required several days to generate.\n",
    "\n",
    "```\n",
    "ViT-B/32: true\n",
    "```\n",
    "\n",
    "We're using the smallest of openai's pre-trained vision transformer (ViT) CLIP models to guide the animation. This is the AI component which computes the similarity between the image and the text prompt, hereafter referred to as \"the perceptor\".\n",
    "\n",
    "```\n",
    "cutouts: 60\n",
    "cut_pow: 1\n",
    "```\n",
    "\n",
    "For each optimization step, we will take 60 random crops from the image to show the perceptor. `cut_pow` controls the size of these cutouts: 1 is generally a good default, smaller values create bigger cutouts. Generally, more cutouts = nicer images. Setting the number of cutouts too low can result in the image segmenting itself into regions: you can observe this phenomenon manifesting towards the end of many of the animations generated in this experiment. In addition to turning up the number of cutouts, this could also potentially be fixed be setting the cut_pow lower to ask the perceptor to score larger regions at a time.\n",
    "\n",
    "```\n",
    "seed: 12345\n",
    "```\n",
    "\n",
    "If a seed is not specified, one will be generated randomly. This value is used to initialize the random number generator: specifying a seed promotes deterministic (repeatable) behavior. This is an especially useful parameter to set for comparison studies like this."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "3eff1e1332ed0784bebe5613522d192d113df675730803c3b8984f113f4e15fd"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 ('pytti-book-l72HEyWC')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}