Experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment\n",
    "\n",
    "## Graph Classification\n",
    " * dataset: MUTAG, PTC_MR, PROTEINS, NCI1, DD, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, COLLAB\n",
    " * model: 6L with #fc-neuron=128,256 and drop-rate=0.5,0.8,0.9\n",
    "     - should be changed manually in `run-ego-cnn.py`\n",
    " * [default] preprocess: relabel graph(set `-s`) + node/edge labels\n",
    "\n",
    "## Scale-Free Regularizer\n",
    " * dataset: REDDIT-BINARY\n",
    " * model: 6L, 2L, 6L-SF with #fc-neuron=256 and drop-rate=0.8\n",
    " * preprocess\n",
    "     * do not relabel graph(unset `-s`)\n",
    "\n",
    "## Visualizing Critical Structures\n",
    " * dataset: Alkane-vs-Alcohol, Asym-vs-Sym, REDDIT-BINARY\n",
    " * model: 6L, 6LAttention with #fc-neuron=256 and drop-rate=0.8\n",
    " * preprocess\n",
    "     * must not relabel graph (unset `-s`)\n",
    "     * must not include node/edge labels (hardcode in related files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detailed Choices (all options)\n",
    " * [v]: default setting and is used in the submission to ICML'19\n",
    " * [switch]: the switch between options is implemented and can be controlled by setting the arguments/flags in command line\n",
    " * [unused]: experiments that are not chosen to present in paper\n",
    "\n",
    "### Preprocess Settings\n",
    " * [switch] Choice 1. relabeling graphs?\n",
    "     * [v] don't do anything => for visualization experiment\n",
    "     * [v] sort all vertex by degree => for other experiment\n",
    " * Choice 2. selection of neighbors\n",
    "     * [v] 1-hop neighbors\n",
    "     * BFS selection upto k\n",
    "     * [unsued] consecutive nodes in 1D list for Convolution\n",
    " * Choice 3. ordering of neighbors\n",
    "    * 3-1. according to which criterion?\n",
    "         * degree\n",
    "         * [v] 1-WL: label all nodes by running Weisfeiler-Lehman algorithm for 1 run, and calculate #occurence of each 1-WL labels\n",
    "    * 3-2 sorting order?\n",
    "         * larger value first\n",
    "         * [v] smaller value first\n",
    "         \n",
    "### Model Settings\n",
    " * Choice 1. Patchy-San layer\n",
    "    * 1-1. [switch] k: the size of neighborhood\n",
    "        * [v] k=10\n",
    "    * 1-2. consider node/edge labels?\n",
    "        * [v] no => for visualization experiments\n",
    "        * [v] yes if it's provided => for other experiments\n",
    " * Choice 2. Ego-Convolution\n",
    "     * 2-1 [switch] k: # of nodes in a receptive field (self + neighbors)\n",
    "        * k=9\n",
    "        * [v] 17\n",
    " * Choice 3. Classificaion Layers\n",
    "     * 3-1. # of layer: 2 fc layers\n",
    "     * 3-2. # of neurons between the 2 fc layers\n",
    "         * [v] 128: mostly better\n",
    "         * 256: for social network datasets\n",
    " * Choice 4. Regularization\n",
    "     * 4-1 Dropout rate\n",
    "         * 0.5\n",
    "         * [v] 0.8\n",
    "         * 0.9\n",
    "     * 4-2 Batch Normalization\n",
    "         * [v] yes\n",
    "         * no\n",
    " * [unused] Choice 5. Convolution\n",
    "     * 5-1. k: # of consecutive nodes in a receptive field\n",
    "         * same as Ego-Convolution\n",
    "     * 5-2. stride: #nodes to skip in 1-D list\n",
    "         * stride=1\n",
    "         \n",
    "### Visualization Settings\n",
    " * Choice 0. preprocess\n",
    "     * [v] 0-1. must not relabel graphs [do not set `-s` for `preprocess.py`]\n",
    "     * [v] 0-2. must not include node/edge labels [do not let Patchy-San include it]\n",
    " * Choice 1. fixed embedding layers while retrain the visualization model?\n",
    "     * [v] yes, fixed Patchy-San and Ego-Convolution layers\n",
    " * Choice 2. Classification layers\n",
    "     * [v] 1 layer => for REDDIT-BINARY, Asym-Sym\n",
    "     * [v] 2 layers => for Alk-Alc\n",
    "\n",
    "### Plot Settings\n",
    " * Choice 1. threshold to choose importance neighborhoods\n",
    "     * 0.7\n",
    "     * [v] 0.8\n",
    "     * 0.9\n",
    " * Choice 2. threshold to identify edges as critical in a neighborhood\n",
    "     * [v] > 0\n",
    "     * 0.5\n",
    "\n",
    "### Graph Visualization Tool - [Gephi 0.9.0](https://gephi.org/users/download/)\n",
    " * Force Atlas 2 gives better separation of nodes => less manual nodes dragging\n",
    " * By default, the properties of node/edges are not loaded, except the edge-size\n",
    "     * node-color, node-size, and edge-color must be set manually\n",
    "     * click \"Appearance\" -> node/edge -> \"color\"/\"size\" bar -> \"Partition\" by color/size\n",
    " * To generate figures for paper, draw critical node/edges in `#696969` and normal node in `#FFFFFF` and edge in `#000000`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"Monitor the experiment(10-CV) performance\"\"\"\n",
    "import os\n",
    "import _pickle as cPickle\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "import networkx as nx\n",
    "import tensorflow as tf\n",
    "#os.environ[\"CUDA_VISIBLE_DEVICES\"]='0'\n",
    "%matplotlib inline\n",
    "\n",
    "def show_cv(name, size=None, fold=10):\n",
    "    fname = '{}_cv_hist.pkl'.format(name)\n",
    "    if os.path.exists(fname):\n",
    "        hist = cPickle.load(open(fname, 'rb'))\n",
    "        tacc = hist['test_acc']\n",
    "        if len(tacc)==fold:\n",
    "            tacc = np.mean(tacc)\n",
    "            print('{}: test={:.3f}, val={:.3f} takes {:.3f}s'.format(name, tacc, np.mean(hist['cv_vacc']), np.mean(np.array(hist['time']))))\n",
    "        else:\n",
    "            print('{}: test={}, val={:.3f} takes {:.3f}s'.format(name, tacc, np.mean(hist['cv_vacc']), np.mean(np.array(hist['time']))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}