Come with me now on a journey through code and data...

My Python Setup and More

I've started using Jupyter Notebooks for making notes on everything I'm learning. Even JavaScript. From a quick Google, it looks like kernels are available for most other commonly used programming languages too so I'll probably continue adding kernels and write notebooks for everything.

Here's my setup:

JavaScipt Notebooks

For JavaScript I just install a kernel from IJavaScript. It relies on Jupyter Notebooks already being installed which I'll go into below, but it was as simple to set up as just installing it globally (I tried locally first and it didn't work) using Node:

node ijsinstall

and running:

jupyter notebook.

Python Workspace Setup

I try to only ever use Python3 these days but every now and then come across legacy code so need Python2.7 installed too. Here's my setup to try keep things clean. Most of the process was copied from here.

I use:

  1. pyenv to install Python interpreters
  2. pyenv-virtualenv to manage virtualenvs
  3. pyenv-virtualenvwrapper to automate virtualenv management for my projects

Install PyEnv and Python Versions

Before installing Python interpreters I first install the pyenv tools above.

virtualenvwrapper puts all virtualenvs in a one directory, and all project code into another.

  • All my virtualenvs go in ~/.ve

  • All my projects go in ~/workspace

virtualenvwrapper relies on a couple of variables which need to be exported in .bashrc to work. Initializing pyenv should also be added here.

In [5]:
export WORKON_HOME=~/.ve
export PROJECT_HOME=~/workspace
eval "$(pyenv init -)"

(Remember to reload the file after changes: source ~/.bashrc)

Then use pyenv to install the interpreters that will be used globally. The script from the following repo simplifies getting the latest version avaialble.

In [ ]:
git clone https://github.com/momo-lab/pyenv-install-latest.git "$(pyenv root)"/plugins/pyenv-install-latest

pyenv install-latest
pyenv install-latest 2.7

Or to specify precisely which version simply use

In [ ]:
pyenv install 3.6.0
pyenv install 2.7.13

Installing Jupyter

virtualenvs are great for avoiding library conflict issues. But there are some things I'm not going to be installing multiple versions of - like Jupyter Notebook, and it's associated iPython consoles for Python3 and Python2.

I'll still make separate virtual environments for these things, but I'll make them globally available via the PATH so I don't have to repeatedly install them for each new project I make.

Setup Jupyter and IPython for Python3

When I install Jupyter on Python3 it will by default install iPython and the kernel too

In [ ]:
pyenv virtualenv 3.6.6 ipython3

pyenv activate ipython3
pip install jupyter
python -m ipykernel install --user
pyenv deactivate

Setup IPython for Python2

For Python2 I only need to install iPython and the kernel

In [ ]:
pyenv virtualenv 2.7.15 ipython2

pyenv activate ipython2
pip install ipykernel
python -m ipykernel install --user
pyenv deactivate

Setup Kernel for Bash

A bash kernel for Jupyter Notebooks is available here. Executing bash in a notebook instead of directly in the terminal is good for keeping notes, creating a script that can be rerun, and keeping track of what's already happened. Sure I could just write a script, but this way I can write up what's happening with lovely Markdown. And yeah you can run bash in a Python kernel with magic but what if I don't want to.

In [ ]:
pyenv virtualenv 3.6.6 bash

pyenv activate bash
pip install ipykernel
pip install bash_kernel
python -m bash_kernel.install
pyenv deactivate

Now I add the above Python versions and virtualenvs to the PATH in order of priority so they can be accessed in the right order without having to activate any virtualenv. This means I can call jupyter notebook from anywhere and have access to the same kernel installs.

In [ ]:
pyenv global 3.6.0 2.7.13 ipython3 ipython2 bash

Creating new virtualenvs for projects

This is where pyenv-virtualenvwrapper and the ~/workspace directory come in.

It's time to add the following to the bottom of the ~/.barshrc file and reload (source ~/.bashrc):

In [ ]:
pyenv virtualenvwrapper_lazy

This will install virtualenvwrapper's dependencies if they’re not present, and allows use of the virtualenvwrapper commands.

New Project

For example, now if I run:

In [ ]:
mkproject myProject

An empty project directory will be created at ~/workspace/myProject and we will be relocated to this directory. A new virtualenv of the same name which has been stored in ~/.ve/myProject will be activated. By default the virtualenv is made with Python3.

Existing Project

If I want to work on an existing project which has already been set up using the method described above I can run:

In [ ]:
workon myProject

And I'll again be transported to the relevant project directory in ~/workspace with the project's virtualenv activated.

Python2 Project

If I've cloned a project which runs on Python2 to my workspace ~/workspace/myProject and I need a virtualenv to work on it.

In [ ]:
mkvirtualenv -a ~/workspace/myProject -p python2 myProject

The above creates a virtualenv using Python2 at ~/.ve/myProject and associates it to the project directory ~/workspace/myProject.

Now running

In [ ]:
workon myProject

Activates the virtualenv and changes the directory to the project's path

Getting Jupyter to Detect the Active virtualenv

For this, I copy a trick from the author of this article.

In [ ]:
ipython profile create
curl -L http://hbn.link/hb-ipython-startup-script > ~/.ipython/profile_default/startup/00-venv-sitepackages.py

This cusomizes iPython's startup process and adds the active virualenvs site-packages to the system path.

I've only recently started using this hack - beforehand I installed ipykernels in every virtualenv so different kernels would be listed for every virtualenv (and every project) I had. This allows working with two different virtualenvs in two different notebooks at the same time, but the list did begin to get out-of-hand.

For now, I'm testing out this method, which only allows working with one virtualenv kernel at a time. So far I've not had any problems.