ZMQ Version Guide

Update: Until this issue is resolved, there's no need to use the ZMQ versions described in this guide.

ZMQ is an awesome messaging library that we use for communication between clients and servers.

Install

The easy road doesn't work for us

The directions currently in the README (sudo apt-get install libzmq-dev python-zmq) will install the following package versions on Ubuntu 12.04 as of the time of writing.

daniel@daniel-u bot$ dpkg -s libzmq-dev
Package: libzmq-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 873
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: zeromq
Version: 2.1.11-1ubuntu1
Depends: libzmq1 (= 2.1.11-1ubuntu1)
Description: ZeroMQ lightweight messaging kernel (development libraries and header files)
 The 0MQ lightweight messaging kernel is a library which extends the
 standard socket interfaces with features traditionally provided by
 specialised messaging middleware products. 0MQ sockets provide an
 abstraction of asynchronous message queues, multiple messaging
 patterns, message filtering (subscriptions), seamless access to
 multiple transport protocols and more.
 .
 This package contains the ZeroMQ development libraries and header
 files.
Homepage: http://www.zeromq.org/
Original-Maintainer: Martin Lucina <mato@kotelna.sk>

daniel@daniel-u bot$ dpkg -s python-zmq
Package: python-zmq
Status: install ok installed
Priority: optional
Section: python
Installed-Size: 844
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: pyzmq
Version: 2.1.11-1
Depends: python2.7, python (>= 2.7.1-0ubuntu2), python (<< 2.8), libc6 (>= 2.14), libzmq1
Description: Python bindings for 0MQ library
 Python bindings for 0MQ. 0MQ is a small, fast, and free
 software library that gives you message-passing concurrency
 for applications in most common languages.
 .
 The 0MQ lightweight messaging kernel is a library which
 extends the standard socket interfaces with features
 traditionally provided by specialised messaging middleware
 products. 0MQ sockets provide an abstraction of asynchronous
 message queues, multiple messaging patterns, message
 filtering (subscriptions), seamless access to multiple
 transport protocols and more.
Original-Maintainer: Debian Python Modules Team <python-modules-team@lists.alioth.debian.org>
Homepage: http://www.zeromq.org/bindings:python

daniel@daniel-u bot$ pip freeze | grep -i zmq # Requires PIP
pyzmq==2.1.11

This is undesirable, as ZMQ 2.1.11 doesn't support filtering of published topics at the publisher.

From ØMQ v3.x, filtering happens at the publisher side when using a connected protocol (tcp: or ipc:). Using the epgm:// protocol, filtering happens at the subscriber side. In ØMQ v2.x, all filtering happened at the subscriber side.

PyZMQ supports libzmq up to the 4.0 API:

pyzmq aims to support multiple versions of libzmq. pyzmq-14.0 supports
libzmq >= 2.1.4, and has full support for libzmq's 4.0 API.

Getting rid of old libzmq/pyzmq versions

If you have an old version of libzmq/pyzmq installed (like if you followed the directions, quoted above, that lived in the README for quite some time) then you'll want to remove that code. You can quickly check your pyzmq version with pip freeze | grep -i zmq.

daniel@daniel-u bot$ sudo apt-get remove libzmq-dev python-zmq
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  libzmq-dev python-zmq
0 upgraded, 0 newly installed, 2 to remove and 176 not upgraded.
After this operation, 1,758 kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 201626 files and directories currently installed.)
Removing libzmq-dev ...
Removing python-zmq ...
Processing triggers for man-db ...

Building from source

To get a version of ZMQ that supports filtering of PUB/SUB topics at the publisher, I'm currently trying to build libzmq and pyzmq from source. I found these directions useful for libzmq, and these useful for pyzmq.

I started out by reserving a VCL image of Ubuntu 12.04, to work from a clean install.

The following should get you libzmq with support for ZMQ 4.0.3.

drfarrel@vi1-182:~$ python --version # Just showing my Python version
Python 2.7.3
drfarrel@vi1-182:~$ wget http://download.zeromq.org/zeromq-4.0.3.tar.gz # Grab latest ZMQ source
...
drfarrel@vi1-182:~$ tar xzvf zeromq-4.0.3.tar.gz
...
drfarrel@vi1-182:~$ sudo apt-get install libtool autoconf automake # From the ZMQ guide linked above
...
drfarrel@vi1-182:~$ sudo apt-get install uuid-dev # Not sure if this is needed, from ZMQ guide
drfarrel@vi1-182:~$ sudo apt-get install build-essential # Fixes: "configure: error: Unable to find a working C++ compiler" at `./configure` step
drfarrel@vi1-182:~$ cd zeromq-4.0.3/
drfarrel@vi1-182:~$ ./configure
...
drfarrel@vi1-182:~$ make
...
drfarrel@vi1-182:~$ sudo make install # Installs system-wide
...
drfarrel@vi1-182:~/zeromq-4.0.3$ sudo ldconfig # From ZMQ guide linked above
sudo: unable to resolve host vi1-182

This should get you PyZMQ 14.0.1 (currently the latest version), which supports ZMQ 4.0.3.

cd # Back to home dir
drfarrel@vi1-182:~$ wget https://pypi.python.org/packages/source/p/pyzmq/pyzmq-4.0.1.tar.gz#md5=c35fa03e58d48e6f3df2ab2c2dfa1413 # Grab latest version of pyzmq source
...
drfarrel@vi1-182:~$ tar xzvf pyzmq-14.0.1.tar.gz\
...
drfarrel@vi1-182:~$ cd pyzmq-14.0.1/
drfarrel@vi1-182:~$ python setup.py configure --zmq=/usr/local # See PyZMQ link above
drfarrel@vi1-182:~$ sudo apt-get install python-dev # Fixes error about Python.h you would get at the next step
drfarrel@vi1-182:~$ sudo python setup.py install
drfarrel@vi1-182:~$ cd ~ # Get out of PyZMQ source dir so the next line doesn't break
drfarrel@vi1-182:~$ python
Python 2.7.3 (default, Sep 26 2013, 20:03:06) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq # Worked!
>>> 

Here's a preliminary bash script to automate this process.

#!/usr/bin/env bash
# Builds (currently) the latest versions of libzmq and pyzmq from source
# Where most of this came from:
## https://github.com/zeromq/pyzmq/wiki/Building-and-Installing-PyZMQ
## http://zeromq.org/area:download

# Dependencies
sudo apt-get install libtool autoconf automake uuid-dev build-essential python-dev

# Install libzmq from source
wget http://download.zeromq.org/zeromq-4.0.3.tar.gz # Grab latest ZMQ source
tar xzvf zeromq-4.0.3.tar.gz
cd zeromq-4.0.3/
./configure
make
sudo make install # Installs system-wide
sudo ldconfig
cd .. # Back out of ZMQ source dir

# Install pyzmq from source
wget https://pypi.python.org/packages/source/p/pyzmq/pyzmq-14.0.1.tar.gz#md5=c35fa03e58d48e6f3df2ab2c2dfa1413
tar xzvf pyzmq-14.0.1.tar.gz
cd pyzmq-14.0.1/
python setup.py configure --zmq=/usr/local
sudo python setup.py install
cd .. # Back out of pyzmq source dir

# Show results
if [ $? -eq 0 ]; then
  echo "Success! Try to \`import zmq\` in a Python interpreter to test the results."
else
  echo "Failure! Sorry this isn't helpful. Look at the errors above?"
fi

Put that in a file, say one called install_zmq.sh, and then use the following to execute it:

drfarrel@vi1-182:~$ vim install_zmq.sh # Drop the code above in a file
drfarrel@vi1-182:~$ chmod ug+x install_zmq.sh # Make it as executable, for the lulz
drfarrel@vi1-182:~$ ./install_zmq.sh # Takes a few minutes, requires user to accept package install

I've successfully done this on a fresh install of Ubuntu 12.04 (VCL image) as well as my dev laptop, which is also Ubuntu 12.04. I'd like to give it a shot on Debian, as that's what the bot runs on.

Success:

daniel@daniel-u ~$ pip freeze | grep -i zmq # Requires PIP
pyzmq==14.0.1
daniel@daniel-u ~$ ipython --no-banner

In [1]: import zmq # Worked! :)

In [2]: