My former colleague’s from Parse.ly wrote the fantastic pykafka library with an optional c-backed using rdkafka. I’ve had trouble getting it to work, and here are a few things I’ve learned:

• The version of rdkafka installable with apt-get was out of data, and pykafka couldn’t find the headers it need. I instead used the simple build instructions in the rdkafka README to build it from head.
• I was getting the error ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory when trying to use rdkafka from Pykafka. It could be set in the short term by using LD_LIBRARY_PATH=/usr/local/lib. However, I fixed it permanently by running sudo ldconfig after building rdkafka.
• Pykafka has to be installed after building rdkafka. At the moment, Pykafka tries to build a C-extension to connect to rdkafka, and if that fails, it will install without offering the rdkafka backend. Check the output of pip install pykafka to see if the rdkafka extension built.